Chimp
QA: testing your website or app
QA checks the built website or app against its own spec before anyone outside your team sees it. It runs on a locked test machine, never on your live site. Open the app and click QA in the steps across the top.
Run QA
Press Run QA (Run QA again after the first run). The first time, Chimp writes a test for every acceptance criterion in your spec. Then the run checks:
| Acceptance criteria | Each criterion is a browser test, run at phone, tablet and desktop sizes, along with the home page loading without errors. |
|---|---|
| Accessibility | Every page is checked for serious accessibility problems. |
| Fits every screen | No page scrolls sideways on a phone, a tablet or a desktop; a screenshot of each is kept. |
| Content and links | No placeholder text left behind, no broken images or links, and every image described. |
| Security | No secrets in the files, no form that sends to another site, no look-alike sign-in page, no redirect away, and the Report this site link on every page. |
| Speed and search | Speed, Accessibility, Best practices and Search (SEO) scores for the home page, each against its budget. |
It takes a few minutes. You can leave the page; the results and any bugs are there when you come back.
If a run fails on the test machine before it can test your app, it files no bugs and asks for no fixes. Checks shows "QA run N could not finish" with the reason: a problem on our side, not in your app. Press Run QA again; if it fails the same way again, contact VeraGen support.
Forms are checked after release
A site's forms send only from the published site (Forms send to you). A criterion that a form's message is sent, confirmed or received, such as "After sending, the visitor sees Request received", is marked Checked on the published site on every kind of site, and the smoke test sends a test message through the forms after you publish (Release). Checks of a form's fields and its errors, such as "Email is required", run on the test machine as usual.
Online stores: the shop is checked after release
An online store's shop (its products, prices, stock, cart, checkout, orders, gift cards, bookings and customer accounts) opens only on the published site, never on the test machine. Criteria that need the shop are marked Checked on the published site: by what they say, by the story they belong to (such as "Request a quote" or "Book a trip"), and by where their test goes (the shop's pages, such as the product pages, the cart, the checkout and My account, and any page that shows "The shop opens on the published site."). QA does not run them, files no bugs for them, and The QA gate lists them on their own line instead of passing or failing them. Everything else in the store, such as its pages, layout, accessibility, content, links, security and scores, is checked as usual. A client portal's My account area is treated the same way. A bug an earlier run filed for such a criterion is closed, with a note saying why. After you publish, the smoke test checks the shop on the live address. See Release.
The first run, the first run after the spec changes, and the first run after Chimp improves how tests are written, starts with Writing the test plan…: the AI QA engineer turns every acceptance criterion into a test, usually in a minute or two, and the run then starts by itself. The tests use the buttons, links, field labels and words your built pages really show. A test that would need a control the page does not show is marked for a person to check instead. Pressing Run QA again while it is writing does not start a second one. If the plan cannot be written, nothing is saved and Checks says why, for example "The test plan came back incomplete. Press Run QA to try again." The same reason is in the app's History.
Bugs
Everything that fails becomes a bug in the project's Skunk Works: how bad it is, the steps to reproduce it, what was expected and what happened, the failing check, the story it breaks, and a screenshot (one for each failing criterion, at desktop width). The same failure in a later run is added to the same bug, never filed twice. When a check passes again, its bug is closed as fixed. Closing or reopening a bug in Skunk Works does the same in QA.
The fix loop
When a run fails, the AI engineer fixes the bugs QA found and QA runs again by itself when the fix is built. Each round takes up to 12 bugs, the most severe first; the rest stay open in Skunk Works for the next round, and History says how many were sent and how many are still open. It gets up to three rounds; owners and admins can choose fewer (down to none) under Fix rounds on Checks. When the rounds are used up, Chimp stops, says so, and assigns the open bugs to the person who started the app. Fix them, then press Run QA again.
Bugs reported by testers, and criteria marked It does not, are not fixed automatically: they wait for a person.
If you close all the bugs behind a fix in Skunk Works, for example a bug that is not really in your app, the fix is cancelled and Build goes on without it. A fix can also be dropped by hand on Build with Drop this fix. See Build.
The tabs
| Checks | The latest run, the fix loop, the bugs, and The QA gate. |
|---|---|
| Staging | A private copy of the build for your team, and the people you invite to test it. See People testing, below. |
| Report | One run in full: each acceptance criterion with a pass or fail, the scores against their budgets, every check, the bugs by severity, screenshots and simulated customers. Choose another run from Run. |
| Tests | The tests, written as steps, the criteria checked on the published site (online stores), and the criteria only a person can check (such as "the confirmation email arrives within a minute"). For those, try it and press It works or It does not; It does not files a bug. Rewrite from the spec writes the tests again after the spec changes. |
Try it as customers (on Checks, after a run) has simulated customers from your spec's audiences try the main tasks, and says where they got stuck. It takes a minute or so; you can leave the page while they try. If they cannot finish, it says why and keeps the earlier result.
The AI box at the top, Ask about a failure, or ask to run the checks again…, answers questions about the run, such as "Why did the booking test fail?", and can offer to run QA again.
People testing
On Staging, Open staging shows your team the build QA tests. It is locked to the people you invite, and the link lasts half an hour.
Under Testers, add email addresses and press Send invitations: each person gets a private link by email, with no account needed. They try the app, press Report a bug, draw on the page to show where, say what went wrong and how bad it is, and press Send bug. It becomes a Skunk Works bug with their drawing, the page, their browser and window size, and the pages they went through. A bug marked High blocks approval until it is fixed. Remove stops a tester's link working at once. An app can have up to 25 testers.
Approving QA
QA can be approved when every part of The QA gate is green:
- QA has run.
- Every acceptance criterion passes.
- Every criterion a person checks is confirmed.
- The scores meet the budget (a run without scores does not count).
- No blocker or critical bug is open.
For an online store, Checked on the published site shows how many criteria wait for the live shop. It does not stop approval: the smoke test checks the shop after release.
Minor and major bugs may stay open; the release notes list them. Owners and admins approve, with Approve QA and continue at the foot of Checks. Then Release opens. See Release.