Learn
How to review test cases created by AI agents
Review AI-written test cases the way you would a new colleague's: have the agent create them in a Draft state, then check each one for a real requirement, a clear expected result, the right template fields, a link to its issue and no duplicates, before a person moves it to Ready or Approved. The agent drafts; a person approves.
01Why agent-written cases need a review step
An agent connected over MCP can turn a ticket into ten cases in a minute. Most will be useful; some will restate the ticket instead of testing it, repeat an existing case, or describe behaviour the product does not have. Reviewing is cheaper than finding out during a release that the "covered" requirement was covered by a case nobody would ever run.
02How states work in TestOrchestrator
Every case has a lifecycle state. In a new workspace the states are:
| State | Use it for |
|---|---|
| Draft | Everything an agent creates, until a person has read it |
| In Review | A person is reviewing it now |
| Ready | Reviewed and usable in runs; the default state for a new case |
| Approved | Signed off, where your team signs off cases |
| Rejected | Reviewed and not wanted; keep it out of runs |
| Archived | No longer relevant |
Because Ready is the default, a case created without a state skips review. Tell the agent which state to use: in our Codex run, the agent called get_lifecycles to find the id of Draft and passed it to create_test_case, and the new case appeared in the list as Draft next to an older case in Ready. A reviewer changes the state in the case's Properties panel.
03The review checklist
For each agent-written case, check:
- It tests a real requirement. It is linked to the issue it covers (
link_to_issue), and the issue says the behaviour it checks. - The expected result is observable. "Works correctly" is not an expected result; "the order total shows the discount" is.
- The steps are reproducible. Someone new to the feature could follow them without asking.
- The template is filled. Priority, severity and any custom fields your project requires are set, not left empty. The agent can read them with
get_fields. - It is not a duplicate. Search the folder for the same scenario before accepting it.
- It is in the right folder and uses your tags.
- It does not invent behaviour. Anything the product does not do yet is either removed or turned into a question on the ticket.
- The title says what is tested, so the case reads well in a run.
Then move it to Ready or Approved, or to Rejected with a comment saying why.
04Keep human approval in the loop
- Write the rule into the agent's instructions: create cases in Draft, never set Ready, Approved or Rejected, never close a test run.
- Restrict tools where the client allows it. Codex's
enabled_tools, or Claude Code's per-tool permissions, can leaveupdate_test_caseandclose_test_runout of an authoring session. - Review from the case list. The State column shows which cases are still Draft.
- Know the limit. An agent acts with its owner's permissions. TestOrchestrator does not stop it from changing a state its owner could change, and its edits are recorded under the owner's name. Approval stays human because your instructions and your review say so, not because the tool enforces it.
06Frequently asked questions
Should AI-generated test cases be reviewed?
How do I keep human approval in an AI testing workflow?
Can I tell which cases an agent created?
What lifecycle states does TestOrchestrator have?
get_lifecycles tool.