Report automated results

One request at the end of a CI job uploads its results file. TestOrchestrator files the results on a test run named after the job, matches each one to a test case, and keeps anything it cannot match on the run’s Unmatched tab so you can link it later. It reads JUnit XML, Playwright JSON and its own JSON.

Result upload opens in a following release; keys can be created now. Until it opens, POST /api/v1/ingest/results checks the key and its scope and then answers 503 INGEST_NOT_AVAILABLE with Retry-After. Everything below describes the route once it is open.

Before you start

1. A workspace service key. A Tenant Admin creates one on Settings → API keys → Workspace service keys — see Create a service key. Every service key holds the ingest:results scope; tick read too if the job will read results back. Store it as a CI secret, for example TO_API_KEY. A key made for one project needs no project field.
2. Cases that results can find. Give a case an automation key equal to the test’s id, or put the case reference in the test name (C-123). To key many cases at once, select them on Test Cases, open Actions and choose Set automation keys… — it previews every key and skips conflicts before anything is written.

Formats

Send the file your test runner already writes. The format field names it.

TestOrchestrator’s own JSON is a version and a list of results. Each result needs test_id, title and outcome (passed, failed, errored, timed_out, skipped, flaky_passed or inconclusive); everything else is optional:

{
  "version": 1,
  "results": [
    {
      "test_id": "checkout.applies-coupon",
      "title": "Applies coupon",
      "outcome": "passed",
      "duration_ms": 812
    },
    {
      "test_id": "checkout.renders-receipt",
      "title": "Renders receipt",
      "outcome": "failed",
      "environment": "chromium",
      "failure": {
        "type": "AssertionError",
        "message": "expected 42.00, received 41.00"
      }
    }
  ]
}

A file that cannot be read is refused whole with 400 RESULTS_FILE_INVALID, and details says where (a line and column, or a JSON pointer such as /results/1/outcome). Nothing is written.

The request

Send the file as multipart/form-data to https://api.testorchestrator.com/api/v1/ingest/results, with the key as a bearer token. file and format are required.

An accepted upload answers 202 with the run it landed on and what happened to the results. This is the GitLab snippet below, run against a project whose cases have no keys yet:

{
  "ingest_id": "ing_f0httptky5h8rd2j",
  "status": "accepted",
  "run_id": 7,
  "run_ref": "TR-7",
  "run_key": "gitlab-1179291",
  "counts": {
    "total": 6, "matched": 0, "unmatched": 6, "unmatched_overflow": 0, "superseded": 0,
    "by_outcome": { "passed": 4, "failed": 1, "skipped": 1, "errored": 0, "timed_out": 0, "flaky_passed": 0, "inconclusive": 0 }
  },
  "complete": true,
  "warnings": [
    "1 unmatched failure(s) count in this run's Failed total; link them to cases or set automation keys.",
    "The run was closed with unmatched failures; they stay on the run as unmatched."
  ]
}

Every field and error code is in the API reference; the refusals and what to do about each are in API keys → Troubleshooting.

CI snippets

Each one uploads from a step that runs even when the tests failed, so failures are reported too. Store the key as a CI secret named TO_API_KEY.

GitHub Actions

# .github/workflows/e2e.yml
- name: Run Playwright
  run: npx playwright test --reporter=json > results.json
- name: Report to TestOrchestrator
  if: always()
  run: |
    curl -sS --fail-with-body -X POST https://api.testorchestrator.com/api/v1/ingest/results \
      -H "Authorization: Bearer ${{ secrets.TO_API_KEY }}" \
      -H "Idempotency-Key: ${{ github.run_id }}-${{ github.run_attempt }}" \
      -F file=@results.json \
      -F format=playwright-json \
      -F run_key=gha-${{ github.run_id }} \
      -F ci_provider=github \
      -F ci_branch=${{ github.ref_name }} \
      -F ci_commit=${{ github.sha }} \
      -F ci_build_number=${{ github.run_number }} \
      -F complete=true

GitLab CI

# .gitlab-ci.yml
pytest:
  script:
    - pytest --junitxml=report.xml
  after_script:
    - |
      curl -sS --fail-with-body -X POST https://api.testorchestrator.com/api/v1/ingest/results \
        -H "Authorization: Bearer $TO_API_KEY" \
        -H "Idempotency-Key: $CI_PIPELINE_ID-$CI_JOB_ID" \
        -F file=@report.xml \
        -F format=junit-xml \
        -F run_key=gitlab-$CI_PIPELINE_ID \
        -F ci_provider=gitlab \
        -F ci_branch=$CI_COMMIT_REF_NAME \
        -F ci_commit=$CI_COMMIT_SHA \
        -F ci_build_number=$CI_PIPELINE_IID \
        -F complete=true

CircleCI

# .circleci/config.yml (a step of the job)
- run:
    name: Report to TestOrchestrator
    when: always
    command: |
      curl -sS --fail-with-body -X POST https://api.testorchestrator.com/api/v1/ingest/results \
        -H "Authorization: Bearer $TO_API_KEY" \
        -H "Idempotency-Key: $CIRCLE_WORKFLOW_ID-$CIRCLE_NODE_INDEX" \
        -F file=@test-results/junit.xml \
        -F format=junit-xml \
        -F run_key=circle-$CIRCLE_WORKFLOW_ID \
        -F ci_provider=circleci \
        -F ci_branch=$CIRCLE_BRANCH \
        -F ci_commit=$CIRCLE_SHA1 \
        -F ci_build_number=$CIRCLE_BUILD_NUM \
        -F complete=true

Sharded suites. Every shard uploads with the same run_key and without complete; one final step sends complete=true, with an empty file if it has no results of its own. The shards land on one run.

Screenshots, traces and logs

A result in TestOrchestrator’s own JSON can carry files. Upload each file first with the same key — prepare, PUT the bytes, create — then reference it from the result: "artifacts": [{ "name": "trace", "attachment_id": 1234 }]. The steps, sizes and media types are in API keys → Attach screenshots, traces and logs. A Playwright JSON report only names its attachments; the files on the CI machine are never fetched.

How results find their case

For every result, the first of these rules that names exactly one case wins, and the result shows which one (Matched by):

1. A case reference. C-123 in the test name, its tags or its annotations (TC-123 works too). #123 counts only inside a tag or an annotation, never in the test name, where it usually means an issue.
2. The automation key. The test’s id equals a case’s automation key — exactly, case-sensitive, after trimming spaces.
3. The title, only when Match by title is on for the project: the test’s title equals one case’s title, ignoring case, accents and punctuation. Turn it on in the Automation panel of the project’s settings. It is off by default and applies from the next upload.

A case reference always matches. Turning Match by title off does not stop a C-123 in a test name from matching that case — the reference is rule 1, and the setting only governs rule 3.

A rule that names two cases matches neither: the result is unmatched as ambiguous. A result no rule matches stays on the run’s Unmatched tab; when it named a case that was deleted, the tab says so.

Outcomes and statuses

Each runner’s verdict becomes one of seven outcomes, and each outcome lands on a status of your workspace. A new workspace maps them like this:

Change where an outcome lands on Settings → Statuses, in the Automation outcomes panel: pick a status for it, or Unmap it. A change applies to the next upload; results already stored keep their status. An outcome that is a status’s own system name (failed for Failed) always lands there. An outcome no status catches is stored with its raw outcome and shows as Unmapped.

When the run closes

• complete=true closes the run after that upload.
• Otherwise the run closes by itself a set number of hours after its last upload — 24 by default, any whole number from 1 to 720, on Settings → Automation (Tenant Admins change it). Lowering it applies to runs already open: a run whose last upload is older than the new value closes at the next sweep, within 5 minutes.
• Closing it in the app asks you to confirm when the run still has unmatched failures, and lists them.

A closed run keeps its Unmatched tab, so you can still link those results to cases.

Unmatched results

A result no rule matched is kept on its run’s Unmatched tab, and an unmatched failure still counts in the run’s Failed total. From there:

• Link to case files the results on a case. Linking one test also writes its id as the case’s automation key, so the next upload matches by itself.
• Dismiss takes results off the tab; dismissed results can be restored.
• Set automation keys on Test Cases keys many cases at once, so the next upload matches them.

Reading results back

With a key that holds the read scope, GET /api/v1/projects/{project}/automated-results lists the project’s automated results, oldest first: every result, including ones a later upload replaced (current: false), unmatched ones (case: null) and dismissed ones (dismissed: true). Narrow it with since, environment, outcome, fingerprint, case (C-123, or unmatched) and run (TR-12). The answer is a page, { "data": […], "next_cursor": … }: pass next_cursor back as cursor until it is null.

curl -sS "https://api.testorchestrator.com/api/v1/projects/prj_8kd2hq0vxc1a/automated-results?outcome=failed,timed_out&since=2026-10-01" \
  -H "Authorization: Bearer $TO_API_KEY"

…/automated-results.csv takes the same filters and returns the same rows as a CSV file (UTF-8, no byte-order mark, one header row). When more rows match than one file holds, the answer carries X-Next-Cursor and a Link: <…>; rel="next" header for the next file. A text cell that starts with =, +, - or @ gets a leading ', so a spreadsheet never runs it as a formula.

Limits

Each key has three rate limits of its own, counted separately: uploads, result pages and CSV exports. A request over one is refused with 429 RATE_LIMIT_EXCEEDED and a Retry-After header — wait that many seconds and send it again. Uploads also count toward fair-use limits on results per day and uploads per month. The numbers are on API keys → Retries and rate limits and Fair-use limits.

Related

• API keys — service keys, scopes, limits and every error code.
• API reference — every route and field, generated from the OpenAPI document.
• Run Detail → Unmatched tab — linking and dismissing results.
• Case Detail → Automation key — the key a result matches.