---
title: "Report automated results"
description: "Upload JUnit XML, Playwright JSON or TestOrchestrator JSON from CI: formats, the request, GitHub Actions, GitLab CI and CircleCI snippets, how results find their case, outcomes and statuses, when runs close, unmatched results and reading results back."
source: "https://testorchestrator.com/docs/api-keys/report-automated-results"
---

# Report automated results

One request at the end of a CI job uploads its results file. TestOrchestrator files the results on a test run named after the job, matches each one to a test case, and keeps anything it cannot match on the run’s **Unmatched** tab so you can link it later. It reads **JUnit XML**, **Playwright JSON** and its own JSON.

> **Result upload opens in a following release; keys can be created now.** Until it opens, `POST /api/v1/ingest/results` checks the key and its scope and then answers `503 INGEST_NOT_AVAILABLE` with `Retry-After`. Everything below describes the route once it is open.

## Before you start

1. **A workspace service key.** A Tenant Admin creates one on **Settings → API keys → Workspace service keys** — see [Create a service key](https://testorchestrator.com/docs/api-keys#create-a-service-key). Every service key holds the `ingest:results` scope; tick `read` too if the job will read results back. Store it as a CI secret, for example `TO_API_KEY`. A key made for one project needs no `project` field.
2. **Cases that results can find.** Give a case an [automation key](https://testorchestrator.com/docs/test-cases/case-detail#automation-key) equal to the test’s id, or put the case reference in the test name (`C-123`). To key many cases at once, select them on **Test Cases**, open **Actions** and choose **Set automation keys…** — it previews every key and skips conflicts before anything is written.

## Formats

Send the file your test runner already writes. The `format` field names it.

| `format` | Written by, and the test’s id |
| --- | --- |
| `junit-xml` | pytest `--junitxml`, JUnit (Maven Surefire, Gradle), jest-junit, the Vitest and Playwright `junit` reporters, and Cypress through mocha-junit-reporter. The test’s id is `classname#name` — for pytest, `tests.test_cart#test_adds_line_items`. |
| `playwright-json` | Playwright’s `json` reporter (`--reporter=json`). The test’s id is the title path joined with ` › `: file, `describe` blocks, test title. The browser project is not part of it — it becomes the result’s configuration, so one test run in Chromium and Firefox is one case with two configurations. |
| `to-results-json` | Your own reporter, with your own `test_id`. |

TestOrchestrator’s own JSON is a `version` and a list of results. Each result needs `test_id`, `title` and `outcome` (`passed`, `failed`, `errored`, `timed_out`, `skipped`, `flaky_passed` or `inconclusive`); everything else is optional:

```json
{
  "version": 1,
  "results": [
    {
      "test_id": "checkout.applies-coupon",
      "title": "Applies coupon",
      "outcome": "passed",
      "duration_ms": 812
    },
    {
      "test_id": "checkout.renders-receipt",
      "title": "Renders receipt",
      "outcome": "failed",
      "environment": "chromium",
      "failure": {
        "type": "AssertionError",
        "message": "expected 42.00, received 41.00"
      }
    }
  ]
}
```

A file that cannot be read is refused whole with `400 RESULTS_FILE_INVALID`, and `details` says where (a line and column, or a JSON pointer such as `/results/1/outcome`). Nothing is written.

## The request

Send the file as `multipart/form-data` to `https://api.testorchestrator.com/api/v1/ingest/results`, with the key as a bearer token. `file` and `format` are required.

| Field | What it does |
| --- | --- |
| `file`, `format` | The results file and its format (above). The file may be empty for a final `complete=true` request. |
| `run_key` | Your id for the logical run, for example the CI run id. Uploads with the same key add to the same run, so shards and matrix jobs land together. Uploading a test again on the same key replaces its result and keeps the earlier one as history. |
| `run_name` | The new run’s name. Without it the run is called `CI · <branch> · #<build>`. |
| `complete` | `true` closes the run after this upload. Unmatched failures do not stop it. |
| `environment` | An Environment for the whole upload. Leave it out for multi-browser suites: each result’s configuration (the Playwright project, for example `chromium`) is matched to the Environment of the same name, ignoring case. |
| `tags` | Comma-separated names of **Automation** tags (**Settings → Tags**). Matching tags are added to the run; a name that is not an Automation tag is reported in `warnings` and never created. |
| `ci_provider`, `ci_branch`, `ci_commit`, `ci_build_number`, `ci_job_url`, `ci_actor` | Facts about the CI job, shown on the run. |
| `project` | Only for an all-projects key: the project’s `prj_` id — see [project binding](https://testorchestrator.com/docs/api-keys#project-binding). |
| `Idempotency-Key` (header) | Sending the same upload again with the same key returns the first answer with `Idempotent-Replay: true` and writes nothing twice. Use the CI run id plus the attempt and the shard. |

An accepted upload answers `202` with the run it landed on and what happened to the results. This is the GitLab snippet below, run against a project whose cases have no keys yet:

```json
{
  "ingest_id": "ing_f0httptky5h8rd2j",
  "status": "accepted",
  "run_id": 7,
  "run_ref": "TR-7",
  "run_key": "gitlab-1179291",
  "counts": {
    "total": 6, "matched": 0, "unmatched": 6, "unmatched_overflow": 0, "superseded": 0,
    "by_outcome": { "passed": 4, "failed": 1, "skipped": 1, "errored": 0, "timed_out": 0, "flaky_passed": 0, "inconclusive": 0 }
  },
  "complete": true,
  "warnings": [
    "1 unmatched failure(s) count in this run's Failed total; link them to cases or set automation keys.",
    "The run was closed with unmatched failures; they stay on the run as unmatched."
  ]
}
```

Every field and error code is in the [API reference](https://testorchestrator.com/docs/api-keys/reference#post-api-v1-ingest-results); the refusals and what to do about each are in [API keys → Troubleshooting](https://testorchestrator.com/docs/api-keys#troubleshooting).

## CI snippets

Each one uploads from a step that runs even when the tests failed, so failures are reported too. Store the key as a CI secret named `TO_API_KEY`.

### GitHub Actions

```yaml
# .github/workflows/e2e.yml
- name: Run Playwright
  run: npx playwright test --reporter=json > results.json
- name: Report to TestOrchestrator
  if: always()
  run: |
    curl -sS --fail-with-body -X POST https://api.testorchestrator.com/api/v1/ingest/results \
      -H "Authorization: Bearer ${{ secrets.TO_API_KEY }}" \
      -H "Idempotency-Key: ${{ github.run_id }}-${{ github.run_attempt }}" \
      -F file=@results.json \
      -F format=playwright-json \
      -F run_key=gha-${{ github.run_id }} \
      -F ci_provider=github \
      -F ci_branch=${{ github.ref_name }} \
      -F ci_commit=${{ github.sha }} \
      -F ci_build_number=${{ github.run_number }} \
      -F complete=true
```

### GitLab CI

```yaml
# .gitlab-ci.yml
pytest:
  script:
    - pytest --junitxml=report.xml
  after_script:
    - |
      curl -sS --fail-with-body -X POST https://api.testorchestrator.com/api/v1/ingest/results \
        -H "Authorization: Bearer $TO_API_KEY" \
        -H "Idempotency-Key: $CI_PIPELINE_ID-$CI_JOB_ID" \
        -F file=@report.xml \
        -F format=junit-xml \
        -F run_key=gitlab-$CI_PIPELINE_ID \
        -F ci_provider=gitlab \
        -F ci_branch=$CI_COMMIT_REF_NAME \
        -F ci_commit=$CI_COMMIT_SHA \
        -F ci_build_number=$CI_PIPELINE_IID \
        -F complete=true
```

### CircleCI

```yaml
# .circleci/config.yml (a step of the job)
- run:
    name: Report to TestOrchestrator
    when: always
    command: |
      curl -sS --fail-with-body -X POST https://api.testorchestrator.com/api/v1/ingest/results \
        -H "Authorization: Bearer $TO_API_KEY" \
        -H "Idempotency-Key: $CIRCLE_WORKFLOW_ID-$CIRCLE_NODE_INDEX" \
        -F file=@test-results/junit.xml \
        -F format=junit-xml \
        -F run_key=circle-$CIRCLE_WORKFLOW_ID \
        -F ci_provider=circleci \
        -F ci_branch=$CIRCLE_BRANCH \
        -F ci_commit=$CIRCLE_SHA1 \
        -F ci_build_number=$CIRCLE_BUILD_NUM \
        -F complete=true
```

**Sharded suites.** Every shard uploads with the same `run_key` and without `complete`; one final step sends `complete=true`, with an empty file if it has no results of its own. The shards land on one run.

## Screenshots, traces and logs

A result in TestOrchestrator’s own JSON can carry files. Upload each file first with the same key — prepare, `PUT` the bytes, create — then reference it from the result: `"artifacts": [{ "name": "trace", "attachment_id": 1234 }]`. The steps, sizes and media types are in [API keys → Attach screenshots, traces and logs](https://testorchestrator.com/docs/api-keys#attach-screenshots-traces-and-logs). A Playwright JSON report only names its attachments; the files on the CI machine are never fetched.

## How results find their case

For every result, the first of these rules that names exactly one case wins, and the result shows which one (**Matched by**):

1. **A case reference.** `C-123` in the test name, its tags or its annotations (`TC-123` works too). `#123` counts only inside a tag or an annotation, never in the test name, where it usually means an issue.
2. **The automation key.** The test’s id equals a case’s automation key — exactly, case-sensitive, after trimming spaces.
3. **The title**, only when **Match by title** is on for the project: the test’s title equals one case’s title, ignoring case, accents and punctuation. Turn it on in the **Automation** panel of the project’s settings. It is off by default and applies from the next upload.

> **A case reference always matches.** Turning Match by title off does not stop a `C-123` in a test name from matching that case — the reference is rule 1, and the setting only governs rule 3.

A rule that names two cases matches neither: the result is unmatched as _ambiguous_. A result no rule matches stays on the run’s **Unmatched** tab; when it named a case that was deleted, the tab says so.

## Outcomes and statuses

Each runner’s verdict becomes one of seven outcomes, and each outcome lands on a status of your workspace. A new workspace maps them like this:

| Outcome | When | Status |
| --- | --- | --- |
| `passed` | the test passed | Passed |
| `flaky_passed` | it passed on a retry; the result keeps a Flaky flag | Passed |
| `failed` | an assertion failed | Failed |
| `timed_out` | the runner’s timeout hit | Failed |
| `errored` | the test threw before asserting | Exception |
| `skipped` | skipped by the runner | Skipped |
| `inconclusive` | the runner reported no verdict | Untested |

Change where an outcome lands on **Settings → Statuses**, in the **Automation outcomes** panel: pick a status for it, or **Unmap** it. A change applies to the next upload; results already stored keep their status. An outcome that is a status’s own system name (`failed` for Failed) always lands there. An outcome no status catches is stored with its raw outcome and shows as **Unmapped**.

## When the run closes

- **`complete=true`** closes the run after that upload.
- Otherwise the run **closes by itself** a set number of hours after its last upload — 24 by default, any whole number from 1 to 720, on **Settings → Automation** (Tenant Admins change it). Lowering it applies to runs already open: a run whose last upload is older than the new value closes at the next sweep, within 5 minutes.
- **Closing it in the app** asks you to confirm when the run still has unmatched failures, and lists them.

A closed run keeps its Unmatched tab, so you can still link those results to cases.

## Unmatched results

A result no rule matched is kept on its run’s [Unmatched tab](https://testorchestrator.com/docs/test-runs/run-detail#unmatched-tab), and an unmatched failure still counts in the run’s Failed total. From there:

- **Link to case** files the results on a case. Linking one test also writes its id as the case’s automation key, so the next upload matches by itself.
- **Dismiss** takes results off the tab; dismissed results can be restored.
- **Set automation keys** on **Test Cases** keys many cases at once, so the next upload matches them.

## Reading results back

With a key that holds the `read` scope, `GET /api/v1/projects/{project}/automated-results` lists the project’s automated results, oldest first: every result, including ones a later upload replaced (`current: false`), unmatched ones (`case: null`) and dismissed ones (`dismissed: true`). Narrow it with `since`, `environment`, `outcome`, `fingerprint`, `case` (`C-123`, or `unmatched`) and `run` (`TR-12`). The answer is a page, `{ "data": […], "next_cursor": … }`: pass `next_cursor` back as `cursor` until it is null.

```bash
curl -sS "https://api.testorchestrator.com/api/v1/projects/prj_8kd2hq0vxc1a/automated-results?outcome=failed,timed_out&since=2026-10-01" \
  -H "Authorization: Bearer $TO_API_KEY"
```

`…/automated-results.csv` takes the same filters and returns the same rows as a CSV file (UTF-8, no byte-order mark, one header row). When more rows match than one file holds, the answer carries `X-Next-Cursor` and a `Link: <…>; rel="next"` header for the next file. A text cell that starts with `=`, `+`, `-` or `@` gets a leading `'`, so a spreadsheet never runs it as a formula.

## Limits

Each key has three rate limits of its own, counted separately: uploads, result pages and CSV exports. A request over one is refused with `429 RATE_LIMIT_EXCEEDED` and a `Retry-After` header — wait that many seconds and send it again. Uploads also count toward fair-use limits on results per day and uploads per month. The numbers are on [API keys → Retries and rate limits](https://testorchestrator.com/docs/api-keys#retries-and-rate-limits) and [Fair-use limits](https://testorchestrator.com/docs/api-keys#fair-use-limits).

## Related

- [API keys](https://testorchestrator.com/docs/api-keys) — service keys, scopes, limits and every error code.
- [API reference](https://testorchestrator.com/docs/api-keys/reference) — every route and field, generated from the OpenAPI document.
- [Run Detail → Unmatched tab](https://testorchestrator.com/docs/test-runs/run-detail#unmatched-tab) — linking and dismissing results.
- [Case Detail → Automation key](https://testorchestrator.com/docs/test-cases/case-detail#automation-key) — the key a result matches.