---
title: "CI integrations"
description: "Report automated results from CI with the Playwright reporter, the report-results GitHub Action for pytest, Jest, JUnit and Playwright, or TestOrchestrator JSON with its published schema and validator."
source: "https://testorchestrator.com/docs/api-keys/ci-integrations"
---

# CI integrations

Three ready-made ways to send automated results from CI, from the shortest to the most flexible: the **Playwright reporter** (one line in `playwright.config.ts`), the **report-results GitHub Action** (any runner that writes JUnit XML or a Playwright JSON report), and **TestOrchestrator JSON** with its published schema, for runners you report from yourself. All three use the same upload as [Report automated results](https://testorchestrator.com/docs/api-keys/report-automated-results), so runs, matching and Unmatched results work the same way.

## Before you start

1. **A workspace service key.** A Tenant Admin creates one on **Settings → API keys → Workspace service keys** — see [Create a service key](https://testorchestrator.com/docs/api-keys#create-a-service-key). Store it as a CI secret named `TO_API_KEY`.
2. **The project’s `prj_` id**, from the project’s settings page — the GitHub Action always takes it; the reporter needs it only with an all-projects key ([project binding](https://testorchestrator.com/docs/api-keys#project-binding)).
3. **Cases that results can find** — an [automation key](https://testorchestrator.com/docs/test-cases/case-detail#automation-key) equal to the test’s id, or a `C-123` reference in the test. Until then every result is kept on the run’s [Unmatched tab](https://testorchestrator.com/docs/test-runs/run-detail#unmatched-tab), where **Link to case** writes the key for the next run.

| You run | Use |
| --- | --- |
| Playwright, anywhere | [The Playwright reporter](https://testorchestrator.com/docs/api-keys/ci-integrations#playwright-reporter) — retries, browsers, shards and attachments included |
| pytest, Jest, JUnit, Vitest, Cypress or Playwright on GitHub Actions | [The GitHub Action](https://testorchestrator.com/docs/api-keys/ci-integrations#github-action) with the JUnit XML or Playwright JSON file your runner writes |
| anything else, or your own reporter | [TestOrchestrator JSON](https://testorchestrator.com/docs/api-keys/ci-integrations#results-schema) — validate it, then upload it with the Action or `curl` |

## Playwright reporter

`@testorchestrator/playwright-reporter` writes the run’s results at the end of `npx playwright test` and uploads them. It maps every test exactly as an upload of Playwright’s own `json` report is mapped, so the two give the same results.

```bash
npm install --save-dev @testorchestrator/playwright-reporter
```

```ts
// playwright.config.ts
import { defineConfig } from '@playwright/test'

export default defineConfig({
  reporter: [['list'], ['@testorchestrator/playwright-reporter']],
})
```

A config that says `reporter: 'html'` becomes `reporter: [['html'], ['@testorchestrator/playwright-reporter']]`. Then run the tests with the key in the environment:

```bash
TO_API_KEY=<your workspace service key> npx playwright test
```

```text
[testorchestrator] 26 results → TR-3 (run 3) · 16 matched, 10 unmatched · 18 passed, 2 failed, 2 skipped, 2 timed_out, 2 flaky_passed · run closed
```

- **The test’s id** is its title path without the browser project: `checkout.spec.ts › checkout › applies coupon`. Each project (`chromium`, `firefox`) is the result’s configuration, matched to the Environment of the same name.
- **Retries** are kept as attempts; a test that passed on a retry is `flaky_passed`. `test.fail()`, skips, tags and annotations are reported too.
- **On GitHub Actions** the reporter sets the run key to `<run id>-<run attempt>` and adds the branch, commit, run number and a link to the workflow run. Elsewhere, set `TO_RUN_KEY` to an id that is new for every pipeline attempt.
- **Shards** with the same run key land on one run, and **no shard closes it**: shards finish in any order and a run closes on the first upload that says `complete`, so with `--shard` the reporter uploads with `complete: false`. Close the run once every shard has uploaded — with an empty upload with `complete=true` from a final job ([below](https://testorchestrator.com/docs/api-keys/ci-integrations#close-the-run)) — or let it close by itself after the workspace’s auto-close delay. Merging blob reports with `npx playwright merge-reports` and this reporter in the merge config uploads once and closes the run.
- **Screenshots and traces**: with `attachments: 'upload'` the files are uploaded and linked to the result; by default they are recorded by name.
- **Your test run is never failed by the upload.** A refused upload is logged with what to do, and the results stay in `testorchestrator-results.json`. Set `failOnError: true` to fail the run instead. Without a key the reporter writes the file and uploads nothing.

| Option | Environment | What it does |
| --- | --- | --- |
| `apiKey` | `TO_API_KEY` | The service key. Never printed. |
| `project` | `TO_PROJECT` | The `prj_` id, for an all-projects key. |
| `runKey`, `runName` | `TO_RUN_KEY`, `TO_RUN_NAME` | The run to upload to, and the name of a new run (by default `CI · <branch> · #<build>`). |
| `environment` | `TO_ENVIRONMENT` | One Environment for every result, instead of the project names. |
| `complete` | `TO_COMPLETE` | Close the run after the upload: on by default, off when Playwright runs with `--shard`. |
| `attachments` | — | `'names'` (default), `'upload'` or `'none'`. |
| `dryRun` | `TO_DRY_RUN` | Write the file, upload nothing. |
| `failOnError` | — | Fail the test run when the upload fails. Off by default. |
| `projectUrl` | — | Your project’s address in the app, to print a link to the run. |

Every option, the mapping table and the reporter’s messages are in its [README](https://www.npmjs.com/package/@testorchestrator/playwright-reporter).

## GitHub Action

`testorchestratorhq/report-results` uploads the files your test step wrote, in one step. It works out each file’s format, masks the key, prints the run and writes a job summary.

```yaml
# .github/workflows/tests.yml (the steps of a job)
- run: pytest --junitxml=reports/pytest.xml
- name: Report to TestOrchestrator
  if: ${{ !cancelled() }}
  uses: testorchestratorhq/report-results@v1
  with:
    api-key: ${{ secrets.TO_API_KEY }}
    project: prj_8kd2hq0vxc1a
    files: reports/*.xml
```

`if: ${{ !cancelled() }}` runs the step when tests failed, so failures are reported. The same step reports Jest (`npx jest --ci --reporters=default --reporters=jest-junit`, `files: reports/jest-junit.xml`) and Playwright’s JSON report (`files: playwright-report/results.json`). `reports/jest-junit.xml` is where jest-junit writes when `package.json` sets `"jest-junit": { "outputDirectory": "reports", "outputName": "jest-junit.xml" }`, as the Action’s Jest example does. Complete workflows for all three are in the Action’s [examples](https://github.com/testorchestratorhq/report-results/tree/main/examples).

| Input | Default | What it does |
| --- | --- | --- |
| `api-key`, `project`, `files` | required | The key (from a secret), the `prj_` id, and glob patterns separated by new lines or commas. |
| `format` | `auto` | `.xml` is JUnit XML; JSON with `suites` is a Playwright report; JSON with `version` and `results` is TestOrchestrator JSON, checked against the schema before it is uploaded. |
| `run-key` | `<run id>-<run attempt>` | Uploads with the same key land on one run; **Re-run jobs** gets a new one. |
| `complete` | `true` | Close the run with the last file — only when every file was uploaded; otherwise the run is left open and the job summary says so. Set `false` in matrix and shard jobs and close the run from a final job. |
| `fail-on-ingest-error` | `true` | Fail the step when a file cannot be read, an upload is refused or the API cannot be reached; `false` makes it a warning. |
| `run-name` | `CI · <branch> · #<build>` | The name of a new run. |
| `base-url` | `https://api.testorchestrator.com` | API host. Change it only to point the step at a test stack. |
| `environment`, `tags`, `project-url` | — | As in [the request](https://testorchestrator.com/docs/api-keys/report-automated-results#the-request); `project-url` adds a link to the run. |

Outputs: `run-id`, `run-ref` (`TR-12`), `run-url` (with `project-url`) and `counts` as JSON. Each file is one upload with an `Idempotency-Key` made from the run key, the file’s position, the upload’s fields and its content, so a retried request is answered from the first upload and nothing is written twice; **Re-run jobs** has a new attempt number and uploads to a new run.

## Close the run from a final job

When several jobs report to one run — Playwright shards, a test matrix — each uploads with `complete: false` (the reporter does this by itself under `--shard`), and one job that runs after all of them closes the run with an empty file:

```yaml
close-run:
  needs: test # the matrix or shard job
  if: ${{ !cancelled() }}
  runs-on: ubuntu-latest
  steps:
    - run: printf '' > close.json # no results: it only closes the run
    - uses: testorchestratorhq/report-results@v1
      with:
        api-key: ${{ secrets.TO_API_KEY }}
        project: prj_8kd2hq0vxc1a
        files: close.json
        complete: true
```

Jobs of one workflow run share the default run key, `<run id>-<run attempt>`. Without the final job the run closes by itself after the workspace’s auto-close delay.

## TestOrchestrator JSON and its schema

For a runner neither of the above covers, write TestOrchestrator’s own format (`format=to-results-json`) — a `version` and a list of results, each with `test_id`, `title` and `outcome` (the fields are on [Report automated results](https://testorchestrator.com/docs/api-keys/report-automated-results#formats)). The JSON Schema is published at [`https://testorchestrator.com/schemas/to-results.v1.json`](https://testorchestrator.com/schemas/to-results.v1.json) (draft 2020-12; a breaking change becomes `v2`), and `@testorchestrator/results-schema` checks a file the way the upload reads it:

```bash
npm install --save-dev @testorchestrator/results-schema
npx to-results-validate results.json
```

```text
results.json: invalid (1 problem)
/results/0/outcome: "outcome" must be one of passed, failed, skipped, errored, timed_out, flaky_passed, inconclusive.
```

It exits `0` when every file is valid and `1` otherwise, with each problem as a JSON pointer, as the upload would answer; `--json` prints a machine-readable report. The package also exports the TypeScript types of the format. Upload a valid file with the Action or `curl`:

```bash
curl -sS --fail-with-body -X POST https://api.testorchestrator.com/api/v1/ingest/results \
  -H "Authorization: Bearer $TO_API_KEY" \
  -F file=@results.json \
  -F format=to-results-json \
  -F run_key=ci-1234-1 \
  -F complete=true
```

## Troubleshooting

| Message | What to do |
| --- | --- |
| `not uploaded: no API key` | The key is not in the step’s environment. In GitHub Actions map the secret: `env: { TO_API_KEY: '${{ secrets.TO_API_KEY }}' }`. |
| `401 API_KEY_UNAUTHORIZED` | The key is wrong, revoked or expired, or it is a personal key. Use a workspace service key. |
| `400 PROJECT_REQUIRED` / `INVALID_PROJECT_ID` | Pass the project’s `prj_` id, never the number in the app’s URL. |
| `403 KEY_PROJECT_MISMATCH` | The key belongs to another project or workspace, or the `prj_` id is wrong. |
| `409 RUN_CLOSED` | The run for this run key is closed. Shards and matrix jobs: upload with `complete: false` and close the run once from a final step; otherwise use a run key per CI attempt. |
| `already uploaded (Idempotent-Replay)` | The same file was uploaded before with the same key; nothing was written twice. |
| `… retried 1 time` | Nothing: the reporter and the Action waited (for `429`, as `Retry-After` said) and sent it again. They retry `429`, `5xx`, `409 BUSY` and the network errors that can heal (`ECONNREFUSED`, `ECONNRESET`, `ETIMEDOUT`, `EAI_AGAIN`) up to 5 times; `ENOTFOUND`, `EHOSTUNREACH` and `ENETUNREACH` fail after one attempt. Each attempt is printed with `TO_DEBUG=true` (reporter) or in the Action’s debug log. |
| `closing run TR-12` | The close step worked: an upload with no results and `complete=true`. Without `complete`, an empty upload shows the warning `The file holds no test results.` instead. |

Every refusal and its fix: [API keys → Troubleshooting](https://testorchestrator.com/docs/api-keys#troubleshooting).

## Related

- [Report automated results](https://testorchestrator.com/docs/api-keys/report-automated-results) — the upload itself: fields, answers, matching, outcomes, closing.
- [API keys](https://testorchestrator.com/docs/api-keys) — service keys, scopes, limits and error codes.
- [Run Detail → Unmatched tab](https://testorchestrator.com/docs/test-runs/run-detail#unmatched-tab) — linking results to cases.
- [API reference](https://testorchestrator.com/docs/api-keys/reference) — every route and field.
