CI integrations

Three ready-made ways to send automated results from CI, from the shortest to the most flexible: the Playwright reporter (one line in playwright.config.ts), the report-results GitHub Action (any runner that writes JUnit XML or a Playwright JSON report), and TestOrchestrator JSON with its published schema, for runners you report from yourself. All three use the same upload as Report automated results, so runs, matching and Unmatched results work the same way.

Before you start

  1. A workspace service key. A Tenant Admin creates one on Settings → API keys → Workspace service keys — see Create a service key. Store it as a CI secret named TO_API_KEY.
  2. The project’s prj_ id, from the project’s settings page — the GitHub Action always takes it; the reporter needs it only with an all-projects key (project binding).
  3. Cases that results can find — an automation key equal to the test’s id, or a C-123 reference in the test. Until then every result is kept on the run’s Unmatched tab, where Link to case writes the key for the next run.
You runUse
Playwright, anywhereThe Playwright reporter — retries, browsers, shards and attachments included
pytest, Jest, JUnit, Vitest, Cypress or Playwright on GitHub ActionsThe GitHub Action with the JUnit XML or Playwright JSON file your runner writes
anything else, or your own reporterTestOrchestrator JSON — validate it, then upload it with the Action or curl

Playwright reporter

@testorchestrator/playwright-reporter writes the run’s results at the end of npx playwright test and uploads them. It maps every test exactly as an upload of Playwright’s own json report is mapped, so the two give the same results.

npm install --save-dev @testorchestrator/playwright-reporter
// playwright.config.ts
import { defineConfig } from '@playwright/test'

export default defineConfig({
  reporter: [['list'], ['@testorchestrator/playwright-reporter']],
})

A config that says reporter: 'html' becomes reporter: [['html'], ['@testorchestrator/playwright-reporter']]. Then run the tests with the key in the environment:

TO_API_KEY=<your workspace service key> npx playwright test
[testorchestrator] 26 results → TR-3 (run 3) · 16 matched, 10 unmatched · 18 passed, 2 failed, 2 skipped, 2 timed_out, 2 flaky_passed · run closed
  • The test’s id is its title path without the browser project: checkout.spec.ts › checkout › applies coupon. Each project (chromium, firefox) is the result’s configuration, matched to the Environment of the same name.
  • Retries are kept as attempts; a test that passed on a retry is flaky_passed. test.fail(), skips, tags and annotations are reported too.
  • On GitHub Actions the reporter sets the run key to <run id>-<run attempt> and adds the branch, commit, run number and a link to the workflow run. Elsewhere, set TO_RUN_KEY to an id that is new for every pipeline attempt.
  • Shards with the same run key land on one run, and no shard closes it: shards finish in any order and a run closes on the first upload that says complete, so with --shard the reporter uploads with complete: false. Close the run once every shard has uploaded — with an empty upload with complete=true from a final job (below) — or let it close by itself after the workspace’s auto-close delay. Merging blob reports with npx playwright merge-reports and this reporter in the merge config uploads once and closes the run.
  • Screenshots and traces: with attachments: 'upload' the files are uploaded and linked to the result; by default they are recorded by name.
  • Your test run is never failed by the upload. A refused upload is logged with what to do, and the results stay in testorchestrator-results.json. Set failOnError: true to fail the run instead. Without a key the reporter writes the file and uploads nothing.
OptionEnvironmentWhat it does
apiKeyTO_API_KEYThe service key. Never printed.
projectTO_PROJECTThe prj_ id, for an all-projects key.
runKey, runNameTO_RUN_KEY, TO_RUN_NAMEThe run to upload to, and the name of a new run (by default CI · <branch> · #<build>).
environmentTO_ENVIRONMENTOne Environment for every result, instead of the project names.
completeTO_COMPLETEClose the run after the upload: on by default, off when Playwright runs with --shard.
attachments—'names' (default), 'upload' or 'none'.
dryRunTO_DRY_RUNWrite the file, upload nothing.
failOnError—Fail the test run when the upload fails. Off by default.
projectUrl—Your project’s address in the app, to print a link to the run.

Every option, the mapping table and the reporter’s messages are in its README.

GitHub Action

testorchestratorhq/report-results uploads the files your test step wrote, in one step. It works out each file’s format, masks the key, prints the run and writes a job summary.

# .github/workflows/tests.yml (the steps of a job)
- run: pytest --junitxml=reports/pytest.xml
- name: Report to TestOrchestrator
  if: ${{ !cancelled() }}
  uses: testorchestratorhq/report-results@v1
  with:
    api-key: ${{ secrets.TO_API_KEY }}
    project: prj_8kd2hq0vxc1a
    files: reports/*.xml

if: ${{ !cancelled() }} runs the step when tests failed, so failures are reported. The same step reports Jest (npx jest --ci --reporters=default --reporters=jest-junit, files: reports/jest-junit.xml) and Playwright’s JSON report (files: playwright-report/results.json). reports/jest-junit.xml is where jest-junit writes when package.json sets "jest-junit": { "outputDirectory": "reports", "outputName": "jest-junit.xml" }, as the Action’s Jest example does. Complete workflows for all three are in the Action’s examples.

InputDefaultWhat it does
api-key, project, filesrequiredThe key (from a secret), the prj_ id, and glob patterns separated by new lines or commas.
formatauto.xml is JUnit XML; JSON with suites is a Playwright report; JSON with version and results is TestOrchestrator JSON, checked against the schema before it is uploaded.
run-key<run id>-<run attempt>Uploads with the same key land on one run; Re-run jobs gets a new one.
completetrueClose the run with the last file — only when every file was uploaded; otherwise the run is left open and the job summary says so. Set false in matrix and shard jobs and close the run from a final job.
fail-on-ingest-errortrueFail the step when a file cannot be read, an upload is refused or the API cannot be reached; false makes it a warning.
run-nameCI · <branch> · #<build>The name of a new run.
base-urlhttps://api.testorchestrator.comAPI host. Change it only to point the step at a test stack.
environment, tags, project-url—As in the request; project-url adds a link to the run.

Outputs: run-id, run-ref (TR-12), run-url (with project-url) and counts as JSON. Each file is one upload with an Idempotency-Key made from the run key, the file’s position, the upload’s fields and its content, so a retried request is answered from the first upload and nothing is written twice; Re-run jobs has a new attempt number and uploads to a new run.

Close the run from a final job

When several jobs report to one run — Playwright shards, a test matrix — each uploads with complete: false (the reporter does this by itself under --shard), and one job that runs after all of them closes the run with an empty file:

close-run:
  needs: test # the matrix or shard job
  if: ${{ !cancelled() }}
  runs-on: ubuntu-latest
  steps:
    - run: printf '' > close.json # no results: it only closes the run
    - uses: testorchestratorhq/report-results@v1
      with:
        api-key: ${{ secrets.TO_API_KEY }}
        project: prj_8kd2hq0vxc1a
        files: close.json
        complete: true

Jobs of one workflow run share the default run key, <run id>-<run attempt>. Without the final job the run closes by itself after the workspace’s auto-close delay.

TestOrchestrator JSON and its schema

For a runner neither of the above covers, write TestOrchestrator’s own format (format=to-results-json) — a version and a list of results, each with test_id, title and outcome (the fields are on Report automated results). The JSON Schema is published at https://testorchestrator.com/schemas/to-results.v1.json (draft 2020-12; a breaking change becomes v2), and @testorchestrator/results-schema checks a file the way the upload reads it:

npm install --save-dev @testorchestrator/results-schema
npx to-results-validate results.json
results.json: invalid (1 problem)
/results/0/outcome: "outcome" must be one of passed, failed, skipped, errored, timed_out, flaky_passed, inconclusive.

It exits 0 when every file is valid and 1 otherwise, with each problem as a JSON pointer, as the upload would answer; --json prints a machine-readable report. The package also exports the TypeScript types of the format. Upload a valid file with the Action or curl:

curl -sS --fail-with-body -X POST https://api.testorchestrator.com/api/v1/ingest/results \
  -H "Authorization: Bearer $TO_API_KEY" \
  -F file=@results.json \
  -F format=to-results-json \
  -F run_key=ci-1234-1 \
  -F complete=true

Troubleshooting

MessageWhat to do
not uploaded: no API keyThe key is not in the step’s environment. In GitHub Actions map the secret: env: { TO_API_KEY: '${{ secrets.TO_API_KEY }}' }.
401 API_KEY_UNAUTHORIZEDThe key is wrong, revoked or expired, or it is a personal key. Use a workspace service key.
400 PROJECT_REQUIRED / INVALID_PROJECT_IDPass the project’s prj_ id, never the number in the app’s URL.
403 KEY_PROJECT_MISMATCHThe key belongs to another project or workspace, or the prj_ id is wrong.
409 RUN_CLOSEDThe run for this run key is closed. Shards and matrix jobs: upload with complete: false and close the run once from a final step; otherwise use a run key per CI attempt.
already uploaded (Idempotent-Replay)The same file was uploaded before with the same key; nothing was written twice.
… retried 1 timeNothing: the reporter and the Action waited (for 429, as Retry-After said) and sent it again. They retry 429, 5xx, 409 BUSY and the network errors that can heal (ECONNREFUSED, ECONNRESET, ETIMEDOUT, EAI_AGAIN) up to 5 times; ENOTFOUND, EHOSTUNREACH and ENETUNREACH fail after one attempt. Each attempt is printed with TO_DEBUG=true (reporter) or in the Action’s debug log.
closing run TR-12The close step worked: an upload with no results and complete=true. Without complete, an empty upload shows the warning The file holds no test results. instead.

Every refusal and its fix: API keys → Troubleshooting.

Was this page helpful?Tell us what’s missingOpens an email to support with the page already named.