Three ready-made ways to report automated results: @testorchestrator/playwright-reporter (one line in playwright.config.ts), the testorchestratorhq/report-results GitHub Action (JUnit XML from pytest, Jest, JUnit or Cypress, and Playwright JSON), and to-results.json with a published JSON Schema and the to-results-validate CLI. Results land on a test run, matched to your cases; anything unmatched waits on the run's Unmatched tab. Setup: CI integrations.
Why this exists
Reporting automated results to TestOrchestrator has always been one HTTP
request: send the results file, get a test run back. In practice every team
wrote that request themselves — a curl command with a dozen form fields, a run
key that had to include the CI attempt, an if: always() so failures were
reported too. It worked, and every copy was slightly different.
So we packaged the request three ways. Pick the one that fits how you run tests.
Playwright: a reporter
npm install --save-dev @testorchestrator/playwright-reporter
// playwright.config.ts
export default defineConfig({
reporter: [['list'], ['@testorchestrator/playwright-reporter']],
})
With TO_API_KEY set, the end of npx playwright test prints where the
results went:
[testorchestrator] 26 results → TR-3 (run 3) · 16 matched, 10 unmatched · 18 passed, 2 failed, 2 skipped, 2 timed_out, 2 flaky_passed · run closed
That line comes from our own end-to-end test: a fresh project that installs the
packed reporter and runs a suite with a failing test, a test that only passes on
its retry, a skip, a test.fail(), a parametrised title, a timeout and two
browser projects. Each browser project is a configuration of the same test, so
chromium and firefox results group under one case.
The reporter maps every test the way an upload of Playwright's own json
report is mapped. We checked that on the server: the same Playwright run,
uploaded once through the reporter and once as Playwright JSON, produced the
same 26 stored results with the same outcomes, cases, configurations, attempts
and failures. The one difference is cosmetic: the reporter's format trims text,
so a test's output excerpt loses its final newline.
On GitHub Actions it sets the run key from the run id and attempt and adds the
branch, commit and a link to the workflow run. Sharded runs share one run, and
no shard closes it: shards finish in any order, so the run is closed once, from
a final job, after every shard has reported. With attachments: 'upload' it uploads attachments such as screenshots and
logs and links them to the result.
Any runner on GitHub Actions: an Action
- run: pytest --junitxml=reports/pytest.xml
- if: ${{ !cancelled() }}
uses: testorchestratorhq/report-results@v1
with:
api-key: ${{ secrets.TO_API_KEY }}
project: prj_8kd2hq0vxc1a
files: reports/*.xml
The Action takes the files your test step wrote — JUnit XML from pytest, Jest (with jest-junit), Maven, Gradle, Vitest or Cypress, or a Playwright JSON report — works out each file's format, masks the key in the log, and writes a job summary with the run and its totals. Its repository has complete workflows for Playwright, pytest and Jest, each tested with its real test runner and reporter.
Everything else: a schema
For a runner neither of those covers, write TestOrchestrator's own format. The
JSON Schema is published at
/schemas/to-results.v1.json, and
@testorchestrator/results-schema checks a file before you upload it:
npx to-results-validate results.json
results.json: invalid (1 problem)
/results/0/outcome: "outcome" must be one of passed, failed, skipped, errored, timed_out, flaky_passed, inconclusive.
It points at the same place the upload itself would. We measure it against the server's parser on 244 files — the parser's own fixtures, our edge cases and an independent reviewer's probes: it agrees on 243 of them, and the one difference is listed in the package's docs.
What it deliberately does not do
- It does not re-run tests. Retries are your runner's; the reporter records
each attempt and marks a pass after a failed retry as
flaky_passed. - It does not guess at flakiness. There is no score and no hidden rerun — only what happened in this run.
- It does not create cases. A test no case matches stays on the run's Unmatched tab until you link it; linking writes its id as the case's automation key.
- It does not fetch files from your CI machine. Attachments are recorded by name, or uploaded by the reporter when you ask it to.
- It does not fail your test run because of an upload. The reporter logs a
refusal with what to do and keeps the results file; you can opt in to failing
with
failOnError.
Get started
You need a workspace service key — a Tenant Admin creates one on Settings →
API keys → Workspace service keys — stored as the CI secret TO_API_KEY. The
steps for all three are on CI integrations;
the request they send is described on
Report automated results.
Frequently asked
Do I need the reporter if I already upload with curl?
No. The reporter, the Action and curl all send the same upload, so runs, matching and the Unmatched tab behave the same. The reporter and the Action add retries that honour Retry-After, an Idempotency-Key per upload, CI facts and a readable summary.
Can an upload problem fail my test run?
Not with the reporter's defaults: a refused upload is logged with what to do and the results stay in testorchestrator-results.json. Set failOnError to true if you want the run to fail. The Action fails its own step by default (fail-on-ingest-error), which you can turn off.
Does it detect flaky tests?
It records what your runner did. A test that failed and then passed on one of the runner's own retries is stored as flaky_passed with every attempt kept. Nothing is re-run and nothing is guessed.
What happens to tests that match no case?
They are kept on the run's Unmatched tab with their test id. Link to case files them on a case and writes the id as its automation key, so the next run matches by itself. No case is ever created automatically.
Which test runners work?
Playwright through the reporter or its JSON report; anything that writes JUnit XML through the Action, including pytest (--junitxml), Jest (jest-junit), Maven Surefire and Gradle, Vitest and Cypress (mocha-junit-reporter); and any runner you report from yourself in to-results.json.