Blog · Product

Report automated results from CI in one step

Uploading results used to mean a curl command per CI system. Now there is a reporter for Playwright, a GitHub Action for every runner that writes JUnit XML, and a published schema for everything else.

Three ready-made ways to report automated results: @testorchestrator/playwright-reporter (one line in playwright.config.ts), the testorchestratorhq/report-results GitHub Action (JUnit XML from pytest, Jest, JUnit or Cypress, and Playwright JSON), and to-results.json with a published JSON Schema and the to-results-validate CLI. Results land on a test run, matched to your cases; anything unmatched waits on the run's Unmatched tab. Setup: CI integrations.

Why this exists

Reporting automated results to TestOrchestrator has always been one HTTP request: send the results file, get a test run back. In practice every team wrote that request themselves — a curl command with a dozen form fields, a run key that had to include the CI attempt, an if: always() so failures were reported too. It worked, and every copy was slightly different.

So we packaged the request three ways. Pick the one that fits how you run tests.

Playwright: a reporter

npm install --save-dev @testorchestrator/playwright-reporter
// playwright.config.ts
export default defineConfig({
  reporter: [['list'], ['@testorchestrator/playwright-reporter']],
})

With TO_API_KEY set, the end of npx playwright test prints where the results went:

[testorchestrator] 26 results → TR-3 (run 3) · 16 matched, 10 unmatched · 18 passed, 2 failed, 2 skipped, 2 timed_out, 2 flaky_passed · run closed

That line comes from our own end-to-end test: a fresh project that installs the packed reporter and runs a suite with a failing test, a test that only passes on its retry, a skip, a test.fail(), a parametrised title, a timeout and two browser projects. Each browser project is a configuration of the same test, so chromium and firefox results group under one case.

The reporter maps every test the way an upload of Playwright's own json report is mapped. We checked that on the server: the same Playwright run, uploaded once through the reporter and once as Playwright JSON, produced the same 26 stored results with the same outcomes, cases, configurations, attempts and failures. The one difference is cosmetic: the reporter's format trims text, so a test's output excerpt loses its final newline.

On GitHub Actions it sets the run key from the run id and attempt and adds the branch, commit and a link to the workflow run. Sharded runs share one run, and no shard closes it: shards finish in any order, so the run is closed once, from a final job, after every shard has reported. With attachments: 'upload' it uploads attachments such as screenshots and logs and links them to the result.

Any runner on GitHub Actions: an Action

- run: pytest --junitxml=reports/pytest.xml
- if: ${{ !cancelled() }}
  uses: testorchestratorhq/report-results@v1
  with:
    api-key: ${{ secrets.TO_API_KEY }}
    project: prj_8kd2hq0vxc1a
    files: reports/*.xml

The Action takes the files your test step wrote — JUnit XML from pytest, Jest (with jest-junit), Maven, Gradle, Vitest or Cypress, or a Playwright JSON report — works out each file's format, masks the key in the log, and writes a job summary with the run and its totals. Its repository has complete workflows for Playwright, pytest and Jest, each tested with its real test runner and reporter.

Everything else: a schema

For a runner neither of those covers, write TestOrchestrator's own format. The JSON Schema is published at /schemas/to-results.v1.json, and @testorchestrator/results-schema checks a file before you upload it:

npx to-results-validate results.json
results.json: invalid (1 problem)
/results/0/outcome: "outcome" must be one of passed, failed, skipped, errored, timed_out, flaky_passed, inconclusive.

It points at the same place the upload itself would. We measure it against the server's parser on 244 files — the parser's own fixtures, our edge cases and an independent reviewer's probes: it agrees on 243 of them, and the one difference is listed in the package's docs.

What it deliberately does not do

  • It does not re-run tests. Retries are your runner's; the reporter records each attempt and marks a pass after a failed retry as flaky_passed.
  • It does not guess at flakiness. There is no score and no hidden rerun — only what happened in this run.
  • It does not create cases. A test no case matches stays on the run's Unmatched tab until you link it; linking writes its id as the case's automation key.
  • It does not fetch files from your CI machine. Attachments are recorded by name, or uploaded by the reporter when you ask it to.
  • It does not fail your test run because of an upload. The reporter logs a refusal with what to do and keeps the results file; you can opt in to failing with failOnError.

Get started

You need a workspace service key — a Tenant Admin creates one on Settings → API keys → Workspace service keys — stored as the CI secret TO_API_KEY. The steps for all three are on CI integrations; the request they send is described on Report automated results.

Frequently asked

Do I need the reporter if I already upload with curl?

No. The reporter, the Action and curl all send the same upload, so runs, matching and the Unmatched tab behave the same. The reporter and the Action add retries that honour Retry-After, an Idempotency-Key per upload, CI facts and a readable summary.

Can an upload problem fail my test run?

Not with the reporter's defaults: a refused upload is logged with what to do and the results stay in testorchestrator-results.json. Set failOnError to true if you want the run to fail. The Action fails its own step by default (fail-on-ingest-error), which you can turn off.

Does it detect flaky tests?

It records what your runner did. A test that failed and then passed on one of the runner's own retries is stored as flaky_passed with every attempt kept. Nothing is re-run and nothing is guessed.

What happens to tests that match no case?

They are kept on the run's Unmatched tab with their test id. Link to case files them on a case and writes the id as its automation key, so the next run matches by itself. No case is ever created automatically.

Which test runners work?

Playwright through the reporter or its JSON report; anything that writes JUnit XML through the Action, including pytest (--junitxml), Jest (jest-junit), Maven Surefire and Gradle, Vitest and Cypress (mocha-junit-reporter); and any runner you report from yourself in to-results.json.

Product

Connect your AI agent to TestOrchestrator with MCP

Practice

Exploratory testing that actually gets recorded

Practice

How small QA teams set up test management from scratch (in an afternoon)

Try it on the free plan

5 users, 2 projects, 200 active test cases, 1 GB. No credit card.