Tests and Scans on the Passport
What a pipeline tells the Studio about its tests and vulnerability scans, what the Studio keeps of it — counts and a digest, never the findings — when a result becomes evidence on a work package’s or a release’s passport, and how to report from a pipeline the Studio generates, one you wrote yourself, another CI system or a tool of your own.
What is recorded, and what is not
A pipeline reports a result for one commit, with the repository’s deploy token. The Studio keeps:
| Kind | Categories |
|---|---|
| Tests | Unit tests, integration tests, end-to-end tests |
| Scans | Dependency scan, static analysis (SAST), secret scan, container image scan, infrastructure scan, dynamic scan (DAST) |
A pipeline can also report what it published — a container image, an npm, NuGet, Maven or PyPI package, an archive or a function bundle — by its name, its digest and the registry it went to. A release’s passport lists what was published from its cut and, for an image, whether a container scan at that commit scanned an image of the same digest. That agreement is shown, never enforced.
Where you see it
When a result becomes evidence
A reported result is held against its commit for 180 days. It becomes permanent evidence, in the artefact’s ledger, when the Studio reads it at one of these moments:
- a Definition of Done item that reads the pull request is assessed (the package’s results at that head);
- Approve & merge (at the head merged);
- release approval and release (each package’s approved head, into the package’s ledger, and each repository’s head in the release’s cut, into the release’s own ledger).
A result reported after a release was approved still counts when it is released. A result that none of these moments read is not evidence.
A delivery policy can name the controls a test or scan is evidence for, so a release’s passport can show a control as satisfied by a check that was met at a promotion. What it rests on is still the counts, never the findings. See The Delivery Policy.
Which runs to report from
Report from your pull requests’ runs and from the runs on pushes to your main branch. A pull request’s results are its work package’s evidence, at its heads. The release’s own results come from the run on main at the commit the release is cut at: the container image, the infrastructure and the dependency set that ship are built from that commit, not from any one pull request, and it is the only run that covers commits made outside the Studio.
Bound to its run
A result is what the pipeline says. The deploy token is a repository secret, so anyone who can change a workflow that sees it could report any counts. A result therefore counts for a delivery policy only when it is bound to the workflow run that made it:
permissions: id-token: write. The report step asks GitHub for a token that names the repository, the workflow run and the commit; the Studio checks its signature against GitHub’s published keys, and that it names this repository and this commit. The token is never stored.Only GitHub Actions gives a run such a token. A result from another CI system is recorded and shown, and is not bound.
Pipelines the Studio generates
A GitHub Actions pipeline generated from Repositories → DevOps → Create runs and reports the tests and scans that fit each part of the repository. The Studio decides what applies from the files it reads, never from what the pipeline says. Generate the pipelines again to take this into a pipeline made earlier; the passport check in it is unchanged, so it still reads Passport enforced meanwhile.
| Category | Run by the Studio’s pipeline | Where |
|---|---|---|
| Dependency scan | Trivy, installed by the scan job from its release and checked against a pinned SHA-256 | Every part with code to build. A part with no lockfile or manifest Trivy can read is reported as not completed, never as a clean scan |
| Secret scan | Trivy’s secret scanner | Every part |
| Infrastructure scan | Trivy for Terraform and SAM; Checkov for Serverless and Bicep | Parts with infrastructure as code. AWS CDK and Pulumi are code, and are not scanned: report one yourself where a policy asks |
| Container image scan | The image is built in the scan job, then scanned with Trivy | Only parts with a Dockerfile |
| Static analysis | By the repository’s setting: automatically CodeQL on a public repository and nothing generated on a private one, or CodeQL always (with GitHub Advanced Security), or none. On a private repository, add your own SAST tool, such as Semgrep run under your own licence | Parts with code to build |
| Unit tests | The stack’s own test command, where its tooling writes a report with no new dependency: pytest, Maven, Gradle, dotnet test, and Vitest where package.json’s test script is just vitest or vitest run | Go, Jest and other Node runners report nothing until you set your own command (below) |
Integration tests, end-to-end tests and a dynamic scan are never generated: the files cannot say how to run them. Set your own tool for them, or report them from where they run.
report-checks collects each job’s reports and sends the results, bound to the run. It runs no code from the repository and no tool of your own. The build and scan jobs, where the repository’s code and your own commands run, never hold the deploy token.v tag, the same tag the release’s cut is read from. New findings are counted against the scan reports of the base commit’s own push run, which GitHub keeps for 90 days by default. A first release has no tag and so no base..trivyignore, and .checkov.yaml and checkov:skip comments. What they suppress is not counted, so a change can lower its own scan’s counts: the suppression is part of the change to review. The scan job names the files in effect in its log.Your own tools
On Repositories → DevOps → Create, Test and scan tools sets, for each category, what the generated pipelines run. Only the studio owner changes it, because it changes what runs on every pull request; everyone in the studio can read it. It takes effect when the pipelines are generated again.
| Tool | What the pipeline does |
|---|---|
| The Studio’s tool | Runs the tool in the table above. For static analysis, choose automatic (CodeQL on a public repository, nothing on a private one) or CodeQL always (which needs GitHub Advanced Security on a private repository) |
| Your own command | Runs your command in the build or scan job, in the part’s directory, then reports the file it writes, in the format you name (JUnit or TRX for tests; SARIF, Trivy JSON or npm audit JSON for scans) |
| Reported by your own tool elsewhere | Runs nothing for the category. The proposal and the pipeline description say who reports it, such as a hosted scanner, which reports with the deploy token from wherever it runs |
| Nothing | Runs and reports nothing for the category. A policy that asks for it finds nothing reported |
pipx run semgrep==1.180.0 scan --config p/default --metrics=off --sarif --output semgrep.sarif ., report path semgrep.sarif, format SARIF) or as a tool that reports from elsewhere. The Studio neither provides nor licenses Semgrep. CodeQL with GitHub Advanced Security is the other route. Your command runs with the job’s read permission only, never with the deploy token. It runs on every pull request, so any code of the pull request it runs can read the secrets you name for it: name only what the tool needs, scoped as narrowly as the tool allows.
From a pipeline you wrote yourself, or another CI system
The same route takes a result from any pipeline. Copy scripts/report-check-result.mjs into the repository — it needs Node.js and nothing else — and run it after each test suite or scanner, with these set in its environment:
PW_API_URL, PW_STUDIO_ID and PW_REPO_ID — the API address, your studio and the repository connection (its id is on the Repositories page).PW_DEPLOY_TOKEN — this repository’s deploy token, from a secret.PW_COMMIT — the commit tested. On a pull request’s run, the pull request’s head, not the temporary merge commit GitHub builds.node scripts/report-check-result.mjs --category unit-tests --tool jest --format junit junit.xml --coverage coverage/lcov.infoIt reads JUnit, TRX, SARIF, Trivy JSON and npm audit --json; with --format other and --outcome it records the digest alone. --diff-base adds coverage of new code and, with --baseline, findings new since the base; --publish reports a published artefact by its digest. Report a failed run too: a failure is evidence. The step reports; it does not judge. The full set of options and what each message means are in the operator guide, docs/operations/passport-pipeline-check.md.
Reporting adds nothing to the passport check: a pipeline you wrote yourself is still not held before a deploy unless it carries the same check job. See Wire a Pipeline to the Passport.