Review application A.4.4
Running the quality checks
On this page
This page explains how to run, maintain and troubleshoot the quality checks of the application and the engine: the commands, the supported setups, CI, the module catalog and its baselines. Start in the application repository. To read the results, see A4.3 Code quality.
Choose a command
| Command | Purpose | Requirements |
|---|---|---|
make quality-fast-container |
Catalog, architecture, Python/JS/CSS lint, template/config syntax, vendor checks and focused quality/contract tests | Docker Compose v2 and an engine checkout |
make quality |
Full app, browser and engine suites; PostGIS publication and producer/consumer checks; coverage sanity; duplicated blocks; report and code map. CI runs this command too | Docker Compose v2 and an engine checkout |
make quality-fast |
Same fast checks using installed local tools | Python 3.13, Node 22.x at least 22.13, requirements.txt and requirements-dev.txt |
make quality-down |
Remove the dedicated quality Compose stack and its disposable data | Docker Compose v2 |
A fast pass ends by naming make quality. It does not execute the complete browser or application suite, so it establishes neither coverage nor publication safety. Dependency audits and CodeQL are separate CI checks; make quality does not run those network services.
Even a full pass leaves some things unchecked. There is no type checking, no end-to-end, rendering or visual test in a real browser and no performance budget. Mutation testing is the engine's own check, on a sample of its result validation.
Supported local setup
Docker: the complete reproducible path
Keep a quality-enabled engine checkout beside the application:
workspace/
stop_sync_osm_atlas/
engine/
quality/modules.yml
From stop_sync_osm_atlas/:
make quality-fast-container
make quality
make quality-down
For another engine location, set ENGINE_DIR, in .env or on the command line:
make quality ENGINE_DIR=/absolute/path/to/engine
Every tool finds the engine the same way: ENGINE_DIR if set (relative paths start at the application checkout), else engine/ inside the application, else ../engine. The Python tools share engine_dir() in backend/version.py; Docker Compose also reads ENGINE_DIR from .env.
Dockerfile.quality supplies Python 3.13, Node 22, PDF rendering libraries, app tools and an isolated engine virtual environment. compose.quality.yml bind-mounts app source and the engine checkout, keeps engine source read-only, and gives the runner its image's Node dependencies. The dedicated osm-atlas-quality project provisions PostGIS 16/3.4 with disposable in-memory database storage. It does not share production volumes or use the deployment's database credentials. Full publication tests can replace test data, so this separation is essential.
Both container commands build the image. Rebuild after requirement, lockfile or engine dependency changes; changing a bind-mounted source file does not update installed packages. Preflight rejects an installed Python package that does not satisfy the requirement files.
A full run clears quality/raw/ before collecting evidence, so a failed run cannot inherit an earlier passing result, then writes outputs back into the app checkout. Save any previous run you need to investigate before starting another full run. Stop editing either repository while the final run is in progress: source changes invalidate its fingerprints, including changes to documentation, configuration and tests.
Native development
Use Python 3.13 and Node 22.x, version 22.13 or later. The runner checks those versions; an arbitrary newer Node major is not the supported toolchain. Native PDF tests also need the platform's Pango libraries. Prefer the container when those libraries are unavailable.
python3.13 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt -r requirements-dev.txt
npm ci
make quality-fast PYTHON=python
requirements-dev.txt holds the test tools and the exact analyser versions the quality runner uses. It includes Hypothesis and JSON Schema because ordinary pytest collection includes property and quality-schema tests. See A3.1 Dependencies and builds.
Do not install osm-public-transport-matcher into the app interpreter. Full verification checks this boundary explicitly. To run the full command natively, create a separate engine environment and supply a database dedicated to tests:
python3.13 -m venv ../engine/.venv
../engine/.venv/bin/python -m pip install '../engine[swiss,gtfs,quality]'
TEST_POSTGRES_URI=postgresql+psycopg://test:test@localhost:5432/transport_test \
python scripts/run_quality.py full --engine-python ../engine/.venv/bin/python
The example assumes that disposable PostGIS database already exists. It is not a command for connecting to a production database. For engine-only work, use the engine's own quality guide, engine/quality/README.md.
CI and repository setup
The Required quality job of the Quality workflow (.github/workflows/tests.yml) runs on pull requests, pushes to main and manual dispatch. It checks out the engine commit pinned in engine.ref into engine/ and runs make quality, so CI builds the same image and disposable PostGIS database as a local run. The app suite runs without an engine installation; the engine suite uses the image's separate engine interpreter, which the app's engine import contract also runs as a subprocess.
Repository administrators must configure branch protection or rulesets requiring Required quality, Dependency security audit, and the applicable Security analysis checks for Python and JavaScript/TypeScript. Confirm the exact check names after the first run.
Committing workflow files does not configure repository protection or publish engine changes. Update engine.ref as part of a reviewed cross-repository contract change.
The required workflow always uploads the quality-<app SHA> artifact with the raw evidence, including failed-run outputs, and keeps it for 30 days. Inspect job output alongside the artifact: missing outputs may indicate that collection itself failed. On a push to main the job also publishes quality/evidence.json for the dashboard (How the evidence reaches the site). No workflow commits or pushes.
The Periodic quality workflow (.github/workflows/deep-quality.yml) runs production Python dependency auditing and an npm audit on pull requests, weekly and on manual invocation. The npm audit fails at high severity; the Python audit fails on its reported vulnerabilities. Its dependency-security-<app SHA> artifact preserves both audit reports. Dependency changes must update the corresponding requirement or lockfile and any vendored assets, then rerun behavioural tests.
CodeQL (.github/workflows/codeql.yml) analyses app Python and first-party browser JavaScript on pull requests, pushes to main and its weekly schedule. Vendor files are excluded. Findings are available through GitHub code scanning; a successful job is not itself a per-module security score. The report has no security column: read the audit and CodeQL results directly.
The standalone engine has its own production dependency audit and Python CodeQL workflow. Configure its required branch checks independently, as its quality guide describes. An app audit does not establish that the engine's separate dependency closure was scanned.
How the evidence reaches the site
The dashboard shows the evidence of the commit the site runs. Nothing is committed for it, and nobody runs a command before a deployment.
- On every push to
main, the Required quality job uploadsquality/evidence.jsonas the assetquality-<commit>.jsonof the GitHub releasequality-data, also when a check failed. Anybody can download a release asset; a CI artifact needs a GitHub token, even from a public repository. The same step uploads the documentation's rendered Mermaid diagrams asdiagrams-<commit>.tar.gz, for the PDF of a site without Node (A2.5). Of each kind, the job keeps the files of the newest 60 commits and of every commit a Git tag names, so a release tag deployed weeks later still finds its files. - The application finds its own commit with
app_revision()inbackend/version.py: theAPP_REVISIONenvironment variable, else aREVISIONfile beside the code, else the.gitfolder of its checkout. - On the first visit to the dashboard,
backend/services/quality_report.pydownloads the file of that commit, checks its shape and keeps it in.cache/quality/and in memory. The file of a commit never changes, so GitHub is asked once for each deployed commit. - When GitHub has no file for the commit, the dashboard says so and names the commit, and the application asks again after five minutes. This is the normal state while CI is still running, and for a commit that was never pushed to
main.
A file quality/evidence.json in the checkout wins over the download. make quality and python scripts/quality.py map write it, Git ignores it, and the dashboard marks it as a local run. Delete it to see the CI evidence again.
The server therefore needs:
| Requirement | Why |
|---|---|
The code is a Git clone, or the deployment writes the commit ID into a REVISION file or the APP_REVISION variable |
Otherwise the application does not know which evidence to ask for |
Outgoing HTTPS to github.com and to the githubusercontent.com host that GitHub redirects downloads to |
To download the file, and the diagrams of the documentation PDF, once for each deployed commit |
A writable .cache/ folder |
To keep the file across restarts; without it each worker downloads it once after starting |
An image built from the dockerfile contains no .git: set APP_REVISION when starting it. The development Compose setup mounts the checkout, .git included.
Maintain the module catalog
| File | Who maintains it | Meaning |
|---|---|---|
quality/modules.yml |
Maintainers | Module responsibility, criticality, owner, invariants, source/test paths, entry points, documentation and declared dependencies/contracts |
quality/modules.schema.json |
Quality maintainers | Validated catalog structure; the catalog uses JSON-compatible YAML |
quality/exclusions.json |
Maintainers, with reasons | Shared source exclusions and additional clone exclusions |
quality/baseline.json |
Reviewed changes | The accepted duplicated blocks |
quality/report.schema.json |
Quality maintainers | Structure of the generated report |
quality/decisions.md |
Maintainers | Why ownership, contracts, evidence and the baseline work this way |
Every inventoried production source has exactly one ownership module. Source inventory combines tracked and new nonignored Git files, configured source roots and explicit files. It recognises Python, JS/MJS/CJS, CSS, templates, SQL, shell, YAML, JSON and TOML; explicit catalog entries include extensionless deployment files and requirement lists. Symlinks are not followed as source files.
When the patterns of several modules match a file, the pattern with the longest fixed start wins: a file named in full beats the pattern of its directory, and a deeper directory beats a shallower one. A directory therefore has a default owner, and the catalog names only the files that belong elsewhere:
| Pattern | Owner |
|---|---|
backend/** |
app.query-api, the default for the backend |
backend/jobs/**, backend/services/pipeline*.py |
app.pipeline-orchestration: a deeper directory and a narrower pattern |
backend/blueprints/docs.py |
app.docs-delivery: a file named in full |
A new file in such a directory needs no catalog change unless the default owner is wrong for it. A file in a directory no pattern covers, or matched equally well by two modules, fails the check.
Tests may support multiple modules and are associated through each module's test patterns. Contract nodes connect producer and consumer owners; a contract does not assign a second owner to their implementation files. Documentation paths are obligations, not an instruction to document every private helper.
The source exclusions cover generated evidence and documentation, the built CSS bundle, vendored/installed dependencies, runtime data, bytecode, the engine's mutation worktree, and tests/fixtures that are modelled as evidence. Migrations remain owned source, but their immutable steps are excluded from clone detection. Each repository can add explicit exclusions to its catalog; always explain the category being excluded.
Runtime coverage denominators are narrower than source ownership: app backend/**/*.py, engine src/**/*.py, and every first-party static/js/**/*.js. CSS, templates, deployment files and build tooling remain owned and checked, but are not assigned runtime coverage percentages.
When adding or moving code:
- Choose the existing responsibility owner, or add a module with a coherent responsibility and failure boundary.
- Add a source pattern only when the directory's default owner is wrong for the file. Update test patterns and required explanation/reference/operations documents. Set criticality with the rubric.
- Declare stable entry points, dependencies and cross-repository contracts. The dashboard flags code dependencies that the catalog does not declare.
- Run
check, relevant focused tests andmake quality-fast; finish with full evidence appropriate to the change.
Command-line reference
Run python scripts/quality.py --help for all options. Examples below assume the supported environment; each command finds the engine like every other tool, and --engine-dir PATH overrides it.
| Command | Result |
|---|---|
python scripts/quality.py check |
Validates the catalog schema, single ownership, documentation/test references, entry points and enforced architecture boundaries |
python scripts/quality.py coverage |
Validates current raw coverage against source fingerprints and all expected runtime files; does not execute tests |
python scripts/quality.py debt |
Runs jscpd, writes the duplicated blocks to raw/debt.json and fails on one that the baseline does not accept |
python scripts/quality.py report |
Reads the evidence of the last full run and writes the report and the code map to evidence.json; does not recollect tests |
python scripts/quality.py map |
Reads the source of both repositories and writes the code map alone to evidence.json, in a few seconds |
debt --write-baseline is an explicit baseline-maintenance operation, described in Duplicated blocks and the baseline. It must not be added to CI or to an automatic fix command.
The lower-level runner is python scripts/run_quality.py fast|full|syntax. fast and full accept --engine-dir; full also uses --engine-python (defaulting to ENGINE_PYTHON or /opt/engine-venv/bin/python). syntax parses Jinja templates and configured YAML/JSON without importing the app. Prefer the Make targets for ordinary use.
Evidence files and freshness
The full runner records in raw/checks.json both repositories' revisions and source fingerprints, the Python, Node and analyser versions, and each check's command, exit status and duration. It continues collecting independent checks after failures and returns nonzero if any required check failed. Report generation is useful even after a test failure; producing a report is not equivalent to passing the run. To look at a local run, open the dashboard of a local site: it shows quality/evidence.json when the file exists (How the evidence reaches the site).
Artifact, relative to quality/ |
Contents |
|---|---|
evidence.json |
What the dashboard shows, in one file: report (app/engine revisions, generation time, completeness, tools, checks, each module's measurements and each measured file's coverage; null after map) and code_map (modules with their catalog entries, files, functions, links counted by kind, each module's names used from other modules, routes, blind spots, unused functions and duplicated blocks) |
raw/checks.json |
Full-run manifest and check results |
raw/fast-checks.json |
Fast-run manifest; does not certify earlier full-run evidence |
raw/app-tests.xml, raw/engine-tests.xml |
Pytest JUnit test outcomes |
raw/app-python-coverage.json, raw/app-python-coverage.xml |
App line/branch coverage in JSON and XML |
raw/engine-python-coverage.json, raw/engine-python-coverage.xml |
Engine line/branch coverage in JSON and XML |
raw/javascript-tests.json |
Jest suites, individual assertions and outcomes |
raw/javascript-coverage.json, raw/lcov.info |
Browser production coverage: Istanbul statements, functions and branches keyed by source path, and LCOV for other coverage viewers |
raw/jscpd/jscpd-report.json, raw/debt.json |
Raw clone-detector output and the normalised duplicated blocks |
raw/python-audit.json, raw/npm-audit.json |
Separate dependency-audit workflow outputs, when copied/published with the evidence |
raw/engine-python-audit.json |
Standalone engine production dependency audit, when copied/published with the evidence |
Analyser formats retain their own details; the report is not a replacement for their raw reports. Coverage paths from another checkout are normalised for module association, and pytest class-based test names are attributed to their source test file. A shared test can count as evidence for multiple modules; module counts are not additive suite totals.
report refuses evidence that was not measured on the current source: the full-run manifest must record the same app and engine revisions and the same source fingerprints. Fingerprints hash tracked and new nonignored inputs while excluding generated quality/documentation outputs. Editing a document or test after collection therefore makes report fail; run make quality again. There is no stale state to show. Never hand-edit generated JSON.
Missing declared test files or skipped scenarios never become a clean module pass. The docs count measures required catalog slots, not the editorial quality or exhaustive correctness of those documents. Read the source documents when reviewing behaviour and invariants.
Coverage sanity
JavaScript tests use an instrumented browser-script loader. Coverage includes shared scripts and never-loaded first-party files, which correctly contribute zero hits. npm run test:coverage runs Jest and then scripts/check-js-coverage.cjs; that check requires every production JS file and actual executed statements and branches. Use A4.2 JavaScript tests when adding tests.
The full coverage sanity gate additionally requires all three app/engine/JS reports, matching source fingerprints, Python branch coverage, complete runtime file denominators, executed production statements in each report and valid count bounds. A green test process with broken instrumentation or missing files must fail this gate.
These are correctness checks on the evidence, not thresholds: no coverage percentage fails a run. The report keeps the line and branch coverage of each measured file beside the module totals, so the dashboard can colour a file by its own coverage.
Coverage measures execution, not correctness. A high percentage does not replace publication rollback, invalid-input, concurrency or contract scenarios. For a one-off look at backend coverage, run python -m pytest tests --cov=backend --cov-branch --cov-report=term-missing; it does not produce a supported quality report.
Duplicated blocks and the baseline
The clone gate uses jscpd at a minimum of 10 lines and 70 tokens. quality/baseline.json lists each accepted clone by its identity and its files; a clone that is not in it fails the gate. Unused code is not a gate: the code map marks functions that no production code names, and the dashboard lists them.
Baseline identities include content and participating files. Distinct physical occurrences are counted: adding another identical copy within an already baselined file pair still creates a new finding. Ordinary line movement does not change the content identity, but editing the code inside an accepted clone does: the edited clone is reported as new and its baseline entry matches nothing. Deleting one clone does not buy permission for a different new clone.
When the gate reports a new finding, read both copies in raw/debt.json and decide:
- Refactor when both copies do the same job in one repository: move the shared code into one function and let the finding disappear.
- Accept when the duplication is intentional, such as the bundle contract validated independently by the app and the engine, or when an accepted clone only changed content: re-record the baseline as below.
To re-record the baseline after removing or accepting findings, run:
python scripts/quality.py debt
python scripts/quality.py debt --write-baseline
git diff -- quality/baseline.json
Review the resulting JSON, confirm it reflects intended removals, and rerun the normal debt command. Do not refresh the baseline merely to make an unexpected failure disappear. Producer/consumer validation can intentionally duplicate a wire contract across independent repositories; sharing app/engine implementation imports would violate the boundary. Review the responsibility before extracting an abstraction.
Code map
python scripts/quality.py map parses the source of both repositories statically without running tests or needing a database, joining module ownership, files, functions, Python imports and calls, browser-script globals, template script order, requests to Flask routes, and declared contracts into quality/evidence.json (plus any findings from the latest debt run). See A4.3 Code quality for how the static graph, cleanup candidates, boundary checks and parser blind spots are defined.
Troubleshooting and review workflow
| Symptom | What to check |
|---|---|
| Unsupported Python/Node or missing package | Use the supported versions with requirements.txt and requirements-dev.txt, or rebuild/run the container |
| Engine catalog missing | Set ENGINE_DIR to a quality-enabled checkout and update engine.ref |
App interpreter contains osm_public_transport_matcher |
Recreate an isolated app environment; use ENGINE_PYTHON for engine tests |
| PostGIS test missing/skipped | Use make quality or provide a dedicated TEST_POSTGRES_URI; SQLite does not establish publication safety |
| Coverage all zero or missing JS files | Use load-browser-script, keep production roots included in Jest, and run the coverage sanity command |
| Unexpected console/error or leaked timer | Fix the cause or lifecycle cleanup; assert expected errors only within their test, as shown in the JS guide |
| New duplicated block | Inspect raw/debt.json and both places before considering a reviewed baseline change |
| Report remains partial after passing tests | The tests or the coverage of a module were not collected or are invalid; the module table shows which |
report finds no evidence of the current source |
A file of either repository changed after collection; rerun make quality once inputs are stable |
| Dashboard has no results for its commit | CI is still running or stopped before the report, the commit is not on main, or the server cannot reach GitHub (How the evidence reaches the site) |
| Dashboard does not know its commit | The code is no Git clone: write the commit ID into a REVISION file or set APP_REVISION |
| Dashboard shows a local run | Delete quality/evidence.json to see the CI evidence of the commit |
Before requesting review, follow the checklist in Contributing.
Code
| File | Role |
|---|---|
Makefile |
The quality* targets |
scripts/run_quality.py |
Runs the fast and full check sequences and records their results |
scripts/quality.py, scripts/quality_js_symbols.mjs |
Catalog check, coverage sanity, duplicated blocks, report and code map |
.github/workflows/tests.yml |
Runs make quality and publishes the evidence of each commit of main |
backend/version.py, backend/services/quality_report.py |
Find the running commit and load its evidence |
quality/ |
The catalog, schemas, baseline and decisions |
Dockerfile.quality, compose.quality.yml |
The reproducible quality environment |
Related pages
- A4.3 Code quality: how to read the dashboard and the report.
- A4 Tests, A4.1 Backend tests and A4.2 JavaScript tests: what the suites cover and how to write tests.
- A3.1 Dependencies and builds: the requirement files and npm packages.