Matching engine E.5.3
Pipeline progress contract
On this page
While it runs, the command-line tool prints its progress as JSON lines, so that a scheduler or user interface can show live, stage-by-stage status. This page defines that small, versioned contract. It is separate from the result bundle: progress is transient run state, a bundle is a durable result. The review application's side is described in A3.5.
Events
src/osm_public_transport_matcher/progress.py is the catalog of the engine's public stages. Each stage has a stable ID, a parent phase, a label, a description, an order and a progress kind (determinate or indeterminate). Internal timing spans can be finer and change freely.
The CLI first prints the plan of the selected run as one pipeline_plan event, then one event per transition: stage_started, stage_progress, stage_finished, stage_failed or stage_outcome. Every public event carries progress_schema_version: 2. A consumer rejects live events of another version, so the engine and the review application are deployed together. Consumers identify stages by stage_id; labels are only for display.
{
"event": "stage_started",
"progress_schema_version": 2,
"stage_id": "osm.download",
"stage_spec": {
"id": "osm.download",
"parent_id": "source_files",
"phase": "source_files",
"label": "Download OSM data from Overpass",
"description": "Request the Swiss public-transport snapshot and save the returned OSM XML.",
"order": 50,
"progress_kind": "indeterminate"
}
}
- A percentage is sent only when a stage has real
processedandtotalcounts; otherwise consumers show the state and the elapsed time. - A cache hit is a
stage_outcomewithreused; work that does not apply to the run is astage_outcomewithskipped. - A failure carries its error text. Heartbeats stop before a stage's final event.
- Each planned stage runs once per CLI run. Rules that share a public stage must be adjacent.
- Other JSON lines the process prints, such as timing logs, are not part of the contract.
Consumers should ignore progress that arrives after completion, and close open stages themselves when the process dies without a final event.
The plan depends on the run
The plan lists only stages that can still run, so a waiting stage always means real pending work. The CLI resolves its inputs before choosing the plan:
- A local Swiss run includes
atlas.load; cached products addtimetable.load; an explicit--gtfsaddstimetable.local_feed. - A matching run on an existing OSM snapshot includes
osm.preparebut notosm.download. - A generic GTFS run uses
timetable.local_feedfor feed loading and route preparation. - A download run with an explicit local feed lists both the acquisition work and the local-feed preparation, each with its own lifecycle.
Phases group stages by work, not by data source: downloads belong to the source_files phase; reading local inputs and parsing OSM belong to matching_inputs. Phases and their stages follow execution order. Consumers still need the explicit states to tell completed, reused, skipped and failed work apart.
Progress is reported only within a CLI run, and reporting is reset even after a failure. Library calls emit no public progress unless a caller opts in with progress_reporting(), as the tests and custom runners do.
Public engine stages
| Phase | Stable ID | Label |
|---|---|---|
| Check sources | source_check.remote_sources |
Check ATLAS and timetable sources |
| Prepare source files | atlas.prepare |
Download and prepare ATLAS stops |
| Prepare source files | timetable.download |
Download timetable data |
| Prepare source files | timetable.patterns |
Prepare timetable patterns |
| Prepare source files | timetable.identities_routes |
Match GTFS stops to ATLAS and build timetable routes |
| Prepare source files | osm.download |
Download OSM data from Overpass |
| Load matching inputs | atlas.load |
Read ATLAS stops |
| Load matching inputs | timetable.load |
Read cached timetable products |
| Load matching inputs | timetable.local_feed |
Prepare the supplied GTFS feed |
| Load matching inputs | osm.prepare |
Parse and prepare OSM data |
| Stop matching | stop_matching.groups |
Build OSM stop groups |
| Stop matching | stop_matching.identifier |
Identifier matching |
| Stop matching | stop_matching.two_sided_stop |
Two-sided stop matching |
| Stop matching | stop_matching.reference |
Reference matching |
| Stop matching | stop_matching.name |
Name matching |
| Stop matching | stop_matching.gtfs_route |
GTFS route matching |
| Stop matching | stop_matching.distance |
Distance matching |
| Stop matching | stop_matching.unmatched |
Identify remaining unmatched stops |
| Stop matching | stop_matching.problems |
Detect data problems |
| Route matching | route_matching.normalize |
Normalize route data |
| Route matching | route_matching.families |
Match line families |
| Route matching | route_matching.itineraries |
Match itineraries |
| Build result bundle | bundle.fingerprints |
Record source fingerprints |
| Build result bundle | bundle.metrics |
Calculate quality metrics |
| Build result bundle | bundle.serialize |
Serialize and validate the result bundle |
The catalog deliberately merges internal steps that are not useful to an operator: the Overpass request and saving its answer are one download stage; GTFS scan, partition and trip reduction are one pattern stage; the distance rules share one stage, and problem detection is one stage. The rule order itself is documented in E2.
The review application adds its own phases and stages for staging and publishing the database, and merges both plans without importing engine code.
Adding or changing a stage
- Prefer an existing stage when the new work is a detail of the same visible outcome.
- If a new boundary is justified, add one catalog entry with a new stable ID. IDs must never be renamed within a schema version; better labels and descriptions are compatible.
- Wrap the work in
progress_stage(stage_id). Generic rule runners assign stage metadata; algorithms contain no presentation code. - Add catalog and lifecycle tests. Do not assert wall-clock values or heartbeat counts.
The progress tests run tiny Swiss, explicit-GTFS and generic GTFS CLI runs, check that every planned stage resolves exactly once and in plan order, and validate the bundles. Acquisition tests check the reported outcomes against real cold and warm cache reuse, corrupt-cache rebuilds and failure recovery. Matching correctness is tested separately.
Code
| File | Role |
|---|---|
src/osm_public_transport_matcher/progress.py |
The stage catalog, the run's plan, progress_stage(), stage_outcome events and PROGRESS_SCHEMA_VERSION |
src/osm_public_transport_matcher/cli.py |
Chooses the plan from the command's inputs and reports progress for one run |
tests/test_progress.py |
The catalog, the event lifecycle and tiny CLI runs that execute their plan |
tests/test_adapters_acquisition.py |
Reported outcomes on cold and warm cache reuse, corrupt caches and failures |
Related pages
- A3.5 Pipeline progress: how the review application merges and shows the plan
- E5.2 Pipeline performance: the timing logs, which are not part of the contract
- E5.1 Source snapshot caching: when a stage is reused
- E2 Matching process: the rules behind the stop matching stages