Matching engine E.5.3

Pipeline progress contract

While it runs, the command-line tool prints its progress as JSON lines, so that a scheduler or user interface can show live, stage-by-stage status. This page defines that small, versioned contract. It is separate from the result bundle: progress is transient run state, a bundle is a durable result. The review application's side is described in A3.5.

Events

src/osm_public_transport_matcher/progress.py is the catalog of the engine's public stages. Each stage has a stable ID, a parent phase, a label, a description, an order and a progress kind (determinate or indeterminate). Internal timing spans can be finer and change freely.

The CLI first prints the plan of the selected run as one pipeline_plan event, then one event per transition: stage_started, stage_progress, stage_finished, stage_failed or stage_outcome. Every public event carries progress_schema_version: 2. A consumer rejects live events of another version, so the engine and the review application are deployed together. Consumers identify stages by stage_id; labels are only for display.

{
  "event": "stage_started",
  "progress_schema_version": 2,
  "stage_id": "osm.download",
  "stage_spec": {
    "id": "osm.download",
    "parent_id": "source_files",
    "phase": "source_files",
    "label": "Download OSM data from Overpass",
    "description": "Request the Swiss public-transport snapshot and save the returned OSM XML.",
    "order": 50,
    "progress_kind": "indeterminate"
  }
}
  • A percentage is sent only when a stage has real processed and total counts; otherwise consumers show the state and the elapsed time.
  • A cache hit is a stage_outcome with reused; work that does not apply to the run is a stage_outcome with skipped.
  • A failure carries its error text. Heartbeats stop before a stage's final event.
  • Each planned stage runs once per CLI run. Rules that share a public stage must be adjacent.
  • Other JSON lines the process prints, such as timing logs, are not part of the contract.

Consumers should ignore progress that arrives after completion, and close open stages themselves when the process dies without a final event.

The plan depends on the run

The plan lists only stages that can still run, so a waiting stage always means real pending work. The CLI resolves its inputs before choosing the plan:

  • A local Swiss run includes atlas.load; cached products add timetable.load; an explicit --gtfs adds timetable.local_feed.
  • A matching run on an existing OSM snapshot includes osm.prepare but not osm.download.
  • A generic GTFS run uses timetable.local_feed for feed loading and route preparation.
  • A download run with an explicit local feed lists both the acquisition work and the local-feed preparation, each with its own lifecycle.

Phases group stages by work, not by data source: downloads belong to the source_files phase; reading local inputs and parsing OSM belong to matching_inputs. Phases and their stages follow execution order. Consumers still need the explicit states to tell completed, reused, skipped and failed work apart.

Progress is reported only within a CLI run, and reporting is reset even after a failure. Library calls emit no public progress unless a caller opts in with progress_reporting(), as the tests and custom runners do.

Public engine stages

Phase Stable ID Label
Check sources source_check.remote_sources Check ATLAS and timetable sources
Prepare source files atlas.prepare Download and prepare ATLAS stops
Prepare source files timetable.download Download timetable data
Prepare source files timetable.patterns Prepare timetable patterns
Prepare source files timetable.identities_routes Match GTFS stops to ATLAS and build timetable routes
Prepare source files osm.download Download OSM data from Overpass
Load matching inputs atlas.load Read ATLAS stops
Load matching inputs timetable.load Read cached timetable products
Load matching inputs timetable.local_feed Prepare the supplied GTFS feed
Load matching inputs osm.prepare Parse and prepare OSM data
Stop matching stop_matching.groups Build OSM stop groups
Stop matching stop_matching.identifier Identifier matching
Stop matching stop_matching.two_sided_stop Two-sided stop matching
Stop matching stop_matching.reference Reference matching
Stop matching stop_matching.name Name matching
Stop matching stop_matching.gtfs_route GTFS route matching
Stop matching stop_matching.distance Distance matching
Stop matching stop_matching.unmatched Identify remaining unmatched stops
Stop matching stop_matching.problems Detect data problems
Route matching route_matching.normalize Normalize route data
Route matching route_matching.families Match line families
Route matching route_matching.itineraries Match itineraries
Build result bundle bundle.fingerprints Record source fingerprints
Build result bundle bundle.metrics Calculate quality metrics
Build result bundle bundle.serialize Serialize and validate the result bundle

The catalog deliberately merges internal steps that are not useful to an operator: the Overpass request and saving its answer are one download stage; GTFS scan, partition and trip reduction are one pattern stage; the distance rules share one stage, and problem detection is one stage. The rule order itself is documented in E2.

The review application adds its own phases and stages for staging and publishing the database, and merges both plans without importing engine code.

Adding or changing a stage

  1. Prefer an existing stage when the new work is a detail of the same visible outcome.
  2. If a new boundary is justified, add one catalog entry with a new stable ID. IDs must never be renamed within a schema version; better labels and descriptions are compatible.
  3. Wrap the work in progress_stage(stage_id). Generic rule runners assign stage metadata; algorithms contain no presentation code.
  4. Add catalog and lifecycle tests. Do not assert wall-clock values or heartbeat counts.

The progress tests run tiny Swiss, explicit-GTFS and generic GTFS CLI runs, check that every planned stage resolves exactly once and in plan order, and validate the bundles. Acquisition tests check the reported outcomes against real cold and warm cache reuse, corrupt-cache rebuilds and failure recovery. Matching correctness is tested separately.

Code

File Role
src/osm_public_transport_matcher/progress.py The stage catalog, the run's plan, progress_stage(), stage_outcome events and PROGRESS_SCHEMA_VERSION
src/osm_public_transport_matcher/cli.py Chooses the plan from the command's inputs and reports progress for one run
tests/test_progress.py The catalog, the event lifecycle and tiny CLI runs that execute their plan
tests/test_adapters_acquisition.py Reported outcomes on cold and warm cache reuse, corrupt caches and failures