Matching engine E.5.2
Pipeline performance
On this page
A national Swiss run processes the complete national timetable and a nationwide OSM extract. This page explains what keeps it fast: reducing GTFS trips early, caching stages independently, indexed route comparison and fast bundle compression. It also lists the settings and logs, and how to measure performance. Timings depend on the feed, storage, memory and concurrent work.
GTFS: many trips, few patterns
stop_times.txt lists every trip, and most trips repeat the same ordered stops. The Swiss adapter therefore reduces trips to distinct patterns inside an embedded DuckDB database before any Python itinerary object is built. No DuckDB service or database connection is involved.
The implementation, src/osm_public_transport_matcher/adapters/gtfs_columnar.py, keeps these properties:
- Identifiers stay strings, leading zeros included. Only the numeric
stop_sequenceorders calls, and a repeated visit to the same stop stays a separate call. - First and last stops are taken before geographic filtering, so direction labels keep their termini abroad.
- 256 hash partitions keep every trip together, even when its rows are interleaved with other trips; trip metadata is partitioned the same way. Partitions bound working memory, but one extreme trip or partition can still exceed the budget.
- Trips are reduced by route, direction, headsign (or short name when there is no headsign) and ordered raw stop IDs. Trip counts are summed and the first representative provides the sequence numbers. Different platform sequences stay separate until the identity mapping merges them as variants.
- Python receives the reduced patterns and only the trip metadata they need.
- Identity decisions, sequence hashes, dominant-pattern selection and itinerary thresholds are the same as in the pandas path.
Temporary DuckDB tables and partition files live in the process temporary directory and are removed when processing ends. Reusable products are Parquet files in data/cache/gtfs.
GTFS_PROCESSING_BACKEND=pandas selects the chunked pandas reference reader, used for comparison and troubleshooting. It assumes that a trip's rows are not spread across chunks; the DuckDB path handles interleaved trips. The DuckDB path serves the Swiss acquisition and runner; the generic osm-public-transport-matcher gtfs adapter is a separate implementation without a national-scale benchmark.
Independent cache stages
Acquisition caches the filtered ATLAS platforms, the GTFS download, the parsed GTFS products and the GTFS-to-ATLAS identities separately, so a change to one source rebuilds only the stages that depend on it. E5.1 describes the stages, what invalidates each of them and --force.
Route comparison and OSM processing
- Family matching builds three indexes, by exact GTFS route ID, normalized route ID and display route ID, and scores only the candidates reachable through them. Scores and the final ordering by score and identifiers are the same as for an exhaustive comparison, so the greedy choice is unchanged.
- Itinerary matching uses longest-common-subsequence dynamic programming. Identity sets and parsed variant lists are prepared before the inner loop. Only the score is needed, so a single rolling row keeps memory linear.
- OSM routes: route construction does not materialize unused way-member records, and only the call frame that is modified is copied when keys are scoped. The runner builds the OSM route products once from the current XML.
Bundle compression
Bundles are compressed with gzip level 1 by default, which encodes faster than Python's default level 9 at the cost of larger files. --compression-level 0 to 9 changes it on the CLI; the Python writer takes compresslevel=. Readers and the schema are unaffected, and validation, checksums and the atomic publication always run.
Database loading
Loading a bundle into the review database is the review application's work. It uses PostgreSQL COPY into a private staging schema, builds secondary indexes after the load, and publishes in one short transaction; see A1.1 Bundle import and publication.
Settings and logs
| Setting | Engine default | Meaning |
|---|---|---|
GTFS_PROCESSING_BACKEND |
duckdb |
pandas selects the comparison path |
GTFS_THREADS |
2 |
DuckDB worker threads |
GTFS_MEMORY_LIMIT |
768MB |
DuckDB's managed memory budget; not a limit on the whole process or container. The review application's Compose file sets 1536MB for its scheduler. |
--compression-level |
1 |
Gzip level of the new bundle |
The three GTFS_* settings apply only to Swiss GTFS processing, during acquisition or with an explicit --gtfs folder; the in-memory core API and the generic gtfs command do not read them. More threads can raise memory use; benchmark before increasing both threads and memory. Python's stop and trip tables and the matching state use memory on top of DuckDB, and the native tables and partitions need temporary disk space.
Log records are JSON objects with the event stage_started, stage_progress, stage_finished or stage_failed. Long stages report every 30 seconds. Finished records give wall-clock and process CPU seconds; CPU time covers all threads and can exceed wall time. Nested stages overlap, so their durations must not be added up.
GTFS logs separate metadata loading, scanning, filtering and termini, partitioning, pattern reduction, identity matching, itinerary construction and cache input and output, and report input bytes and the reduced row, trip and pattern counts. cache_decision records explain each reuse or rebuild: changed inputs, missing or corrupt products, expired ATLAS rows, a forced rebuild or an unverifiable source.
Reproduce the measurements
Install the Swiss extra and run each backend in a fresh process on the same files. These commands neither download nor change source data; they write only temporary processing files, removed at the end, and the requested report:
python -m pip install -e '.[swiss,test]'
python benchmarks/gtfs.py --gtfs /path/to/gtfs --atlas /path/to/stops_ATLAS.csv --boundary /path/to/switzerland.geojson --backend pandas --report /tmp/gtfs-pandas.json
python benchmarks/gtfs.py --gtfs /path/to/gtfs --atlas /path/to/stops_ATLAS.csv --boundary /path/to/switzerland.geojson --backend duckdb --report /tmp/gtfs-duckdb.json --compare /tmp/gtfs-pandas.json
python -m pytest tests -q
The benchmark records wall and CPU time, peak memory of the process, product counts and content hashes. --compare ignores row order and compares the actual route, itinerary, stop-call and identity products. Run on an otherwise idle machine and repeat when evaluating hardware settings; the memory figure includes Python state and the benchmark's inputs, not only DuckDB.
The tests cover equal products of both backends, repeated calls, platform variants, foreign termini, leading-zero identifiers, missing directions, interleaved trips and empty selections, and compare indexed route candidates with exhaustive scoring and the rolling-row score with an exhaustive longest common subsequence.
documentation/benchmarks/pipeline-performance-2026-09-12.json records one set of national measurements of GTFS preprocessing with both backends, matching on cached products, the import into the review database and bundle compression, with their environments, product hashes and limitations. They are single observations, not a controlled study; for current evidence, rerun the commands above and the engine tests.
Technical references
- DuckDB CSV loading: typed CSV input and parallel reading.
- DuckDB Parquet support: columnar storage and partitioned processing.
- PostgreSQL bulk loading:
COPY, and loading before building secondary indexes.
Code
| File | Role |
|---|---|
src/osm_public_transport_matcher/adapters/gtfs_columnar.py |
The DuckDB pattern reduction, its thread and memory settings and the Parquet feed cache |
src/osm_public_transport_matcher/adapters/get_atlas_gtfs.py |
load_gtfs_data_streaming(), the chunked pandas reference reader |
src/osm_public_transport_matcher/swiss.py |
Chooses the backend from GTFS_PROCESSING_BACKEND |
src/osm_public_transport_matcher/routes.py |
The indexed family candidates and the rolling-row itinerary score |
src/osm_public_transport_matcher/results/bundle.py |
write_bundle() and its compresslevel |
src/osm_public_transport_matcher/timing.py |
The stage_* log records and their 30-second heartbeat |
benchmarks/gtfs.py |
The backend benchmark and its --compare check |
tests/test_gtfs_columnar.py |
Equal products of both backends, interleaved trips, foreign termini and empty selections |
Related pages
- E5.1 Source snapshot caching: the cache stages
- E1.2 GTFS and ATLAS: what the GTFS processing produces
- E3.2 Route-route matching: the rules route comparison applies
- A1.1 Bundle import and publication: how the review application loads a bundle quickly