Matching engine E.1.3
OSM input and acquisition
On this page
OpenStreetMap provides the other side of every comparison: the stop elements and the route relations. This page explains which OSM data the engine downloads, which elements it keeps, and how one XML extract becomes stop records, route evidence and route products.
Three modules share the work:
acquisition/overpass.pyoptionally downloads one XML extract from the Overpass API.adapters/osm.pyparses the stop elements and their route memberships intoOsmNoderecords and anOsmState.adapters/osm_routes.pybuilds the route products (route masters, route relations, members and stops), keeping the PTv2 structure.
Both adapters read the same explicit file and never access the network.
The download
| Property | Value |
|---|---|
| Endpoint | https://overpass-api.de/api/interpreter, or OVERPASS_API_URL: one URL or a comma-separated list, tried in order |
| Area | Switzerland, ISO3166-1=CH |
| Request | Overpass QL in the body of a POST (text/plain; charset=utf-8), with the user agent OVERPASS_USER_AGENT |
| Timeouts | 30 s to connect, 600 s to read; the query itself asks for at most 360 s |
| Validation | The answer must parse as an <osm> document without a <remark> runtime error: …</remark>. Overpass reports a query timeout or memory exhaustion that way, inside an HTTP 200 answer, after any partial data |
| Retries | HTTP 429, 502, 503 and 504, connection failures, timeouts and answers that fail validation are retried OVERPASS_MAX_RETRIES times per endpoint (default 2), waiting OVERPASS_RETRY_BACKOFF_SECONDS (default 5 s) and doubling the wait each time. Any other HTTP error moves on to the next endpoint at once |
| Data time | The answer's <meta osm_base="…">, the time of the newest OSM edit the server had, is recorded as osm_data_at in the acquisition metadata. last_overpass_query_at is when the query ran |
Only a validated answer is written and replaces data/raw/osm_data.xml. When every endpoint has failed, osm.download raises one error that lists each endpoint's last failure and its number of attempts, for example Overpass request failed: https://overpass-api.de/api/interpreter: HTTP 504 (3 attempts), and the run stops before matching, so no bundle is written. The text of an Overpass error page goes to the log only.
The complete query:
[out:xml][timeout:360];
area["ISO3166-1"="CH"]->.searchArea;
(
node(area.searchArea)["public_transport"~"platform|stop_position|station|halt|stop"];
node(area.searchArea)["railway"="tram_stop"];
node(area.searchArea)["amenity"="ferry_terminal"];
node(area.searchArea)["amenity"="bus_station"];
node(area.searchArea)["highway"="bus_stop"];
node(area.searchArea)["railway"="halt"];
node(area.searchArea)["railway"="station"];
node(area.searchArea)["aerialway"="station"];
node(area.searchArea)[~"^(disused|abandoned|removed|demolished|razed|destroyed|was):(public_transport|railway|amenity|highway|aerialway)$"~"^(platform|stop_position|station|halt|stop|tram_stop|ferry_terminal|bus_station|bus_stop)$"];
)->.pt_nodes;
(
way(area.searchArea)["aerialway"="station"]["public_transport"="station"];
way(area.searchArea)["uic_ref"];
way(area.searchArea)["ref:sloid"];
way(area.searchArea)["railway:ref:sloid"];
)->.candidate_ways;
(
relation(bn.pt_nodes)[type=route][route~"^(bus|trolleybus|minibus|share_taxi|coach|tram|light_rail|train|subway|monorail|ferry|funicular|aerialway)$"];
relation(bw.candidate_ways)[type=route][route~"^(bus|trolleybus|minibus|share_taxi|coach|tram|light_rail|train|subway|monorail|ferry|funicular|aerialway)$"];
relation(bn.pt_nodes)[type=route][!route];
relation(bw.candidate_ways)[type=route][!route];
)->.seed_routes;
relation(br.seed_routes)[type=route_master][route_master~"^(bus|trolleybus|minibus|share_taxi|coach|tram|light_rail|train|subway|monorail|ferry|funicular|aerialway)$"]->.route_masters;
(
.seed_routes;
relation(r.route_masters)[type=route][route~"^(bus|trolleybus|minibus|share_taxi|coach|tram|light_rail|train|subway|monorail|ferry|funicular|aerialway)$"];
relation(r.route_masters)[type=route][!route];
)->.routes;
.pt_nodes out body qt;
.candidate_ways out body center qt;
.routes out meta;
.route_masters out meta;
It returns the stop nodes, including disused ones, the candidate ways with their centre points, the public transport route relations that contain one of them, the route masters of those relations, and the other route relations of those route masters. Route relations without a route tag are downloaded too, because they can belong to a public transport route master.
Which elements are stops
| Tag | Values |
|---|---|
public_transport |
platform, stop_position, station, halt, stop |
railway |
tram_stop, halt, station |
highway |
bus_stop |
amenity |
ferry_terminal, bus_station |
aerialway |
station |
The query's eight stop lines are generated from these selectors, STOP_TAG_SELECTORS in core/osm_tags.py.
Besides nodes, the engine keeps two kinds of ways as stops, each represented by its centre point:
aerialway=stationwithpublic_transport=station;- ways with a
uic_refthat no active node already carries; a disused node never hides such a way.
The centre is the Overpass <center>; without one, it is the mean of the way's member nodes. A way with neither is not kept. One function, stop_ways() in adapters/osm_stop_ways.py, applies these rules for both readers of the snapshot: read_osm() for stop matching and process_osm_routes_data() for the stop calls of route relations. A route member on a way therefore resolves exactly when stop matching knows that way.
They get identities such as way_123 (key osm:way:123) and are matched like nodes. OSM pairs and two-sided stops are built from nodes only. Station elements are kept but excluded from matching (E2).
A third kind of way is kept only for its stop identifier: any other way with ref:sloid or railway:ref:sloid, in practice a platform area. It is marked identifier_only. Only that identifier can link it, and it adds nothing to the name, uic_ref or direction evidence of the other elements; it is not a stop call of a route (E2.1).
Stop identifiers
Swiss mappers can tag the ATLAS identifier of a platform, its SLOID, on the OSM elements of that platform: ref:sloid=ch:1:sloid:7000:4:7, or railway:ref:sloid on railway elements. One tag may list several SLOIDs separated by ;, such as a railway platform that serves two tracks. The query downloads ways with either key, and read_osm() keeps them; both read the keys from STOP_IDENTIFIER_KEYS in core/osm_tags.py.
Disused stops
An element is disused when a stop tag carries an OSM lifecycle prefix for a past state: disused:, abandoned:, removed:, demolished:, razed:, destroyed: or was:, as in disused:highway=bus_stop. The prefixed key can be any key of the table and the value any value of the table, compared exactly. Prefixes for future features, construction: and proposed:, are not recognised.
The last line of .pt_nodes downloads these elements even when every stop tag is prefixed, and their route relations are seeded like any stop's. Its two patterns, LIFECYCLE_KEY_PATTERN and LIFECYCLE_VALUE_PATTERN, are built from STOP_TAG_SELECTORS and also classify the parsed elements, so what is downloaded and how it is classified cannot drift apart.
lifecycle_of() records the prefix as the element's lifecycle; when several prefixes occur, the first in the order above wins. A disused element never takes part in matching, even when an unprefixed stop tag remains (E2). It gets a disused problem only when it still needs attention (E4.1).
Which route relations are services
Stops are shared by more than public transport: ski pistes, mountain-bike and running routes pass through aerial lift stations, and route=railway relations describe a railway line as infrastructure, not a service run on it. Only public transport services take part in route matching, so a route relation counts only when its route tag is one of
bus, trolleybus, minibus, share_taxi, coach, tram, light_rail, train, subway, monorail, ferry, funicular, aerialway.
A relation without a route tag takes the route_master tag of its route master: some tram and bus lines tag the mode only on the route master. A route master counts when its route_master tag is in the list.
The list is PUBLIC_TRANSPORT_ROUTES in core/osm_tags.py. The query selects with it, and public_transport_relation_ids() applies the same rule to the parsed XML, for both the route evidence and the route products. Other relations in the XML are ignored.
Stop records
adapters.osm.read_osm(xml_path) turns every stop element into an OsmNode:
| Field | From | Used for |
|---|---|---|
node_id |
The node ID, or way_<id> |
Identity |
lat, lon |
The node position, or the way centre | Spatial matching, maps |
local_ref |
local_ref, else ref |
Telling platforms apart |
name |
name |
Name matching, labels |
uic_name |
uic_name |
Name matching and grouping |
uic_ref |
uic_ref |
Reference matching, grouping, route stop identity |
network, operator |
Same-named tags | Display; the operator attribute problem |
public_transport, railway, amenity, aerialway |
Same-named tags | Grouping and element type |
tags |
All tags | Further fields such as gtfs:name, operator:wikidata, network:wikidata |
lifecycle |
lifecycle_of(tags): the lifecycle prefix of a stop tag, such as disused; None for active elements |
Keeping disused stops out of matching |
identifier_only |
True for a way kept only for its stop identifier | Keeping that way out of every rule but identifier matching |
Element type is part of the identity, so a node and a way with the same number stay distinct, and so do co-located elements.
Route evidence for stop matching
While reading the XML, read_osm() also records, for every stop element, the public transport route relations it belongs to:
- Route memberships with the relation's
gtfs:route_id, its normalised form and a direction. The direction comes from theref_tripstag. A relation without it is recorded for both directions, so that its route identifier can still match. - Direction names, such as
Seeplatz → Spital: the names of the relation's first and last members, attached to every member.
The GTFS route rule and the route-contradiction problem use this evidence.
Route products
process_osm_routes_data(xml_text, out_dir) builds the OSM side of route comparison, keeping the PTv2 hierarchy. Both dataset runners build them from the run's own XML with out_dir=None, so they stay in memory. With a directory as out_dir (the default is data/processed/), the same frames are also written there as diagnostic CSV files, named below.
| Product | One row per | Principal columns |
|---|---|---|
osm_route_masters |
public transport type=route_master relation |
route_master_id, route_master (mode), name, ref, operator, operator_wikidata, network, network_wikidata, colour, gtfs_route_id, is_non_gtfs |
osm_route_relations |
public transport type=route relation: one itinerary |
relation_id, route (mode), name, ref, operator and network fields, from_name, to_name, via, public_transport_version, colour, gtfs_route_id, gtfs_trip_id, gtfs_trip_id_sample, gtfs_shape_id, route_master_id, family_origin, synthetic_family_key |
osm_route_master_tags, osm_route_relation_tags |
tag | Key and value |
osm_route_master_members |
member of a route master | The ordered public transport route relations |
osm_route_relation_members |
member of a route relation | member_type, member_ref, member_role, member_sequence, resolved_node_id, derived direction |
osm_route_relation_stops |
stop member of a route relation | See below |
is_non_gtfs is true when the network tag is FlixBus (in any letter case): such routes are not in the Swiss timetable. They are skipped by family matching and listed on the review application's non-GTFS routes tab.
osm_route_relation_stops is the OSM source of stop calls:
| Column | Meaning |
|---|---|
relation_id |
The route relation |
direction_id |
From ref_trips, when available |
stop_sequence |
Order of the member |
osm_node_id |
The member's node or way identity |
stop_role |
The member role, such as stop or platform |
canonical_stop_key |
uic:<uic_ref> when available, otherwise osm:<node_id> |
stop_label |
The best available name |
uic_ref, stop_lat, stop_lon |
Copied from the member |
Stop members without a supported position are left out of the stop calls and counted in the unsupported_or_external_osm_route_members diagnostic, with sample identities. The fingerprinted XML still contains them.
Tags the engine relies on
| Tag | Purpose | Used by |
|---|---|---|
uic_ref |
Station identity | Reference matching, grouping, route stop identity |
ref:sloid, railway:ref:sloid |
The exact ATLAS platform | Identifier matching and identifier problems |
name, uic_name, gtfs:name |
Stop names | Name matching |
local_ref |
Platform code | Reference, name and distance rules |
gtfs:route_id |
Link to a GTFS route | Route matching and route comparison |
ref_trips |
Direction, see below | Route matching and route comparison |
route, route_master |
Whether a relation is a service | Route evidence and route products |
operator, network |
Line metadata | Route families, the operator attribute problem |
The ref_trips tag
In Switzerland, ref_trips holds one or more comma-separated trip references of the timetable the relation follows, such as 92-742-A-j26-1.2.H or 1.TA.92-56-j22-1.1.H. Each ends in .H (Hinfahrt) or .R (Rückfahrt), which the engine reads as GTFS direction_id 0 and 1, taking the first reference with either suffix. No other OSM tag states the GTFS direction: from and to name the terminals, not the direction ID. A relation without ref_trips has no direction; route matching then pairs it on its stops in order (E3.2).
Limits
Ways are represented by one point; full polygon geometry is not compared. Relations are not stop elements. The number of elements with route evidence can be read from the bundle's route_evidence extension.
Code
| File | Role |
|---|---|
src/osm_public_transport_matcher/acquisition/overpass.py |
query_overpass() and build_public_transport_query(): the download |
src/osm_public_transport_matcher/core/osm_tags.py |
STOP_TAG_SELECTORS, STOP_IDENTIFIER_KEYS, LIFECYCLE_PREFIXES, lifecycle_of() and has_active_stop_tag(): stop tags, identifier keys and lifecycle prefixes |
src/osm_public_transport_matcher/adapters/osm.py |
read_osm(): stop records and route evidence |
src/osm_public_transport_matcher/adapters/osm_routes.py |
process_osm_routes_data(): route products |
Related pages
- E2 Matching process: how OSM elements are grouped before matching
- E2.3 GTFS route matching: how the route evidence decides a match
- E3.2 Route-route matching: how the route products are compared with the timetable
- E4.1 Stop problems: what is checked on disused elements