Repository navigation
Make tile content independent of thread scheduling - #981
geoneutrino wants to merge 2 commits into
Conversation
Several internal indices are handed out in insertion order, which depends on thread scheduling, and were used as sort keys or write order. With more than one thread, the features in a tile, their order and the tag order within a feature could differ from run to run. Replace them with content-based keys: - Attribute sets are sorted by a content hash (key names, values, minzoom) instead of by AttributeIndex, with a full content comparison on hash collisions. The hash is cached per thread in a small direct-mapped table (64 KB per thread). - Ties are broken on the OSM id before the objectID. - Stored geometries (relations, shapefiles, ...) get an objectID that depends on which thread stored them. Runs of such objects that are still tied are ordered by first point and point count, then by a hash over all points. OSM ways and nodes carry their OSM id in the objectID and never touch the geometry stores for this. - Tags are written in key-name order instead of pair-index order, using a key rank computed once in AttributeKeyStore::finalize(). The hashes come from a new stable_hash.h (splitmix64 / FNV-1a) instead of std::hash or boost::hash, whose results differ between platforms and Boost versions, so the order is the same on every platform. Test data, with the checks of .github/scripts/verify-generated-tiles.py: repeated runs, 4 vs. all threads and Makefile vs. CMake builds give the same tile content for mbtiles, pmtiles and MLT output, with the OpenMapTiles profile and a custom profile (master: 28,050 of 94,709 tiles differ between two runs). Against master, merged geometry is unchanged; only which id survives a merge differs. No measurable runtime or memory change.
The relation scan runs on several threads and appended each member's parent relations in scan order, so NextRelation() returned them in an order that depended on thread scheduling. Profiles that use the first matching relation (e.g. the network of the first route relation of a way) then wrote different attribute values from run to run. Insert each parent relation at its position by relation id instead. Entries of the same relation keep their member order. For a PBF sorted by type and id this is the order a single-threaded run gives. Test data with a custom profile that takes the network of the first route relation: before, two runs with all threads differed in 26 tiles and all threads vs. one thread in 63 tiles (transport_label); now runs with all threads, 4 threads and 1 thread are byte-identical, and the single-threaded output is unchanged.
|
Pushed one more commit: the relation scan appended each way's/node's parent relations in multi-threaded scan order, so |
|
Planet re-run as requested, and the heap difference is gone: master 2789651 vs. this PR (0bf498b), planet
So the Germany with this PR + #942 (OpenMapTiles profile, Lua 5.3): all 273,199 tiles are byte-identical as stored, between two 32-thread runs and between 32 and 8 threads. The ordered writer for the PMTiles/MBTiles files is ready too. On the Canary Islands, together with this PR and #942, the whole |
|
Independent confirmation from a production tileset: this works and the effect on incremental updates is large. Context: we (@librescoot) build offline vector tiles (MBTiles, ~0-14, custom Shortbread-like profile with streets, labels, buildings, addresses, water, land) for on-device maps from Geofabrik extracts, one file per region, monthly. Many of our users want to download whole regions over cellular, so we want to ship small deltas instead of making them download large region extracts. Test setup: Bremen extract (20 MB PBF), unchanged profile, 4 threads, four builds with commit Same input, two runs:
The stored MBTiles files still differ byte-wise, as you note, because SQLite insert order follows thread completion, but the tile payloads are identical. Decoding the PR's tiles and comparing decoded layers/features (with feature list and MultiPoint coordinate order normalised) against a base build of the same snapshot showed no content difference, so the change is ordering only. Incremental updates between the two OSM snapshots:
The build cost was unchanged within noise. We also built 25 regional extracts with this commit, up to 648 MB input, with the same profile; all files pass SQLite integrity, metadata, gzip and MVT decoding checks. Two follow-ups that might matter to others:
|
Sorry, long text :)
(Co-Authored and data verified by Claude Code)
Follow-up to discussion #969. With more than one thread, the features in a tile, their order and the tag order within a feature could differ from run to run, because several internal indices are handed out in insertion order (i.e. thread scheduling) and were used as sort keys or write order. This replaces them with content-based keys:
AttributeIndex, with a full content comparison on hash collisions. The hash is cached per thread in a small direct-mapped table (64 KB per thread).objectID.objectIDthat depends on which thread stored them. As suggested by @dktue, runs that are still tied are ordered by geometry: first point and point count, then a hash over all points. No ids are added toOutputObjectXY, so no extra memory per object. OSM ways and nodes carry their OSM id in theobjectIDand never touch the geometry stores for this.AttributeKeyStore::finalize(). This is done at write time rather than infinalizeSet, because key indices are not final at that point.Changes since the prototype discussed in #969:
include/stable_hash.h(splitmix64 / FNV-1a) instead ofstd::hash/boost::hash, whose results differ between platforms and Boost versions. The order is therefore the same on every CI runner.thread_localsort buffers are now local vectors. They kept up to a few MB per thread allocated for the whole run, which may explain part of the +0.7–1 GB heap peak @dktue measured on the planet. A re-run of the heap measurement would be welcome.Output formats
-DTILEMAKER_WITH_MLT=ON)std::mapfor properties/columns and first-occurrence dictionaries, so it only depends on feature orderByte-identical archives need ordered writing in the mbtiles/pmtiles writers. That interacts with #966 (batch dispatch, mbtiles write buffer), so I'd suggest a separate PR on top, as discussed in #969.
Testing
Test data (~350 MB extract), checked with
.github/scripts/verify-generated-tiles.pyrun locally:include_ids: true,combine_polygons_below), each built with the Makefile (all threads,--threads=4) and CMake.--store), pmtiles and MLT pmtiles, each with a repeat run.While doing this I found that
--storeprocessed relations several times on extracts spanning more than one shard. That is fixed separately in #980. With both,--storeoutput is byte-identical for 1, 4 and 12 threads and identical to the output without--store.Notes
pairs()order: profiles that depend on it (e.g.GetPOIRank()in the OpenMapTiles profile with LuaJIT) still need Make POI tag key processing deterministic #942. With this PR plus Make POI tag key processing deterministic #942, @dktue got 0 of 273,199 Germany tiles different (32 threads, and 32 vs. 8 threads). The tests above used PUC Lua 5.1.verify-generated-tiles: it probably needs Make POI tag key processing deterministic #942 first, since the Ubuntu runners use LuaJIT. Remaining gaps in that job: the MLT output has no-repeatrun, and the semantic fallback only decodes MVT.