Repository navigation
Fix relations read several times with --store when relation blocks are subdivided - #980
geoneutrino wants to merge 1 commit into
Conversation
…e subdivided With sharded stores (the default with --store) the relations phase runs once per shard. When there are fewer than 2 * threadNum relation blocks, each pass subdivided the remaining blocks into threadNum chunks. ReadBlock never erases subdivided blocks, so the pass for the next shard subdivided the chunks again: threadNum * threadNum entries, each chunk threadNum times, and every relation of that shard was processed threadNum times. This only happens with --store, an extract whose ways fall into more than one shard, and few relation blocks, e.g. a country extract crossing a shard border. The duplicated multipolygons end up as identical features in the tiles, so the output also depended on the thread count. Subdivide only once. On test data with --store the tiles had 14,347,612 features with 12 threads and 14,249,548 with 4 threads (12,258 extra features per additional copy); now both have 14,212,774, the same as without --store.
|
Reviewed: the fix looks right to me. Subdivided blocks are never reported as done by One data point on when it triggers, in case it helps judge how common it is: I couldn't reproduce it on Norway with |
With
--store(sharded stores), relations are processedthreadNumtimes when an extract's ways fall into more than one shard and there are fewer than2 * threadNumrelation blocks. Each shard pass subdivided the remaining relation blocks again, and sinceReadBlocknever erases subdivided blocks, the second pass createdthreadNumcopies of every chunk.Effect: duplicated multipolygon features in the tiles, and output that depends on the thread count. The planet is not affected (enough relation blocks), and neither is the CI test area (single shard).
Fix: subdivide only once.
Measured on test data (OpenMapTiles profile,
--store):--storeExample: one z3 landcover tile contained the same mangrove polygon 12 times with 12 threads and 4 times with 4 threads; now once. Unit tests pass.
Found while verifying the deterministic-output work from discussion #969: together with those changes,
--storeoutput is byte-identical for 1, 4 and 12 threads and to the output without--store.