Repository navigation
Conversation
All ~14M planet tile batches were posted to the thread pool up front, each handler copying options and sortOrders (~15 GB at the start of the tile phase). Workers now take the next batch from a shared counter; the batch partition, and so the output, is unchanged. The mbtiles pending-statement buffer had no limit and reached 3.2 GB on the planet. It is now capped at 256 MB; past that, a worker waits for the SQLite lock and drains the buffer itself. Planet anon RSS peak 25.4 GB -> 12.4 GB, wall time unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…-identical Tiles were written in the order threads finished them, so the tile data offsets, the PMTiles directory and the SQLite page layout differed between runs, even with identical tile content (systemed#969, systemed#981). Batches are now numbered as they're handed out, and a small helper (OrderedBatches) writes each batch only after the ones before it. Whichever thread finds the next batch ready writes it, outside the lock; others hand their batch over and carry on. At most 1 GB is held back; above that, threads wait before taking a new batch. The PMTiles tiny-tile cache is now filled in that order too, which also removes a race where two threads could write the same tiny tile. With systemed#981 and systemed#942, output is byte-identical for any thread count and identical to a single-threaded run. Planet .pmtiles, 32 threads: wall 1,919 -> 1,710 s (-10.9%), system CPU -73%, heap +0.7 GB. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This was referenced Nov 5, 2026
With a fixed 1 GB limit, threads at the default thread count (96 on our server) often had to wait for room while an earlier batch was still being worked on: 925,000 waits, 35,900 thread-seconds on the planet, and the planet was 1-6% slower than without the ordered writer. Allow 32 MB of held tiles per thread instead (1 GB at 32 threads, as before; 3 GB at 96). Planet .pmtiles, 96 threads: 1,594 / 1,605 s without the ordered writer, 1,452 / 1,444 s with it (-9.6%), heap +3 GB. Output unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
At 96 threads, 32 MB per thread still left threads waiting for room for 20,000 thread-seconds on the planet, mostly behind a slow z12 batch. Most likely, with two threads per core each runs at about half speed, so a dense z12 area holds up everything behind it for longer. With 64 MB per thread the planet (in memory, all open performance PRs) takes 14:12 / 14:06 instead of 19:52, about as fast as 48 threads on the 48 physical cores (14:24), and peak RSS stays below master's (161 vs 163 GB). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
end() woke every thread waiting in waitForRoom() after each batch it wrote, although most of the time the held tiles were still over the limit. At 96 threads that meant dozens of threads waking, finding no room and sleeping again for every batch: 125M context switches and 7,700 s of system time on the planet. Now they are only woken once the held tiles are back under the limit. Planet in memory, 96 threads, all open performance PRs: 14:05 -> 12:45 (-9.5%), system time 7,708 -> 2,868 s, context switches 125M -> 35M. Output unchanged (same batches, same order). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This was referenced Nov 8, 2026
At 96 threads the one thread that writes batches in order became the bottleneck of the tile phase: busy 260 s of a 323 s tile phase, with the others waiting for room for 5,100 thread-seconds. Writing the bytes was more than half of its work (ofstream::write, and a tellp() per tile). - The writer keeps its own tile data offset instead of calling tellp() for every tile, and the stream gets a 4 MB buffer. - OrderedBatches can take a second step, flushItem: also in batch order and one thread at a time, but run by whichever thread is free, so it doesn't hold up the first step. PMTiles places tiles (offset, repeated tiny tiles, index) in the first step and writes their bytes in the second. The file is byte-identical. Planet in memory, 96 threads, all open performance PRs: 10:41 -> 9:54 (-7.3%); tile phase 323 -> 245 s, threads busy 76 -> 94 of 96 on average. New test: the second step runs in order, never twice at once, and always after the first (also clean under TSan). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gain:
.pmtilesand.mbtilesfiles are byte-identical between runs, for any thread count, together with #981 and #942. Two planet runs now give the same 76.6 GB file, at 32 and at 96 threads. It's also faster: planet wall −10.9% at 32 threads (1,919 → 1,710 s) and −9.6% at 96 threads (1,600 → 1,448 s), measured with the limit at 32 MB per thread. With the limit now at 64 MB per thread, all open performance PRs together build the planet in 14 minutes at the default 96 threads, instead of 20 minutes with 32 MB (and 25 minutes onmaster). The fourth commit wakes threads waiting for room only once there is room: at 96 threads that takes all open performance PRs from 14:05 to 12:45 (−9.5%), with 63% less system time. The fifth commit places and writes tiles in two ordered steps run by different threads: at 96 threads the tile phase no longer waits for the one writing thread (planet 10:41 → 9:54 on our current stack).On top of #966 (it uses #966's on-demand batch hand-out): the branch is current master with #966 merged in, plus one commit, which is this PR. Follow-up to #969 and #981, as discussed there with @geoneutrino.
Problem
#981 makes the content of every tile independent of thread scheduling. The files still differed, because both writers store tiles in the order threads finish them:
Change
nextBatchnumbers batches as it hands them out (they're already a fixed partition that doesn't depend on thread count).OrderedBatches<T>(new,include/ordered_batches.h). A thread collects a batch's items betweenbegin()andend().end()writes the batch if it's the next one, followed by any held batches that were waiting for it; otherwise it holds the batch. Whichever thread finds the next batch ready writes it outside the lock, so the others just hand their batches over and carry on.waitForRoom()before taking its next batch. That can't deadlock: the batch everyone waits for is always being worked on by a thread that isn't waiting. Threads wait mostly for a slow z12 batch (a dense area with its z13/z14 tiles). Most likely, with two threads per core each runs at about half speed, so such a batch holds everything up for longer; that would explain why the limit matters most at the default thread count on machines with SMT. The first version had a fixed 1 GB, the second 32 MB per thread; both still left threads waiting at 96 threads (see below). Waiting threads are woken only once the held batches are back under the limit (fourth commit).The order is the one a single thread produces, so a multi-threaded run now writes exactly the file a
--threads 1run writes.Results
Determinism (Canary Islands, OpenMapTiles profile, master + #966 + #981 + #942, with and without this PR; whole-file sha256):
.pmtileswithout this PR6e2863e01ceace8ebb5383da16f4bc06.pmtileswith this PR16f4bc0616f4bc0616f4bc0616f4bc06.mbtileswithout this PR42b3801755806e2dc10bc7395770e145.mbtileswith this PR5770e1455770e1455770e1455770e145Cost (planet,
.pmtiles,--store --fast, 2 runs each at 32 and at 96 threads; Germany.mbtiles, 3 rounds):.pmtiles, 32 threads1f43d046/cc336d0e37f16032/37f16032.pmtiles, 96 threads (default)f31fdd19/6ab2520737f16032/37f16032Both builds are master 2789651 + #966 + #981 + #942. We didn't profile where the time went, but the drop in system CPU points to locking: before, every thread took two global locks for each tile (and called
tellp()under one of them); now one thread at a time writes whole batches. The file is also 32 MB smaller, most likely because the race that let two threads write the same tiny tile is gone. The extra heap is the batches held back. At 96 threads user CPU is about 6% higher, and threads still wait for room now and then; most likely the expensive low-zoom tiles at the start of the order hold up everything behind them. Writing those at the end of the file instead would be a possible follow-up..mbtiles(3 rounds)For MBTiles the time is within run-to-run noise.
Choosing the limit (all open performance PRs + #981 + #942 + this PR, planet in memory as in #977, 96 threads; wait statistics from a debug build):
master, 96 threads64 MB per thread is as fast as using only the physical cores, and peak RSS stays below
master's. Our guess for why it's faster than no limit at all: a thread waiting for room frees its share of the core for its twin. 96-thread runs vary by over a minute, so single runs here are rough. The 32-thread numbers above were measured with 1 GB, which is now 2 GB; that can only mean less waiting, for at most 1 GB more memory.Tests
test/ordered_batches.test.cpp(inmake test): batches finishing out of order, an empty batch, and 8 threads with a small limit so that threads have to wait. Also run 20 times under ThreadSanitizer, with no warnings.Waking waiting threads (fourth commit).
end()used to wake all waiting threads after every batch it wrote, although the held batches were usually still over the limit. At 96 threads, dozens of threads woke up, found no room and went back to sleep, for every batch. Now they're only woken once there is room again. Output unchanged.The limit table above was measured before this fix, so its waits include these wake-ups. With the fix, 128 MB per thread is no faster than 64 MB (11:04 vs 11:05 on our current stack) and needs 7 GB more peak RSS, so 64 MB stays.
Placing and writing in two steps (fifth commit). With the wake-up fix in, the one thread that writes batches in order became the bottleneck at 96 threads: busy 260 s of a 323 s tile phase, with the others waiting for room for 5,100 thread-seconds. More than half of its time went into writing the bytes (
ofstream::write, and atellp()per tile). Now the writer keeps its own offset, the stream has a 4 MB buffer, andOrderedBatchestakes an optional second step (flushItem): also in batch order and one thread at a time, but run by whichever thread is free, so placing (offset, repeated tiny tiles, index) and writing run side by side. The file is byte-identical (Germany at 32 and 8 threads, with and without the commit). Parallelpwritefrom all threads was slower (12:11): buffered writes to one file serialize on the inode lock.New test
test_ordered_batches_flush: the second step runs in order, never twice at once, and always after the first (also clean under ThreadSanitizer).With this, #981 and #942, the
verify-generated-tilesCI job could compare files byte for byte again.🤖 Generated with Claude Code