Repository navigation
[core] Reuse the decoded data block across sorted Lookup File lookups - #1456
Open
JeonDaehong wants to merge 1 commit into
Open
JeonDaehong wants to merge 1 commit into
JeonDaehong wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: none
Lookup compaction looks up every changed key in the Lookup File of the older levels. Each
LookupFileCache::readreads the key's SST data block from the local cache, checks its CRC and decompresses it again, even though the parsed index block and the decoded IPC payload are already kept. Compaction looks keys up in sorted order, so consecutive keys almost always fall in the same data block. On a deletion-vector table, compacting 667k updated keys against one 2M-row file takes about 19 s, with zstd decompression at the top of the profile. Java'sSortLookupStoreReaderkeeps decompressed blocks in itsBlockCache.Brief change log
LookupFileCachekeeps the decoded data block of the previous lookup, keyed by Lookup File path and block offset, next to the parsed index and the decoded IPC payload. A lookup in the same block reuses it, and any other block replaces it. The block is checked against the same memory budget and dropped with the others when a read fails.PK table with
deletion-vectors.enabled, 2M keys compacted to one file, then every third key updated and compacted with lookups against that file, in-memory FileIO, release build. Measured in one binary with the cache switched off by an env var, over 3 alternating rounds: the second compaction goes from 19268–19337 to 5480–5509 ms. Full reads are unchanged.Tests
lookups_across_data_blocks_switch_the_cached_block: 20,000 keys over many 8 KiB blocks, looked up ascending, descending and outside the key range. Every key resolves, and an ascending sweep reads far fewer blocks than keys. Caching by path alone, ignoring the block offset, fails it along withinterleaved_payloads_and_sst_blocks_support_point_lookups.parsed_index_is_reused_and_missing_data_still_rebuilds_its_generationnow expects one read per key (the head), and drops the resident block before checking that missing data rebuilds the generation.Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go).
API and Format
None.
Documentation
None.