Repository navigation
SOLR-18664: Add lookupFilter to the graph query parser - #5356
Open
rahulgoswami wants to merge 1 commit into
Open
rahulgoswami wants to merge 1 commit into
rahulgoswami wants to merge 1 commit into
Conversation
Documents reached by the traversal are still collected, but only those matching lookupFilter have their edges followed to the next hop. This avoids adding ids of nodes that can't have children (e.g. files in a file/folder tree) to every hop's frontier query. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rahulgoswami
force-pushed
the
SOLR-18664-graph-lookup-filter
branch
from
November 17, 2026 03:54
605e326 to
ae9bd9f
Compare
|
Automated PR readiness judgements (LLM-generated; may be wrong; not updated on later pushes).
|
rahulgoswami
marked this pull request as draft
November 17, 2026 04:06
rahulgoswami
marked this pull request as ready for review
November 18, 2026 04:22
|
Automated PR readiness judgements (LLM-generated; may be wrong; not updated on later pushes).
|
Member
Author
|
@dsmiley @gerlowskija @mkhludnev Would appreciate a review please. Thanks. |
mkhludnev
approved these changes
Nov 18, 2026
mkhludnev
left a comment
Member
There was a problem hiding this comment.
It's clear and effective. One small suggestion: consider renaming it to expandFilter — I think that better conveys what it does. (Pardon the nitpick.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/SOLR-18664
Description
{!graph}does a breadth-first traversal: at each hop it reads thetofield of every collected document and adds those values to the query used to find the next level,from IN (...). There is no way to collect a node without also expanding it.In a file/folder hierarchy (
{!graph from=parentId to=id traversalFilter=name:prefix*}), only folders can have children, but every file'sidstill becomes a frontier term. Each one costs a docValues read, a hash insert and a terms-dictionary seek per segment, and can never match.traversalFilter=docType:folderwould drop the files from the results. The existing leaf-node optimization doesn't help either, because it only skips documents with no value into, and here every document has one.Solution
A new optional
lookupFilterlocal param. Documents reached by the traversal (and matchingtraversalFilter) are always collected, but only those that also matchlookupFilterhave their edges followed. It applies at every level, including the root. Without the param, behaviour is unchanged.GraphQuerygets the filter's DocSet once per request (via the filterCache) and passes it to the edge collectors.GraphEdgeCollector.collect()checks it only before extracting edges; collection itself is unconditional.graphdomain goes throughGraphQueryParser, so it supports the param too.lookupFilterentry in the graph query parser parameters.This change was developed with Claude Code (implementation, tests and benchmark); root cause analyzed and changes reviewed by me.
Benchmark
Synthetic 1M-doc file-system index: 80k folders, 920k files (~11.5 files per folder), depth up to 15. JDK 21, 4 GB heap, standalone,
rows=0, queryResultCache disabled so every query does the full traversal. Each session ran in a fresh JVM with 10 warmup + 30 measured iterations per root and arm, interleaved. Session order: patched, baseline (unpatched build of the same commit), patched.The query was run from 18 starting folders whose results range from ~100 documents to the whole tree. Each row covers folders that return a similar number of documents (shown as the average
numFound). For each folder we take the median QTime (in ms) of its 60 runs; the row shows the geometric mean of those medians:numFound)lookupFilter(ms)lookupFilter=docType:folder(ms)* QTime is reported in whole milliseconds, so these two rows are coarse (due to rounding).
numFoundcomputed independently from the generated tree, and all variants agree.Tests
GraphQueryTest.doGraph()(runs across the existing string, docValues-only string and int/long/float/double point field matrix, so both collectors are covered):lookupFilteris returned but not expanded;lookupFilter;traversalFilter;maxDepthandreturnRoot=false../gradlew tidy,./gradlew check -x test,TestJsonFacetsand allorg.apache.solr.search.join.*tests.Checklist
mainbranch../gradlew check.🤖 Generated with Claude Code