Skip to content

SOLR-18664: Add lookupFilter to the graph query parser - #5356

Open
rahulgoswami wants to merge 1 commit into
apache:mainfrom
rahulgoswami:SOLR-18664-graph-lookup-filter
Open

rahulgoswami wants to merge 1 commit into
apache:mainfrom
rahulgoswami:SOLR-18664-graph-lookup-filter

Conversation

@rahulgoswami

@rahulgoswami rahulgoswami commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

https://issues.apache.org/jira/browse/SOLR-18664

Description

{!graph} does a breadth-first traversal: at each hop it reads the to field of every collected document and adds those values to the query used to find the next level, from IN (...). There is no way to collect a node without also expanding it.

In a file/folder hierarchy ({!graph from=parentId to=id traversalFilter=name:prefix*}), only folders can have children, but every file's id still becomes a frontier term. Each one costs a docValues read, a hash insert and a terms-dictionary seek per segment, and can never match. traversalFilter=docType:folder would drop the files from the results. The existing leaf-node optimization doesn't help either, because it only skips documents with no value in to, and here every document has one.

Solution

A new optional lookupFilter local param. Documents reached by the traversal (and matching traversalFilter) are always collected, but only those that also match lookupFilter have their edges followed. It applies at every level, including the root. Without the param, behaviour is unchanged.

{!graph from=parentId to=id traversalFilter=name:prefix* lookupFilter=docType:folder}<root query>
  • GraphQuery gets the filter's DocSet once per request (via the filterCache) and passes it to the edge collectors. GraphEdgeCollector.collect() checks it only before extracting edges; collection itself is unconditional.
  • The JSON Facet graph domain goes through GraphQueryParser, so it supports the param too.
  • Ref guide: new lookupFilter entry in the graph query parser parameters.

This change was developed with Claude Code (implementation, tests and benchmark); root cause analyzed and changes reviewed by me.

Benchmark

Synthetic 1M-doc file-system index: 80k folders, 920k files (~11.5 files per folder), depth up to 15. JDK 21, 4 GB heap, standalone, rows=0, queryResultCache disabled so every query does the full traversal. Each session ran in a fresh JVM with 10 warmup + 30 measured iterations per root and arm, interleaved. Session order: patched, baseline (unpatched build of the same commit), patched.

The query was run from 18 starting folders whose results range from ~100 documents to the whole tree. Each row covers folders that return a similar number of documents (shown as the average numFound). For each folder we take the median QTime (in ms) of its 60 runs; the row shows the geometric mean of those medians:

Docs returned (avg numFound) Starting folders Without lookupFilter (ms) With lookupFilter=docType:folder (ms) Speedup (same build)
101 3 2.6 1.3 2.1×*
884 3 7.6 2.3 3.3×*
10,214 3 53 7.8 6.8×
42,546 3 174 24 7.4×
93,708 3 349 42 8.3×
217,834 2 656 73 8.9×
828,781 (whole tree) 1 3,038 298 10.2×

* QTime is reported in whole milliseconds, so these two rows are coarse (due to rounding).

  • Correctness: all 5,760 queries returned the numFound computed independently from the generated tree, and all variants agree.
  • Why: for the whole tree the frontier queries carry 12× fewer terms in total (828,781 → 68,600). What remains is collecting the documents themselves.
  • No regression without the param: unpatched vs. patched ranged from 0.98× to 1.18× by bucket. An index-time control that runs identical code on both builds showed the same per-bucket differences, so these come from the baseline JVM session, not the change.

Tests

  • GraphQueryTest.doGraph() (runs across the existing string, docValues-only string and int/long/float/double point field matrix, so both collectors are covered):
    • a node with children that doesn't match lookupFilter is returned but not expanded;
    • a pure-negative lookupFilter;
    • a root that doesn't match only returns the root;
    • combined with traversalFilter;
    • combined with maxDepth and returnRoot=false.
  • Ran ./gradlew tidy, ./gradlew check -x test, TestJsonFacets and all org.apache.solr.search.join.* tests.

Checklist

  • I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
  • I have created a Jira issue and added the issue ID to my pull request title.
  • I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
  • I have developed this patch against the main branch.
  • I have run ./gradlew check.
  • I have added tests for my changes.
  • I have added documentation for the Reference Guide
  • I have added a changelog entry for my change

🤖 Generated with Claude Code

@github-actions github-actions Bot added documentation Improvements or additions to documentation tests cat:search labels Nov 17, 2026
Documents reached by the traversal are still collected, but only those
matching lookupFilter have their edges followed to the next hop. This
avoids adding ids of nodes that can't have children (e.g. files in a
file/folder tree) to every hop's frontier query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rahulgoswami
rahulgoswami force-pushed the SOLR-18664-graph-lookup-filter branch from 605e326 to ae9bd9f Compare November 17, 2026 03:54
@github-actions

Copy link
Copy Markdown

Automated PR readiness judgements (LLM-generated; may be wrong; not updated on later pushes).

@rahulgoswami
rahulgoswami marked this pull request as draft November 17, 2026 04:06
@rahulgoswami rahulgoswami added this to the 9.x milestone Nov 17, 2026
@rahulgoswami
rahulgoswami marked this pull request as ready for review November 18, 2026 04:22
@github-actions

Copy link
Copy Markdown

Automated PR readiness judgements (LLM-generated; may be wrong; not updated on later pushes).

@rahulgoswami

Copy link
Copy Markdown
Member Author

@dsmiley @gerlowskija @mkhludnev Would appreciate a review please. Thanks.

@mkhludnev mkhludnev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's clear and effective. One small suggestion: consider renaming it to expandFilter — I think that better conveys what it does. (Pardon the nitpick.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cat:search documentation Improvements or additions to documentation tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants