Skip to content

list_threads_for_doc treats any GSI query error as "missing," falls back to a scan #11

Closed

Description

@jjbaumert

Summary: crates/storage/src/repo/thread_repo.rs::list_threads_for_doc queries the GSI5-docid-updated index and, on any error from that query (throttling, auth/permission error, transient network failure — not just "index doesn't exist"), falls into the Err(_) => arm, which assumes the GSI is missing (a dev-environment-only condition per the doc comment) and silently degrades to a capped 500-row table scan. In production, where the GSI is expected to always exist, this masks real operational problems (DynamoDB throttling, IAM/permission errors) behind a much more expensive scan, with no error-level log line — only a tracing::warn! that fires exclusively when the scan itself hits its row cap, not when the GSI query failed in the first place.

Severity: S3 — no direct user-visible breakage under normal conditions (the fallback still returns data), but it's an operational and cost hazard: production incidents (throttling, permission drift) get silently converted into full scans instead of surfacing as query errors, which both increases DynamoDB cost/latency and hides the underlying problem from whoever is on call.

Environment:

  • Deploy: prod / staging (dev environments intentionally lack the GSI per the existing comment, so the fallback is expected there — the bug is that prod can't distinguish "GSI genuinely missing" from "GSI query failed for an unrelated reason")
  • Commit: 6411ddb

Repro steps:

  1. Induce any error on the GSI5-docid-updated query in a production-like environment where the index does exist — e.g. a throttled read, an expired/insufficient IAM credential, or a transient DynamoDB service error.
  2. Call list_threads_for_doc(doc_id).
  3. Observe no error-level log line for the GSI failure itself; the call instead falls through to scan_with_filter and (usually) returns data as if nothing were wrong.

Expected: A GSI query failure in production (where the index is expected to exist) should be logged as an error and/or surfaced distinctly from the "index doesn't exist" dev-environment case — not silently absorbed into a scan fallback.

Observed: Err(_) => at crates/storage/src/repo/thread_repo.rs:104 matches every error variant identically and proceeds straight to the scan; the only logging in that path is the tracing::warn! at lines 125-130, which fires only if the scan itself is truncated at its 500-row cap — not when the GSI query errored.

Evidence:

  • Code: crates/storage/src/repo/thread_repo.rs:93-131 (list_threads_for_doc), specifically the Err(_) => { ... } arm at line 104 with no tracing::error!/warn! logging the original GSI error or its kind.

Suspected area: crates/storage/src/repo/thread_repo.rs::list_threads_for_doc — should log the GSI error (at minimum) before falling back, and ideally match on the specific "index not found" error kind so throttling/auth errors propagate as real errors instead of triggering the scan fallback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingverification-findingFinding from a verification run

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions