Skip to content

fix: let detect_head accept a sequence that fills the context window - #2027

Open
faisalhacks wants to merge 1 commit into
TransformerLensOrg:devfrom
faisalhacks:fix/detect-head-full-context
Open

faisalhacks wants to merge 1 commit into
TransformerLensOrg:devfrom
faisalhacks:fix/detect-head-full-context

Conversation

@faisalhacks

Copy link
Copy Markdown

Description

detect_head tokenized with the default truncate=True, so a sequence longer than n_ctx was cut to exactly n_ctx tokens before the length check 1 < len < n_ctx. To reject those, the check had to reject every n_ctx-token sequence, including ones that fit exactly (the error message says the sequence "must fit within the model's context window", and n_ctx tokens do).

This tokenizes with truncate=False and checks len <= n_ctx:

  • a sequence that fills the context window exactly now runs;
  • anything longer keeps its real length and is still rejected.

The existing test_detect_head_with_sequence_length_outside_context_window passed "a " * n_ctx, which on solu-2l is exactly n_ctx (1024) tokens, so it was really testing an exact fit. It now passes one more. Two tests are added: an exact fit (" a" * (n_ctx - 1) plus BOS, asserted to be 1024 tokens) returns [n_layers, n_heads], and "a " * (2 * n_ctx) is rejected.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing tests pass with my changes: tests/integration/test_head_detector.py and tests/unit (8659 passed, 62 skipped); the new exact-fit test fails on dev without the fix
  • black --check, isort --check-only and mypy transformer_lens/head_detector.py are clean

🤖 Generated with Claude Code

detect_head tokenized with truncation, so a sequence longer than n_ctx was
cut to exactly n_ctx tokens, and the check 1 < len < n_ctx then rejected
every n_ctx-token sequence to catch those. That also rejected sequences
that fit exactly. Tokenize without truncation and accept len <= n_ctx: an
exact fit now runs, and anything longer is still rejected. The existing
"outside context window" test passed "a " * n_ctx, which is exactly n_ctx
tokens; it now passes one more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jlarson4 jlarson4 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @faisalhacks, thanks for putting this together. One important change relating to the truncation, let me know if you have any questions.


# Validate inputs and detection pattern shape
assert 1 < tokens.shape[-1] < cfg.n_ctx, SEQ_LEN_ERR
assert 1 < tokens.shape[-1] <= cfg.n_ctx, SEQ_LEN_ERR

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With truncation gone, an over-long prompt reaches this check after the detection pattern has been built at its full length. Rejecting it now costs time and memory that grow relative to the prompt. A 20,000-token prompt takes up to a minute and around 12 GB before the assertion fires, where it used to fail instantly. Can we validate length before anything is sized from tokens?

def test_detect_head_with_sequence_length_outside_context_window():
with pytest.raises(AssertionError) as e:
detect_head(model, "a " * model.cfg.n_ctx, "previous_token_head")
# "a " * n_ctx is exactly n_ctx tokens, which fits; one more does not.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"a " * n_ctx is 1,026 tokens here, not n_ctx. It only looked like an exact fit because truncation was cutting it down. That makes this string 1,027 tokens, nothing tests a prompt that is over by exactly one. Could this case be built and asserted the way the exact-fit test below is?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants