Repository navigation
fix: let detect_head accept a sequence that fills the context window - #2027
faisalhacks wants to merge 1 commit into
Conversation
detect_head tokenized with truncation, so a sequence longer than n_ctx was cut to exactly n_ctx tokens, and the check 1 < len < n_ctx then rejected every n_ctx-token sequence to catch those. That also rejected sequences that fit exactly. Tokenize without truncation and accept len <= n_ctx: an exact fit now runs, and anything longer is still rejected. The existing "outside context window" test passed "a " * n_ctx, which is exactly n_ctx tokens; it now passes one more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jlarson4
left a comment
There was a problem hiding this comment.
Hi @faisalhacks, thanks for putting this together. One important change relating to the truncation, let me know if you have any questions.
|
|
||
| # Validate inputs and detection pattern shape | ||
| assert 1 < tokens.shape[-1] < cfg.n_ctx, SEQ_LEN_ERR | ||
| assert 1 < tokens.shape[-1] <= cfg.n_ctx, SEQ_LEN_ERR |
There was a problem hiding this comment.
With truncation gone, an over-long prompt reaches this check after the detection pattern has been built at its full length. Rejecting it now costs time and memory that grow relative to the prompt. A 20,000-token prompt takes up to a minute and around 12 GB before the assertion fires, where it used to fail instantly. Can we validate length before anything is sized from tokens?
| def test_detect_head_with_sequence_length_outside_context_window(): | ||
| with pytest.raises(AssertionError) as e: | ||
| detect_head(model, "a " * model.cfg.n_ctx, "previous_token_head") | ||
| # "a " * n_ctx is exactly n_ctx tokens, which fits; one more does not. |
There was a problem hiding this comment.
"a " * n_ctx is 1,026 tokens here, not n_ctx. It only looked like an exact fit because truncation was cutting it down. That makes this string 1,027 tokens, nothing tests a prompt that is over by exactly one. Could this case be built and asserted the way the exact-fit test below is?
Description
detect_headtokenized with the defaulttruncate=True, so a sequence longer thann_ctxwas cut to exactlyn_ctxtokens before the length check1 < len < n_ctx. To reject those, the check had to reject everyn_ctx-token sequence, including ones that fit exactly (the error message says the sequence "must fit within the model's context window", andn_ctxtokens do).This tokenizes with
truncate=Falseand checkslen <= n_ctx:The existing
test_detect_head_with_sequence_length_outside_context_windowpassed"a " * n_ctx, which on solu-2l is exactlyn_ctx(1024) tokens, so it was really testing an exact fit. It now passes one more. Two tests are added: an exact fit (" a" * (n_ctx - 1)plus BOS, asserted to be 1024 tokens) returns[n_layers, n_heads], and"a " * (2 * n_ctx)is rejected.Type of change
Checklist
tests/integration/test_head_detector.pyandtests/unit(8659 passed, 62 skipped); the new exact-fit test fails ondevwithout the fixblack --check,isort --check-onlyandmypy transformer_lens/head_detector.pyare clean🤖 Generated with Claude Code