Skip to content

fix: preserve mask-derived positions during residual re-entry - #2023

Merged
jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
emerardd:fix/residual-reentry-mask-positions
Nov 16, 2026
Merged

jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
emerardd:fix/residual-reentry-mask-positions

Conversation

@emerardd

Copy link
Copy Markdown
Contributor

Description

Preserve mask-derived position coordinates when resuming a forward with start_at_layer.

Residual re-entry travels through inputs_embeds, but the mask-position derivation excluded every embeddings input. A cached Llama residual resumed with the original attention mask could therefore use physical column positions instead of the compressed positions used in the token forward. An interior mask gap changes the relative RoPE coordinates: an unmodified residual no longer reproduces the original logits.

With a random tiny Llama (3 layers, width 32, 4 query/2 KV heads), prompt [1, 2, 0, 3, 4] and mask [1, 1, 0, 1, 1], attended-token logit errors at resumed layers 0/1/2 were 5.54e-4, 2.29e-4 and 1.60e-4. Supplying only the original derived positions reduced all three errors to zero.

Allow the existing 2-D-mask position policy to handle residual inputs when start_at_layer is set. Explicit position_ids and the target-model compatibility gate still take precedence; ordinary external inputs_embeds behavior is unchanged. The start_at_layer docstring now describes replaying the original mask and any explicit positions.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Validation

  • Before the fix: 12 failed, 72 passed in the new test file.
  • After the fix: 104 passed across the new tests, position-injection gate and audio re-entry guard tests.
  • 47 passed in the existing start-at-layer integration and generation position-id tests.
  • 84 new cases cover tiny config-built Llama and GPT-2, every resumed layer and negative layer indices, gap/left/right/all-ones/mixed masks, explicit positions, downstream caches, input preservation and unchanged external-embedding/model-gating behavior.
  • Full mypy .: 391 source files passed.
  • Changed-file pycln/isort/Black and git diff --check passed.

The complete suite and GPU/MPS checks were not run. Existing SWIG deprecation warnings remain.

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

Only the affected unit-test surface was run locally; the full-unit-suite checkbox is intentionally unchecked. Documentation changes are in the API docstring.

@jlarson4

Copy link
Copy Markdown
Collaborator

Looks great @emerardd, merging as is!

@jlarson4
jlarson4 merged commit 52e6d47 into TransformerLensOrg:dev Nov 16, 2026
31 checks passed
jlarson4 added a commit that referenced this pull request Nov 16, 2026
Brings in #2019 (explicit dtype on the from-module path), #2020 (dtype,
device and eval() on the no-weights path), #2021 and #2023.

Conflicts resolved in _bridge_builder.py (one multi_gpu import; the docstring
covers both the dtype cast and the device move), source.py (dev's
load_weights docstring) and interchange.py (dev's random_init_bridge, since
boot now applies dtype and device itself).

Follow-ons forced by #2020: boot's no-weights device read-back and its test
are dropped, because that path now moves the model and cfg.device follows the
move; the random-init interchange test returns to dev's version.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants