Skip to content

fix(ActivationCache): support half-precision models in projected and full residual decompositions - #2028

Open
Rohit4507 wants to merge 4 commits into
TransformerLensOrg:devfrom
Rohit4507:fix/half-precision-resid-decomposition
Open

Rohit4507 wants to merge 4 commits into
TransformerLensOrg:devfrom
Rohit4507:fix/half-precision-resid-decomposition

Conversation

@Rohit4507

Copy link
Copy Markdown

Summary

Fixes dtype, device, and batchless-LN bugs when running ActivationCache residual/neuron decompositions and projections on bfloat16 / float16 TransformerBridge models.

Root Causes Fixed

  1. Mixed-dtype projections after apply_ln_to_stack (ActivationCache.py): Half-precision norm layers cache hook_scale in float32, so apply_ln_to_stack outputs float32 while directions derived from model weights (e.g. W_U) remain bfloat16/float16 (or vice versa when projecting bf16 activations onto a float32 probe direction), raising RuntimeError: expected scalar type ... on @. Added _project_onto() using torch.promote_types(tensor.dtype, project.dtype) across get_neuron_results, _stack_neuron_results_apply_ln_projected, stack_neuron_results, and get_full_resid_decomposition.
  2. TransformerBridge.accumulated_bias dtype & single-device inference (transformer_bridge.py): torch.zeros(self.cfg.d_model, device=self.cfg.device) previously defaulted to float32 and failed if cfg.device still held "meta" on a materialized single-device model. Infer dtype from cfg.dtype / model parameters and use first_param.device when n_devices <= 1 (preserving cfg.device destination behavior for multi-device setups).
  3. Post-norm bias correction dtype (_post_norm_bias_correction): Preserve residual/parameter dtype instead of upcasting apply_ln=False half-precision decompositions to float32 via scale.dtype on post-norm models (e.g. Gemma-2).
  4. Batchless apply_ln=True decompositions: Pass has_batch_dim=self.has_batch_dim to internal self.apply_ln_to_stack(...) calls in accumulated_resid, decompose_resid, stack_neuron_results, and get_full_resid_decomposition so remove_batch_dim() caches broadcast cached norm scales along the right axis.

Tests & Verification

  • Added test_half_precision_projected_and_full_resid_decompositions (bfloat16/float16 x apply_ln x model-dtype/float32 directions x batched/batchless, including cfg.device = "meta") and test_half_precision_post_norm_full_resid_decomposition (Gemma-2 post-norm in bfloat16/float16) in tests/unit/test_activation_cache.py.
  • pycln --check --all . --exclude "__init__.py", isort --check-only ., black --check . (23.12.1)
  • mypy (0 errors, no # type: ignore added)
  • pytest tests/unit/test_activation_cache.py tests/unit/test_activation_cache_batch_dim.py tests/integration/model_bridge/test_parallel_mlp_input_decomposition.py (386 passed)

Copilot AI balanced review requested due to automatic review settings November 17, 2026 16:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@jlarson4 jlarson4 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two small comments that will need to be resolved before this can be merged, thanks for putting it together!

to target ``layer``'s MLP input, adding its variant bias unless the block is parallel.
"""
accumulated = torch.zeros(self.cfg.d_model, device=self.cfg.device)
first_param = next(self.parameters(), None)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self.parameters() raises on a bridge that has no wrapped module, so accumulated_bias now fails there. The two TestAccumulatedBiasHybrid tests in tests/unit/test_optional_submodule.py hit this, which is what is turning CI red. Can we get the dtype and device from something available on every bridge?

accumulated = torch.zeros(self.cfg.d_model, device=self.cfg.device)
first_param = next(self.parameters(), None)
cfg_dtype = getattr(self.cfg, "dtype", None)
dtype = (

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cfg.dtype is always set, so the parameter branch below never runs, and it is also the value that goes stale: after bridge.half() it still says float32 and this returns a float32 bias for a float16 model. Device already trusts the parameter first. Could dtype follow the same order?

@Rohit4507

Copy link
Copy Markdown
Author

Good catch, switched to walking the bridge's registered submodules via nn.Module.named_parameters(self) (with fallback to self.cfg when a bridge has no registered parameters), so unwrapped and hybrid bridges work without touching original_model. Added parameterless and .half() coverage to TestAccumulatedBiasHybrid.

@Rohit4507

Copy link
Copy Markdown
Author

Updated dtype to prefer the first standard floating-point parameter's dtype (skipping integer and 1-byte quantizer storage) and only fall back to cfg.dtype when none is present. Added regression tests asserting accumulated_bias returns float16 after model.half() while cfg.dtype is still float32.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants