Repository navigation
fix(ActivationCache): support half-precision models in projected and full residual decompositions - #2028
fix(ActivationCache): support half-precision models in projected and full residual decompositions#2028Rohit4507 wants to merge 4 commits into
Conversation
…full residual decompositions
jlarson4
left a comment
There was a problem hiding this comment.
Two small comments that will need to be resolved before this can be merged, thanks for putting it together!
| to target ``layer``'s MLP input, adding its variant bias unless the block is parallel. | ||
| """ | ||
| accumulated = torch.zeros(self.cfg.d_model, device=self.cfg.device) | ||
| first_param = next(self.parameters(), None) |
There was a problem hiding this comment.
self.parameters() raises on a bridge that has no wrapped module, so accumulated_bias now fails there. The two TestAccumulatedBiasHybrid tests in tests/unit/test_optional_submodule.py hit this, which is what is turning CI red. Can we get the dtype and device from something available on every bridge?
| accumulated = torch.zeros(self.cfg.d_model, device=self.cfg.device) | ||
| first_param = next(self.parameters(), None) | ||
| cfg_dtype = getattr(self.cfg, "dtype", None) | ||
| dtype = ( |
There was a problem hiding this comment.
cfg.dtype is always set, so the parameter branch below never runs, and it is also the value that goes stale: after bridge.half() it still says float32 and this returns a float32 bias for a float16 model. Device already trusts the parameter first. Could dtype follow the same order?
…ameters Refactor parameter retrieval to use named_parameters for better compatibility with bridges without wrapped modules.
|
Good catch, switched to walking the bridge's registered submodules via nn.Module.named_parameters(self) (with fallback to self.cfg when a bridge has no registered parameters), so unwrapped and hybrid bridges work without touching original_model. Added parameterless and .half() coverage to TestAccumulatedBiasHybrid. |
|
Updated dtype to prefer the first standard floating-point parameter's dtype (skipping integer and 1-byte quantizer storage) and only fall back to cfg.dtype when none is present. Added regression tests asserting accumulated_bias returns float16 after model.half() while cfg.dtype is still float32. |
Summary
Fixes dtype, device, and batchless-LN bugs when running
ActivationCacheresidual/neuron decompositions and projections onbfloat16/float16TransformerBridgemodels.Root Causes Fixed
apply_ln_to_stack(ActivationCache.py): Half-precision norm layers cachehook_scaleinfloat32, soapply_ln_to_stackoutputsfloat32while directions derived from model weights (e.g.W_U) remainbfloat16/float16(or vice versa when projectingbf16activations onto afloat32probe direction), raisingRuntimeError: expected scalar type ...on@. Added_project_onto()usingtorch.promote_types(tensor.dtype, project.dtype)acrossget_neuron_results,_stack_neuron_results_apply_ln_projected,stack_neuron_results, andget_full_resid_decomposition.TransformerBridge.accumulated_biasdtype & single-device inference (transformer_bridge.py):torch.zeros(self.cfg.d_model, device=self.cfg.device)previously defaulted tofloat32and failed ifcfg.devicestill held"meta"on a materialized single-device model. Inferdtypefromcfg.dtype/ model parameters and usefirst_param.devicewhenn_devices <= 1(preservingcfg.devicedestination behavior for multi-device setups)._post_norm_bias_correction): Preserve residual/parameterdtypeinstead of upcastingapply_ln=Falsehalf-precision decompositions tofloat32viascale.dtypeon post-norm models (e.g. Gemma-2).apply_ln=Truedecompositions: Passhas_batch_dim=self.has_batch_dimto internalself.apply_ln_to_stack(...)calls inaccumulated_resid,decompose_resid,stack_neuron_results, andget_full_resid_decompositionsoremove_batch_dim()caches broadcast cached norm scales along the right axis.Tests & Verification
test_half_precision_projected_and_full_resid_decompositions(bfloat16/float16xapply_lnx model-dtype/float32directions x batched/batchless, includingcfg.device = "meta") andtest_half_precision_post_norm_full_resid_decomposition(Gemma-2 post-norm inbfloat16/float16) intests/unit/test_activation_cache.py.pycln --check --all . --exclude "__init__.py",isort --check-only .,black --check .(23.12.1)mypy(0 errors, no# type: ignoreadded)pytest tests/unit/test_activation_cache.py tests/unit/test_activation_cache_batch_dim.py tests/integration/model_bridge/test_parallel_mlp_input_decomposition.py(386 passed)