Skip to content

fix(bridge): apply dtype, device and eval() when load_weights=False - #2020

Merged
jlarson4 merged 2 commits into
TransformerLensOrg:devfrom
pablocs116:fix/load-weights-false-dtype-device-eval-2017
Nov 16, 2026
Merged

jlarson4 merged 2 commits into
TransformerLensOrg:devfrom
pablocs116:fix/load-weights-false-dtype-device-eval-2017

Conversation

@pablocs116

@pablocs116 pablocs116 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Description

TransformerBridge.boot_transformers(..., load_weights=False) built the model with from_config and left it on the CPU, in the config's dtype and in training mode. device was only recorded in cfg.device, and dtype was ignored, so cfg.dtype and the weights could disagree. Callers had to cast, move and .eval() the model themselves. With GPT-2's default dropout, two identical forward passes gave different outputs.

The no-weights path now does what the weights path does:

  • eval mode: hf_model.eval(). from_pretrained returns an eval-mode model, but from_config doesn't.
  • dtype: the model is built with from_config(..., dtype=dtype), as from_pretrained does on the weights path, then maybe_cast_floating_params runs as on that path. Buffers built at init, such as XGLM's sinusoidal table, follow the dtype; float32 buffers such as RotaryEmbedding.inv_freq are kept.
  • device: the bridge is moved to device right after it's built. The weights path moves the HF model before building the bridge, but on the no-weights path I move it after. Splitting fused QKV at boot reads the weight values, so moving a GPT-2 to meta first fails with NotImplementedError. meta is what test_random_init_bridge_puts_its_weights_on_the_device_it_records uses to stand in for a GPU, and it's what random_init_bridge itself did (bridge.to(bridge.cfg.device)). from_config already builds on the CPU, so a CPU target is not moved, and the move doesn't print.
  • device_map is still not applied, as agreed in the issue. The parameters stay on the CPU.

The load_weights docstring is updated to match. random_init_bridge in tools/analysis/interchange.py drops its cast / .to / .eval() workaround and just seeds and boots. Its existing tests pass unchanged.

Closes #2017

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • This change requires a documentation update

Checklist:

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

Tests

New TestBridgeWithoutWeights in tests/integration/model_bridge/test_bridge_creation_modes.py. It uses a tiny GPT-2 config through hf_config_overrides, so no weights are downloaded:

  • float16 / bfloat16: every floating parameter has the requested dtype, matching cfg.dtype
  • hf-internal-testing/tiny-random-LlamaForCausalLM in bfloat16: the inv_freq buffers stay float32
  • no module in the bridge or the HF model is in training mode, and two identical forward passes are equal
  • device="meta": cfg.device and every parameter are on meta

Local gates:

  • make check-format and uv run mypy .: clean
  • make unit-test: 8199 passed. The one failure in the first run (test_qwen3_5_adapter.py::TestQwen3_5LoadingGuards::test_load_weights_false_uses_prepared_text_config) came from this change: its stub bridge has no cfg, and .to() reads it. It's fixed by skipping the CPU move, and the test now passes.
  • test_bridge_creation_modes.py, test_interchange.py, test_qwen3_5_adapter.py, test_multi_gpu_bridge.py, test_n_ctx_override.py, test_remote_bridge.py: 162 passed, 3 skipped

Hardware: CPU only (WSL2).

🤖 Generated with Claude Code

@jlarson4 jlarson4 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround on this, the eval and device handling are solid. A couple comments, and you will need to pull in the latest dev elements before this can merge. Let me know if you have any questions!

hf_model.eval()
# Initialise in the config's dtype, then cast like the weights path, so float32
# buffers (e.g. RotaryEmbedding.inv_freq) are preserved
maybe_cast_floating_params(hf_model, dtype)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from_pretrained builds the model under the requested dtype, so buffers created at init follow it. Here, the model is built in the config's dtype and only the parameters are cast, which leaves some architectures mixed. For example, no-weights XGLM or Falcon-H1 booted with dtype=torch.float16 now fail in its first forward pass, where the weights path runs. Can the model be built in the requested dtype, mirroring the weights path?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. I reproduced it: with load_weights=False, dtype=torch.float16, XGLM failed with mixed dtype (CPU) and Falcon-H1 with Input type (float) and bias type (c10::Half). Fixed in 1c8c294. from_config now gets dtype=dtype, as from_pretrained does on the weights path, so buffers built at init follow it. The maybe_cast_floating_params call stays, as on the weights path. inv_freq is still float32, and test_float32_buffers_survive_the_cast still passes. I added test_buffers_built_at_init_follow_the_requested_dtype, which boots a tiny XGLM in float16 and runs a forward pass. It fails without the change.

but not applied, and neither are ``dtype`` nor ``device_map``; move, cast and
``eval()`` the model before running it on another device or in another dtype.
checkpoint: real parameters with the architecture's random initialisation, moved to
``device``, cast to ``dtype`` and in evaluation mode, as with loaded weights.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

benchmarks/main_benchmark.py:750 boots without weights only to read attn_implementation, and passes the run's device and dtype. With this change that probe casts a full random-init model and moves it to the accelerator on every benchmark run. Could you take a look at that call so it stays a cheap CPU probe?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 1c8c294. The probe in main_benchmark.py now boots with device="cpu" and the default dtype, and reads only adapter.cfg.attn_implementation. So it no longer casts the model or moves it to the accelerator. The run's dtype is still detected and applied further down, as before.

torch.default_generator.manual_seed(seed)
bridge = TransformerBridge.boot_transformers(
model_name, load_weights=False, device=device, **boot_kwargs
return TransformerBridge.boot_transformers(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of my work, PR #2018 landed on dev and rewrote this block. boot is no longer passed dtype, and cfg.dtype is set by hand after the cast. Please pull in the latest dev and adjust accordingly, apologies for the conflict I did not mean to overlap with you.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problem! I rebased onto the latest dev (cade4f3). random_init_bridge now passes dtype to boot again and returns the bridge directly. Now that the no-weights path builds in the requested dtype, cfg.dtype already matches the parameters, so the manual cfg.dtype = dtype isn't needed and the "dtype was requested but…" warning doesn't fire. The test_interchange tests pass unchanged.

pablocs116 and others added 2 commits November 14, 2026 10:37
boot_transformers(load_weights=False) built the model with from_config and
left it on the CPU, in the config's dtype and in training mode. It now mirrors
the weights path: eval(), maybe_cast_floating_params(dtype), and a move to
device once the bridge is built (fused-QKV splitting at boot needs real
values, so the move can't precede it). device_map is still not applied.

random_init_bridge drops its cast/.to/.eval() workaround.

Closes TransformerLensOrg#2017

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Address review on TransformerLensOrg#2020:

- from_config now gets dtype=, as from_pretrained does on the weights
  path, so buffers created at init follow it. Casting only the parameters
  afterwards left XGLM's sinusoidal table (and Falcon-H1 buffers) in
  float32, and a float16 no-weights boot failed on its first forward.
- The benchmark's attn_implementation probe boots on the CPU in the
  default dtype, so it no longer casts a random-init model and moves it
  to the accelerator on every run.
- Update comments that said the no-weights path builds in the config's
  dtype.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pablocs116
pablocs116 force-pushed the fix/load-weights-false-dtype-device-eval-2017 branch from 0598838 to 1c8c294 Compare November 14, 2026 10:36
@pablocs116

Copy link
Copy Markdown
Contributor Author

Thanks for the review! I rebased onto the latest dev and addressed all three points in 1c8c294 (details inline). Local checks: make check-format and mypy . are clean. make unit-test: 8443 passed. test_bridge_creation_modes.py, test_interchange.py, test_qwen3_5_adapter.py, test_multi_gpu_bridge.py, test_n_ctx_override.py, test_remote_bridge.py and test_xglm_pos_embed.py: 238 passed, 3 skipped. CPU only.

@jlarson4

Copy link
Copy Markdown
Collaborator

Thank you for getting to those edits @pablocs116, great work. Consider this PR complete, I am merging now

@jlarson4
jlarson4 merged commit 88ea568 into TransformerLensOrg:dev Nov 16, 2026
31 checks passed
jlarson4 added a commit that referenced this pull request Nov 16, 2026
Brings in #2019 (explicit dtype on the from-module path), #2020 (dtype,
device and eval() on the no-weights path), #2021 and #2023.

Conflicts resolved in _bridge_builder.py (one multi_gpu import; the docstring
covers both the dtype cast and the device move), source.py (dev's
load_weights docstring) and interchange.py (dev's random_init_bridge, since
boot now applies dtype and device itself).

Follow-ons forced by #2020: boot's no-weights device read-back and its test
are dropped, because that path now moves the model and cfg.device follows the
move; the random-init interchange test returns to dev's version.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants