Repository navigation
fix(bridge): apply dtype, device and eval() when load_weights=False - #2020
Conversation
jlarson4
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround on this, the eval and device handling are solid. A couple comments, and you will need to pull in the latest dev elements before this can merge. Let me know if you have any questions!
| hf_model.eval() | ||
| # Initialise in the config's dtype, then cast like the weights path, so float32 | ||
| # buffers (e.g. RotaryEmbedding.inv_freq) are preserved | ||
| maybe_cast_floating_params(hf_model, dtype) |
There was a problem hiding this comment.
from_pretrained builds the model under the requested dtype, so buffers created at init follow it. Here, the model is built in the config's dtype and only the parameters are cast, which leaves some architectures mixed. For example, no-weights XGLM or Falcon-H1 booted with dtype=torch.float16 now fail in its first forward pass, where the weights path runs. Can the model be built in the requested dtype, mirroring the weights path?
There was a problem hiding this comment.
Good catch, thanks. I reproduced it: with load_weights=False, dtype=torch.float16, XGLM failed with mixed dtype (CPU) and Falcon-H1 with Input type (float) and bias type (c10::Half). Fixed in 1c8c294. from_config now gets dtype=dtype, as from_pretrained does on the weights path, so buffers built at init follow it. The maybe_cast_floating_params call stays, as on the weights path. inv_freq is still float32, and test_float32_buffers_survive_the_cast still passes. I added test_buffers_built_at_init_follow_the_requested_dtype, which boots a tiny XGLM in float16 and runs a forward pass. It fails without the change.
| but not applied, and neither are ``dtype`` nor ``device_map``; move, cast and | ||
| ``eval()`` the model before running it on another device or in another dtype. | ||
| checkpoint: real parameters with the architecture's random initialisation, moved to | ||
| ``device``, cast to ``dtype`` and in evaluation mode, as with loaded weights. |
There was a problem hiding this comment.
benchmarks/main_benchmark.py:750 boots without weights only to read attn_implementation, and passes the run's device and dtype. With this change that probe casts a full random-init model and moves it to the accelerator on every benchmark run. Could you take a look at that call so it stays a cheap CPU probe?
There was a problem hiding this comment.
Done in 1c8c294. The probe in main_benchmark.py now boots with device="cpu" and the default dtype, and reads only adapter.cfg.attn_implementation. So it no longer casts the model or moves it to the accelerator. The run's dtype is still detected and applied further down, as before.
| torch.default_generator.manual_seed(seed) | ||
| bridge = TransformerBridge.boot_transformers( | ||
| model_name, load_weights=False, device=device, **boot_kwargs | ||
| return TransformerBridge.boot_transformers( |
There was a problem hiding this comment.
Some of my work, PR #2018 landed on dev and rewrote this block. boot is no longer passed dtype, and cfg.dtype is set by hand after the cast. Please pull in the latest dev and adjust accordingly, apologies for the conflict I did not mean to overlap with you.
There was a problem hiding this comment.
No problem! I rebased onto the latest dev (cade4f3). random_init_bridge now passes dtype to boot again and returns the bridge directly. Now that the no-weights path builds in the requested dtype, cfg.dtype already matches the parameters, so the manual cfg.dtype = dtype isn't needed and the "dtype was requested but…" warning doesn't fire. The test_interchange tests pass unchanged.
boot_transformers(load_weights=False) built the model with from_config and left it on the CPU, in the config's dtype and in training mode. It now mirrors the weights path: eval(), maybe_cast_floating_params(dtype), and a move to device once the bridge is built (fused-QKV splitting at boot needs real values, so the move can't precede it). device_map is still not applied. random_init_bridge drops its cast/.to/.eval() workaround. Closes TransformerLensOrg#2017 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Address review on TransformerLensOrg#2020: - from_config now gets dtype=, as from_pretrained does on the weights path, so buffers created at init follow it. Casting only the parameters afterwards left XGLM's sinusoidal table (and Falcon-H1 buffers) in float32, and a float16 no-weights boot failed on its first forward. - The benchmark's attn_implementation probe boots on the CPU in the default dtype, so it no longer casts a random-init model and moves it to the accelerator on every run. - Update comments that said the no-weights path builds in the config's dtype. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0598838 to
1c8c294
Compare
|
Thanks for the review! I rebased onto the latest |
|
Thank you for getting to those edits @pablocs116, great work. Consider this PR complete, I am merging now |
Brings in #2019 (explicit dtype on the from-module path), #2020 (dtype, device and eval() on the no-weights path), #2021 and #2023. Conflicts resolved in _bridge_builder.py (one multi_gpu import; the docstring covers both the dtype cast and the device move), source.py (dev's load_weights docstring) and interchange.py (dev's random_init_bridge, since boot now applies dtype and device itself). Follow-ons forced by #2020: boot's no-weights device read-back and its test are dropped, because that path now moves the model and cfg.device follows the move; the random-init interchange test returns to dev's version. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Description
TransformerBridge.boot_transformers(..., load_weights=False)built the model withfrom_configand left it on the CPU, in the config's dtype and in training mode.devicewas only recorded incfg.device, anddtypewas ignored, socfg.dtypeand the weights could disagree. Callers had to cast, move and.eval()the model themselves. With GPT-2's default dropout, two identical forward passes gave different outputs.The no-weights path now does what the weights path does:
hf_model.eval().from_pretrainedreturns an eval-mode model, butfrom_configdoesn't.from_config(..., dtype=dtype), asfrom_pretraineddoes on the weights path, thenmaybe_cast_floating_paramsruns as on that path. Buffers built at init, such as XGLM's sinusoidal table, follow the dtype; float32 buffers such asRotaryEmbedding.inv_freqare kept.deviceright after it's built. The weights path moves the HF model before building the bridge, but on the no-weights path I move it after. Splitting fused QKV at boot reads the weight values, so moving a GPT-2 tometafirst fails withNotImplementedError.metais whattest_random_init_bridge_puts_its_weights_on_the_device_it_recordsuses to stand in for a GPU, and it's whatrandom_init_bridgeitself did (bridge.to(bridge.cfg.device)).from_configalready builds on the CPU, so a CPU target is not moved, and the move doesn't print.The
load_weightsdocstring is updated to match.random_init_bridgeintools/analysis/interchange.pydrops its cast /.to/.eval()workaround and just seeds and boots. Its existing tests pass unchanged.Closes #2017
Type of change
Checklist:
Tests
New
TestBridgeWithoutWeightsintests/integration/model_bridge/test_bridge_creation_modes.py. It uses a tiny GPT-2 config throughhf_config_overrides, so no weights are downloaded:cfg.dtypehf-internal-testing/tiny-random-LlamaForCausalLMin bfloat16: theinv_freqbuffers stay float32device="meta":cfg.deviceand every parameter are onmetaLocal gates:
make check-formatanduv run mypy .: cleanmake unit-test: 8199 passed. The one failure in the first run (test_qwen3_5_adapter.py::TestQwen3_5LoadingGuards::test_load_weights_false_uses_prepared_text_config) came from this change: its stub bridge has nocfg, and.to()reads it. It's fixed by skipping the CPU move, and the test now passes.test_bridge_creation_modes.py,test_interchange.py,test_qwen3_5_adapter.py,test_multi_gpu_bridge.py,test_n_ctx_override.py,test_remote_bridge.py: 162 passed, 3 skippedHardware: CPU only (WSL2).
🤖 Generated with Claude Code