Skip to content

test: add unit tests for VaultGemma adapter - #2024

Merged
jlarson4 merged 2 commits into
TransformerLensOrg:devfrom
adityasingh-ops:test/vaultgemma-adapter-tests
Nov 17, 2026
Merged

jlarson4 merged 2 commits into
TransformerLensOrg:devfrom
adityasingh-ops:test/vaultgemma-adapter-tests

Conversation

@adityasingh-ops

Copy link
Copy Markdown

Description

VaultGemma is one of the few adapters in supported_architectures/ with no unit test file. This adds test_vaultgemma_adapter.py.

The adapter is Gemma 2 with post_attention_layernorm and post_feedforward_layernorm removed, so the tests cover what the subclass changes:

  • the block only has ln1 / ln2, mapped to input_layernorm / pre_feedforward_layernorm
  • hook_attn_out / hook_mlp_out resolve to attn.hook_out / mlp.hook_out, and no alias targets an ln*_post that no longer exists
  • every HF name in the block mapping exists on a real VaultGemmaDecoderLayer. It's built from a tiny config, so nothing is downloaded (google/vaultgemma-1b is gated, so a hub-backed test wouldn't run in CI without a token)
  • compat mode stays gated off (supports_compatibility_mode=False, phase 3 not applicable), while Gemma 2 still has it on

To check the tests actually catch regressions, I broke the adapter two ways. Removing the _block_norms override, so the post-norms come back, fails 3 of the 5 tests. Pointing ln2 at post_feedforward_layernorm fails 2. Both changes were reverted.

Tests only, no library changes.

Type of change

  • New feature (non-breaking change which adds functionality)

Checklist:

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

VaultGemma had no adapter tests. Covers the bits that differ from Gemma 2:
only pre-norms in the block, hook_attn_out/hook_mlp_out aliases not pointing
at the dropped post-norms, block mapping names matching the HF decoder layer,
and compat mode being gated off.

@jlarson4 jlarson4 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one note about duplicate tests due to the original VaultGemma tests being placed in the wrong location. Take a look and let me know if you have any questions

return VaultGemmaArchitectureAdapter(make_bridge_cfg("VaultGemmaForCausalLM", d_mlp=128))


def test_factory_resolves_vaultgemma_arch_string() -> None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VaultGemma has a couple unit tests in test_thin_subclass_adapters.py (test_vaultgemma_drops_post_norms and a line in test_factory_registrations). This test and test_block_has_only_pre_norms are repeated there. Could you remove them from that file to eliminate the duplication?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, missed that file. Removed test_vaultgemma_drops_post_norms and the VaultGemma line from test_factory_registrations in test_thin_subclass_adapters.py, so everything VaultGemma is in the new file now.

…rs.py

They now live in test_vaultgemma_adapter.py.
@jlarson4
jlarson4 merged commit 8671e55 into TransformerLensOrg:dev Nov 17, 2026
31 checks passed
@adityasingh-ops
adityasingh-ops deleted the test/vaultgemma-adapter-tests branch November 18, 2026 04:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants