Skip to content

Added minimal TransformerBridge support for Qwen2MoeForCausalLM#1477

Merged
jlarson4 merged 3 commits into
TransformerLensOrg:mainfrom
Canonik:add-qwen2-moe-bridge
Jul 2, 2026
Merged

Added minimal TransformerBridge support for Qwen2MoeForCausalLM#1477
jlarson4 merged 3 commits into
TransformerLensOrg:mainfrom
Canonik:add-qwen2-moe-bridge

Conversation

@Canonik

@Canonik Canonik commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Description

Added minimal TransformerBridge support for "Qwen2MoeForCausalLM".

This adapter reuses the existing Qwen2 attention path and existing MoE bridge components, while adding the Qwen2-MoE-specific shared expert path.

The PR includes :

  • registering/detecting "Qwen2MoeForCausalLM"
  • mapping Qwen2-MoE config fields from HF
  • Qwen2-style attention/GQA plumbing
  • routed expert MoE path
  • shared expert MLP path
  • shared expert gate
  • local tiny config HF parity tests
  • cache coverage for MoE related hook points

Fixes #1464

Type of change

  • New model / architecture support
  • Tests

Testing

Ran the narrow Qwen2-MoE checks:

UV_NO_SYNC=1 uv run pytest tests/unit/model_bridge/supported_architectures/test_qwen2_moe_adapter.py
UV_NO_SYNC=1 uv run pytest tests/integration/model_bridge/test_qwen2_moe_bridge.py
UV_NO_SYNC=1 uv run pytest tests/unit/tools/test_model_registry.py -k TestRegistrySyncedWithFactory

@Canonik Canonik marked this pull request as ready for review July 2, 2026 08:18
@Canonik

Canonik commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

should be good to go

@jlarson4

jlarson4 commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Great work on this @Canonik! Thank you for getting this taken care of, I am merging now

@jlarson4 jlarson4 merged commit b1a2992 into TransformerLensOrg:main Jul 2, 2026
25 checks passed
jlarson4 added a commit that referenced this pull request Jul 2, 2026
* Added minimal TransformerBridge support skeleton for Qwen2MoeForCausalLM

* Fix return type annotation in Qwen2MoeRouterBridge.forward method

* Enhance tests for Qwen2-MoE bridge to validate cache capturing of MoE hooks

---------

Co-authored-by: Canonik <alessandro006@icloud.com>
@Canonik Canonik deleted the add-qwen2-moe-bridge branch July 3, 2026 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Proposal] Add Qwen2-MoE shared+routed expert adapter (Qwen2MoeForCausalLM)

2 participants