Skip to content

Add Cohere2 architecture adapter#1485

Open
hsusul wants to merge 1 commit into
TransformerLensOrg:mainfrom
hsusul:cohere2-adapter
Open

Add Cohere2 architecture adapter#1485
hsusul wants to merge 1 commit into
TransformerLensOrg:mainfrom
hsusul:cohere2-adapter

Conversation

@hsusul

@hsusul hsusul commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Addresses #1474

Adds TransformerBridge support for Cohere2ForCausalLM, including Cohere Command-R7B / Command-A style interleaved attention.

The adapter extends the existing Cohere support and handles both Cohere2 interleave encodings:

  • explicit layer_types
  • sliding_window_pattern / _sliding_window_pattern

Sliding-window layers retain RoPE, while full-attention layers suppress RoPE / use NoPE behavior.

Tests

  • Cohere2 unit tests + registry sync: 23 passed
  • Existing Cohere unit tests: 55 passed
  • Tiny Cohere2 integration test: 5 passed
  • Combined Cohere2 unit + integration rerun: 24 passed
  • black --check
  • isort --check-only
  • targeted mypy
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant