diff --git a/.agents/skills/adding-a-new-model/SKILL.md b/.agents/skills/adding-a-new-model/SKILL.md index 8a0ca0f66..9ec1c8887 100644 --- a/.agents/skills/adding-a-new-model/SKILL.md +++ b/.agents/skills/adding-a-new-model/SKILL.md @@ -35,6 +35,8 @@ Read these when you need deeper detail on a specific topic: - Find a small checkpoint on HuggingFace Hub for testing - Have the HuggingFace `transformers` source available to reference the PyTorch implementation +- Pin the checkpoint revision before collecting configs, weights, processors, + parity data, or goldens ## Step-by-step @@ -219,6 +221,12 @@ Then add the new module to `src/mobius/_configs/per_model/__init__.py` so its side-effect registration runs at import time. The dispatcher filters hooks by the declared model_type strings, so unrelated models never see your hook. +Match upstream config semantics, not just field names: preserve transformation +order (rounding/scaling), explicit `None`, alias precedence, zero-as-disabled +sentinels, and wrapped/unwrapped composite configs. Add focused tests for each +nontrivial transform. Test both trusted/custom config classes and raw pinned +JSON, especially nested decoder fields such as head/layer counts. + ### 7. Write tests See the **writing-tests** skill for full details. At minimum: @@ -281,6 +289,7 @@ see the [quality-checklist skill](../quality-checklist/SKILL.md). - [ ] ORT GenAI test added to `tests/ort_genai_test.py` (text-generation and VLM models) - [ ] CLI build works (`mobius build --model ...`) - [ ] Multi-dtype correctness verified (fp32, fp16, bf16) +- [ ] Pinned revision reaches every Hub/processor/weight/golden call **Note:** Default optimizer passes (CSE, deduplicate initializers, identity elimination, remove unused nodes/opsets) are applied automatically. @@ -426,6 +435,18 @@ config-driven features for the offending subclass. > embedding table off-by-one), read > [`references/weight-preprocessing.md`](references/weight-preprocessing.md). +### 8. Compatibility and optional dependency traps + +- Guard imports of symbols from unpinned dependencies when older supported + versions may not define them. +- If remote code imports an optional package, first prove the selected model + path needs it. Scope any test-only shim to that unused import; do not add a + production fallback or require an irrelevant package. +- When extending a public component, append optional parameters after existing + positional parameters and add a positional-call compatibility test. +- Helpers with rank/shape contracts must receive a value of that rank; do not + rely on incidental compatibility from a higher-rank tensor. + ## Reference examples | Complexity | File | Why | diff --git a/.agents/skills/debugging-multimodal/SKILL.md b/.agents/skills/debugging-multimodal/SKILL.md index 7977f661b..0e4df41ee 100644 --- a/.agents/skills/debugging-multimodal/SKILL.md +++ b/.agents/skills/debugging-multimodal/SKILL.md @@ -261,11 +261,11 @@ via `MOBIUS_ORT_LOWER_OPSET_FOR_EP=1`). See `src/mobius/_flags.py`. ### Encoder input dtype alignment -Encoder task inputs should be declared with `dtype=config.dtype` so -entry tensors match the model compute dtype (float32/float16/bfloat16). -In the current codebase, multimodal encoder task builders set encoder -inputs directly to `config.dtype` (there is no `_cast_encoder_input()` -helper in `src/mobius/tasks/_base.py`). +Real vision/audio processors emit float32. Encoder graph inputs should +therefore be float32 even for fp16/bf16 exports, with one Cast to +`config.dtype` at graph entry. Verify this with an actual processor batch +and a graph I/O dtype assertion; synthetic feeds can hide the mismatch. +Some older task builders still use `config.dtype` and are not safe templates. ### GQA for KV-shared layers diff --git a/.agents/skills/diffusion-models/SKILL.md b/.agents/skills/diffusion-models/SKILL.md index 6a03d10ce..959de5499 100644 --- a/.agents/skills/diffusion-models/SKILL.md +++ b/.agents/skills/diffusion-models/SKILL.md @@ -353,6 +353,26 @@ Compare `named_parameters()` output with HF weight names. Use the techniques from the **weight-name-alignment** skill to minimize `preprocess_weights`. +### 7. Prove pipeline executability + +A component graph or metadata file is not a runnable diffusion pipeline. +Before advertising runtime support, execute the complete path: + +`image encoder/VAE -> latent packing -> denoiser loop + scheduler -> target token unpacking -> VAE decoder` + +Verify sample/output ranks, source-vs-target token slicing, CFG semantics, and +the runtime's actual scheduler identifiers/equations. Reload final metadata and +verify every file path, port, preprocessing step, loop edge, and postprocess. +If the runtime cannot express the dataflow, reject it before emitting artifacts; +keep direct component export separate. + +Validate required VAE statistics against latent channel count before graph +construction (`len(mean) == len(std) == z_dim`, positive standard deviations). +Keep model-specific component/task overrides data-driven rather than adding +pipeline-name conditionals. Fold initializers after weight loading and mark +denoiser/VAE roles by semantics, not names. Tests must use `os.path.join`/`Path` +for temporary ONNX and profiling paths so cleanup works on POSIX and Windows. + ## Naming conventions for diffusers models Diffusers uses different conventions than transformers: diff --git a/.agents/skills/multi-agent-coordination/SKILL.md b/.agents/skills/multi-agent-coordination/SKILL.md index eb13d3853..48d095de2 100644 --- a/.agents/skills/multi-agent-coordination/SKILL.md +++ b/.agents/skills/multi-agent-coordination/SKILL.md @@ -40,7 +40,8 @@ git fetch origin && git checkout && git pull ## 2. Commit Coordination Protocol -Multiple agents committing to the same branch requires discipline to avoid divergence. +Choose the branch strategy in §9 first. When multiple agents contribute to one +branch, use this protocol to avoid divergence. **Rules**: 1. **Pull before commit**: Always `git pull origin --rebase` immediately before committing. @@ -67,11 +68,14 @@ Checks at each stage prevent problems from compounding across agents. - Verify the branch exists and is up to date. - Run the relevant tests to establish a baseline — know what was already failing before you touched anything. - Check for uncommitted changes left by previous agents in your worktree. +- Verify `inspect.getfile(mobius)` points into the agent's worktree; editable + installs from another worktree invalidate all test evidence. ### After each commit - Run the affected tests immediately (don't batch; catch regressions early). - Verify the commit landed on the correct branch (`git log --oneline -3`). - Confirm the push succeeded. +- Confirm the remote ref/PR head SHA, not just local `HEAD`. ### Before final review (lead audit) - Check ALL worktrees for unpushed commits or uncommitted changes. @@ -85,6 +89,15 @@ Checks at each stage prevent problems from compounding across agents. - Final test run passes. - PR description updated with an accurate scorecard of what changed. +### CI failures and main refreshes +- Compare the exact failing test on the exact base SHA; a red overall workflow + does not prove the same job failed on base. +- Keep feature scope: document unrelated baseline names/metrics instead of + changing their code or tolerances. +- Rebase independent PRs linearly onto `origin/main`. Resolve shared registries, + exports, helpers, and guards semantically by retaining both sides; then run + shared-surface tests, lint, and review before `push --force-with-lease`. + --- ## 4. Task Dependency Management (DAG) @@ -171,18 +184,14 @@ Match the agent role to the task type. Mismatched roles waste agent capacity. --- -## 9. Single-Branch Strategy - -For large multi-agent efforts, use **one shared feature branch** rather than one branch per agent. - -**Why**: Avoids complex multi-branch merge scenarios at the end. All agents commit to the same branch via their isolated worktrees. - -**Trade-offs**: -- More `pull --rebase` cycles per agent. -- Occasional push conflicts (recoverable with rebase). -- BUT: much simpler final state, single PR, linear history. +## 9. Branch Strategy -**Alternative**: Separate branches per agent merged via separate PRs — only viable when features are truly independent and don't share files. +- Use one branch only when agents contribute to one atomic PR; serialize edits + to shared files and require pull/rebase before every push. +- Use separate branches/PRs for independent deliverables (for example, separate + model architectures). This isolates CI, review, rollback, and publication. +- Never merge main into a linear-history feature branch. Rebase, resolve shared + surfaces semantically, and force-push only with `--force-with-lease`. --- diff --git a/.agents/skills/multimodal-models/SKILL.md b/.agents/skills/multimodal-models/SKILL.md index a69e698d7..156dff435 100644 --- a/.agents/skills/multimodal-models/SKILL.md +++ b/.agents/skills/multimodal-models/SKILL.md @@ -97,6 +97,16 @@ convention), not `vision_encoder`/`audio_encoder`. The embedding model must handle `num_image_tokens=0` (text-only input) by zero-padding `image_features` before Gather so indices stay in-bounds. +Input mixing must also: +- compute feature offsets over the flattened batch, not restart per row; +- preserve separate image/video streams or define and test an explicit packed + order; and +- cover two-row mixed-media prompts with distinguishable feature sentinels. + +For cross-attentive decoders, export total-length padding masks and only caches +that are actually reusable. Do not apply a generic seq2seq cross-cache contract +to fixed encoder states without proving projection/cache semantics. + ### Conditional 3-or-4-model task Some models come in two tiers (e.g. Gemma4): small variants include an @@ -219,72 +229,35 @@ models this overflows ORT's CUDA Gather kernel. **Workaround:** Split into L separate `Embedding([V, D])` tables via `nn.ModuleList`, and use `Slice` instead of `Gather` for per-layer projection indexing. -## Vision/audio encoder f32 input casting - -> **This applies to ALL multimodal models and ALL inference paths** — -> not just ORT GenAI, and not architecture-specific. +## Processor-to-graph contract -Image and audio preprocessing universally produces **float32** output. -This is true across all frameworks and runtimes: +Run the real processor before finalizing graph I/O. Record names, shapes, +dtypes, media-row ordering, sampled frame positions, and empty-media behavior; +synthetic packed video layouts often differ from processor-native rows. -- **PIL / torchvision:** Pixel normalization outputs f32 -- **torchaudio / librosa:** Mel spectrograms are f32 -- **ORT GenAI image_processor:** Resize, normalize, tile → f32 -- **ORT GenAI audio_processor:** Feature extraction → f32 -- **ORT Python API:** Custom preprocessing pipelines → typically f32 -- **Foundry Local:** Uses GenAI processors → f32 - -This means vision and audio encoder ONNX graphs must accept f32 inputs -even when the model is built in f16 or bf16. The encoder adds a -`Cast(f32 → model_dtype)` at its graph entry point so that any runtime -can feed it preprocessed data without worrying about the model's -internal precision. - -### How it works +Vision/audio preprocessors emit float32. Encoder graphs therefore accept +float32 at the boundary and cast once to fp16/bf16 internally: ``` -Input (f32 from ANY preprocessor — PIL, torchaudio, GenAI, etc.) - ↓ -Cast(to=FLOAT16) ← inserted automatically by mobius - ↓ -Vision/Audio encoder (weights in f16/bf16) - ↓ -Output (model_dtype) +processor f32 -> Cast(model dtype) -> reduced-precision encoder ``` -Encoder weights still use the requested dtype (f16/bf16) for memory -efficiency — only the graph inputs are f32. The Cast is a lightweight -op with negligible overhead. - -### Why f32 is the universal preprocessing dtype +Test image-only, video-only, mixed media, two-row batches, and decode with zero +new media. Build kwargs conditionally; strict processors may reject video-only +arguments on image-only calls. -Preprocessing involves floating-point arithmetic (mean subtraction, -std division, resampling interpolation) where f32 is the natural -precision. Converting to f16/bf16 before these operations would lose -precision in the preprocessing itself. The model's internal precision -only matters after the preprocessed data enters the encoder. - -### What mobius does - -Mobius always builds encoder graphs with f32 inputs — this is the -default behavior, not gated behind any flag. It works correctly -regardless of the inference runtime: - -- `--runtime ort-genai` → f32 inputs (GenAI processors output f32) -- No `--runtime` flag → f32 inputs (ORT Python API, custom runtimes) -- Foundry Local → f32 inputs (uses GenAI internally) - -Without the Cast-at-input, ORT throws a type mismatch error: -``` -Type Error: Type parameter (T) bound to different types -(tensor(float) and tensor(float16)) -``` +For composite checkpoints, generate processor assets from the parent config +even if model construction exposes an unwrapped text sub-config; test both +wrapped and unwrapped resolver forms. -### For model authors +For variable-length media, audit symbolic complexity: packed-attention metadata +must scale with rows/windows, not total-patches × media-count. Keep dense masks +only as a portable fallback and assert optimized CUDA/DML graphs lack the +quadratic construction. -If you're adding a new multimodal model, you don't need to handle this -manually — mobius inserts the Cast automatically for all encoder graphs. -If the model dtype is already f32, no Cast is needed. +Config generation is not runtime support. Execute real media through ORT GenAI; +if it cannot supply a required encoder input or position-ID rank, reject export +before writing artifacts and use the same evidence for any Foundry waiver. ## GQA for KV-shared layers (Gemma4) diff --git a/.agents/skills/onnx-export-quantization/SKILL.md b/.agents/skills/onnx-export-quantization/SKILL.md index 50617949d..72a8eb48f 100644 --- a/.agents/skills/onnx-export-quantization/SKILL.md +++ b/.agents/skills/onnx-export-quantization/SKILL.md @@ -322,6 +322,13 @@ FP32 when the model is BF16. Add `op.CastLike(result, input)` to ensure dtype consistency. See the `reusable-components` skill's section on precision behaviour. +### 6. Attention rewrite schema mismatch + +Quantizers may rewrite Attention to a contrib op with fewer outputs or a +different cache contract. Restrict the quantized op set when schemas do not +match, then load and execute the result; successful conversion alone is not +evidence. Document the narrowed recipe and unsupported rewrite explicitly. + ## Testing quantized models ### L4: Golden data generation diff --git a/.agents/skills/ort-genai-config/SKILL.md b/.agents/skills/ort-genai-config/SKILL.md index f12959a82..cb94def19 100644 --- a/.agents/skills/ort-genai-config/SKILL.md +++ b/.agents/skills/ort-genai-config/SKILL.md @@ -244,6 +244,15 @@ decoder_inputs = {name: name for name in decoder_input_names} This means the genai config automatically adapts when `RemoveDeadGraphInputsPass` removes unused inputs (e.g. `position_ids` absorbed by GQA fusion). +Hybrid cache metadata must preserve global layer indices across KV, convolution, +and recurrent states. Derive slot count from the maximum +`past_key_values..*` input index plus one; counting only `.key` inputs +silently drops non-KV or sparsely indexed layers. + +CUDA Graph capture belongs on stable-shape autoregressive decoder sessions, +not one-shot variable-shape vision/embedding stages. Keep an explicit decoder +opt-out and validate capture together with shared KV buffers on the real model. + > For the full generation flow, input routing, QwenImageProcessor output > tensors, and the multimodal processor factory, see > [`references/multimodal-pipeline.md`](references/multimodal-pipeline.md). @@ -273,6 +282,13 @@ processor doesn't provide. Either: 1. Compute them externally and inject via NamedTensors, or 2. Modify the vision model to compute them from `image_grid_thw` internally +### Config writes successfully but runtime cannot execute + +Config/schema success is not runtime support. Run load plus generation through +the exported contract. If the runtime cannot route required feature inputs, +position-ID rank, cache state, scheduler, or multimodal metadata, reject export +before writing artifacts and report the exact runtime version/limitation. + ### "input_ids size exceeds max length" For image prompts, the tokenized input_ids (including image_pad tokens) can diff --git a/.agents/skills/quality-checklist/SKILL.md b/.agents/skills/quality-checklist/SKILL.md index 776a9b691..9f5b6453e 100644 --- a/.agents/skills/quality-checklist/SKILL.md +++ b/.agents/skills/quality-checklist/SKILL.md @@ -32,15 +32,16 @@ before the PR is merged. the model is not a standard text-generation model - [ ] `preprocess_weights()` correctly maps every HuggingFace state-dict key to the ONNX initializer name (verified by the weight-alignment test) -- [ ] All new components use `from mobius.components import ...` (public API), - not private submodule paths +- [ ] Models/tasks import components from the public API; component submodules + import sibling primitives directly to avoid package-init cycles - [ ] No explicit protobuf operations anywhere in new code (`onnx.helper`, `onnx.TensorProto`, etc. are forbidden) - [ ] Tensor shapes are annotated in comments after non-trivial operations - [ ] Automated code review (Copilot/PR review) has been run and all findings are resolved or explicitly dismissed with a reason -- [ ] `lintrunner -a` is clean — zero lint errors before merging - (`lintrunner f --output oneline --all-files` to auto-fix, then re-run to confirm) +- [ ] Run the CI formatter first (`lintrunner f --output oneline --all-files`), + then `lintrunner -a`; confirm the pinned tools actually ran (an + uninitialized/no-op lintrunner is not evidence) ### 2. L1 — Graph builds @@ -56,6 +57,8 @@ before the PR is merged. - [ ] YAML test case created at `testdata/cases//.yaml` - [ ] `test_model_id` field set to a real HuggingFace model ID +- [ ] Revision/trust flags are pinned and forwarded through config, processor, + weights, golden generation, and CLI build - [ ] Schema validates: `python -m pytest tests/yaml_schema_test.py` ### 4. L3 — Synthetic parity @@ -87,6 +90,8 @@ before the PR is merged. - [ ] Generation golden file committed to `testdata/golden//_generation.json` - [ ] `python -m pytest tests/e2e_golden_test.py -m generation -k ""` passes +- [ ] Sequence length is asserted before exact token/frame comparison (no + prefix-only `zip` comparison) > **Speech-language models:** The golden generation script supports the > `speech-language` task type for models that process audio inputs (e.g., @@ -104,10 +109,13 @@ before the PR is merged. ### 7. Multi-dtype correctness -- [ ] fp32 tests pass (target: exact token match in greedy generation) -- [ ] fp16 tests pass (target: logit parity `atol=1e-2`; token match for - first N tokens) -- [ ] bf16 tests pass (target: logit parity `atol=1e-2`) +- [ ] Every claimed dtype runs real, nonzero input and checks semantic output, + not merely export/session creation +- [ ] fp32 passes full-logit parity and deterministic semantic output +- [ ] fp16/bf16 pass full-logit parity (`atol=1e-2`) plus + architecture-appropriate tokens, frame IDs, transcript, or image output +- [ ] A dtype that executes inaccurately is rejected explicitly and is not the + default; never silently downgrade Use the example `--compare-hf --dtype f16/bf16` flag if a comparison script exists: @@ -146,15 +154,23 @@ python examples/_text_generation.py --compare-hf --dtype bf16 uniform KV cache shapes; if so, explicitly override the generated `genai_config.json` before runtime validation rather than assuming the default generated setting is correct -- [ ] Encoder inputs (vision/audio) are declared with `dtype=config.dtype` - at sub-model entry (no stale float32-only cast guidance) +- [ ] Vision/audio graph inputs match the real processor (normally float32); + reduced-precision encoders cast once at graph entry +- [ ] Representative graph evidence covers raw/post-Mobius/post-weight and + ORT EP-optimized op histograms; verify critical fusions, unexpected + Transpose nodes, and activation-vs-scalar metadata Memcpy nodes +- [ ] Final fusion claims use loaded weights/constants; no-weight graphs cannot + prove initializer folding ### 9. ORT GenAI runtime - [ ] Model can be loaded with `ort_genai.Model(output_dir)` without error -- [ ] Greedy text generation produces non-empty, coherent output +- [ ] Actual generation with required media/features produces coherent output; + schema/config emission alone is not runtime support - [ ] ORT GenAI test added to `tests/ort_genai_test.py` (or confirmed covered by an existing parametrized test) +- [ ] Structurally unsupported contracts fail before artifacts are emitted and + have a version-specific, evidence-based waiver Run the ORT GenAI integration test: @@ -176,6 +192,8 @@ python -m pytest tests/ort_genai_test.py -m integration_slow -k "" -sv - [ ] Model can be loaded from the exported ONNX package by Olive - [ ] INT4 / INT8 quantization runs to completion without errors - [ ] Quantized model produces non-degenerate output (coherent text) +- [ ] Quantization uses only required execution providers if unrelated provider + registration fails, and evidence includes size, load, and inference - [ ] If quantization changes the graph structure (e.g. MatMulNBits), verify the `genai_config.json` still loads correctly in ORT GenAI @@ -199,6 +217,21 @@ novel weight layouts (e.g. fused QKV, non-standard expert routing). a comment in the source file or the skill notes explains it - [ ] README model table updated if this is a significant new addition +### 13. Publication and CI triage + +- [ ] Run specialist review after L1-L3, before expensive L4/L5, and again + after optimization or rebase changes +- [ ] Rebase linearly onto `origin/main`; resolve shared registries/helpers + semantically, run shared-surface tests, and push with `--force-with-lease` +- [ ] After rebase, rerun config generation and actual runtime load/generation; + parity-only tests do not catch changed cache/runtime contracts +- [ ] Confirm the remote PR head SHA, mergeability, replacement lint, and + architecture checks after the final push +- [ ] Triage red CI at check/job/test granularity against the exact base SHA; + record test names and metrics, and never change unrelated models +- [ ] For expensive GPU goldens, link targeted L4 and L5 jobs; aggregate + timeout or runner failure is not model evidence + --- ## Waiver policy diff --git a/.agents/skills/reusable-components/SKILL.md b/.agents/skills/reusable-components/SKILL.md index ac1d4c11e..3651d5263 100644 --- a/.agents/skills/reusable-components/SKILL.md +++ b/.agents/skills/reusable-components/SKILL.md @@ -51,6 +51,10 @@ an underscore prefix for local use: from mobius.components import Conv2d as _Conv2d, SiLU as _SiLU ``` +Inside `src/mobius/components/_*.py`, import sibling primitives directly +(`mobius.components._common`, etc.) so a partially initialized public package +cannot create a circular import. + Model-specific compound blocks (e.g. `_TimestepEmbedding`, `_DiTBlock`, `_ResNetBlock2D`) remain in the model files they belong to. @@ -210,6 +214,17 @@ position IDs instead of an explicit mask. `op.Cast(to=ir.DataType.FLOAT)`, compute, then cast back with `op.CastLike(result, input)`. For dtype-adaptive parameters, use `op.CastLike(param, reference)`. +8. **Prefer canonical ONNX operators.** Emit standard `BatchNormalization`, + `RMSNormalization`, and activation ops when semantics match so ORT can fold + and fuse them. Scale-free RMSNorm still needs a schema-valid 1-D scale and + must preserve HF's fp32 variance semantics before casting back. Prefer + `stash_type=FLOAT` over a decomposed graph when it matches HF, and verify + normalize-in-fp32 → cast activation → apply gamma ordering. Keep a manual + form when preventing an incorrect provider fusion is intentional and tested. + +9. **Preserve public call compatibility.** Append new optional arguments after + existing positional parameters and add a positional-call regression test. + ## ONNX op patterns overview Key patterns for building components: diff --git a/.agents/skills/writing-tests/SKILL.md b/.agents/skills/writing-tests/SKILL.md index 79da765c3..2f92dd4f0 100644 --- a/.agents/skills/writing-tests/SKILL.md +++ b/.agents/skills/writing-tests/SKILL.md @@ -239,6 +239,33 @@ integration test alongside any new custom function or Scan op.** - **Vision with real pixel values** — zeros don't exercise the encoder - **All dtypes** (f32, f16, bf16) — each can expose different bugs - **GPU when available** — different kernels on CUDA +- **Actual package wiring** — feed each ONNX stage from the preceding ONNX + stage, not an HF intermediate that bypasses the integration under test +- **Defaults and masks** — assert constructor/config defaults in emitted ONNX + attributes and test padding invariance across prefill plus cached decode +- **Real processor contract** — record input names, shapes, dtypes, media-row + ordering, and sampled frame positions from nonzero image/video/audio data +- **Batch and decode edges** — use two rows with distinguishable media + features, mixed modality order, and a decode step with zero new media + +### Golden tests must be reproducible and exact + +- Pin one revision through config, processor, weight shards, reference + generation, and ONNX build; test that plumbing forwards it. +- Assert sequence lengths before exact token/frame comparison. For CTC, compare + the full argmax frame sequence and collapsed transcript. +- Test image-only, video-only, and mixed media; pass processor kwargs only for + media that are present. +- On hosted runners, isolate and eagerly delete each test's Hub, assets, and + Xet caches. Patching `HF_HOME` after `huggingface_hub` import is insufficient; + patch its imported cache constants too. +- Run the exact L2 discovery path: YAML schema, `test_model_id`, revision, and + trust flags can be missed by model-local tests. +- Run affected-model detection on the full diff before GPU CI. A new model + should select its targeted L4/L5 cases; distinguish all-model timeout or + runner termination from a model assertion failure. +- Reference goldens must come from an independently invoked upstream pipeline, + never from the implementation under test or ad-hoc intermediate features. ### Recurrent state ≠ KV cache