Conversation
Stop actions/checkout from leaving its short-lived ghs_* installation token in .git/config as http.<host>.extraheader after the workflow. On self-hosted runners (build-and-test, tag_release build, tutorial build) the work tree persists between jobs, so an interrupted or killed job can leave the (later-expired) token behind. None of the existing checkouts run authenticated git operations afterwards, so disabling persistence is safe. Public submodules are unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop-in reference for Claude Code (and similar AI tools) covering the three-simulator pipeline (Gem5 -> Spike -> TOGSim), directory map, test entry points, key env vars and YAML knobs, multi-tenant API contract, build steps, and known gotchas. Complements README.md with a denser cheat-sheet aimed at code navigation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/build_from_source.sh now reads release_tag from the manifest that the CI docker image is built against, and clones gem5, llvm-project, and riscv-isa-sim at those tags. Untagged HEAD clones had caused mlir-opt option-name drift (Pass-Options-Parser error followed by an assert(0) in extension_codecache.py). Documents the pin in README.md and CLAUDE.md, and adds a plain-text / ASCII rule for commit messages in CLAUDE.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…overage [Build] Pin third-party deps to thirdparty/github-releases.json tags
scripts/setup_worktree.sh creates a sibling worktree under $PARENT_DIR/<repo>-<purpose>/, wires per-worktree env vars via a generated .envrc (TORCHSIM_DIR, TORCHSIM_DUMP_PATH, TORCHSIM_LOG_PATH, TOGSIM_CONFIG, PYTHONPATH), unsets the upstream so the first `git push -u origin <branch>` creates the right remote branch, and symlinks the TOGSim binary from the worktree the script was run from to skip a ~10-minute TOGSim rebuild per fresh worktree (readlink -f resolves chains so the link points at the real binary). scripts/clear_codegen_cache.sh wipes Inductor's compile cache (.torchinductor/, set via TORCHINDUCTOR_CACHE_DIR in extension_config.py:139) and the per-source-hash dirs identified by the 11-char prefix from extension_codecache.hash_prefix. Run it between iterations on PyTorchSimFrontend/mlir/* or anything else that affects emitted MLIR -- otherwise the next torch.compile silently replays the previous compile. togsim_results/ and unrelated files under outputs/ are preserved. docs/worktrees.md documents the flow: activate via `source .envrc`, build the .so once per worktree, TOGSim binary sharing, codegen iteration loop, and the diagnostic for the common "forgot to source .envrc" case (tracebacks pointing at the canonical worktree path while editing elsewhere). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Running tests section now states that .github/workflows/pytorchsim_test.yml runs an explicit allowlist of tests/*.py (one Docker job per test, ~40 total), not a glob-discovered set, and calls out that test_gqa.py, test_gqa_decode.py, and test_eager.py exist in the repo but are not in CI. The Gotchas section adds a bullet explaining that codegen iteration requires wiping $TORCHSIM_DUMP_PATH between runs -- Inductor's compile cache and the per-source-hash MLIR/wrapper dirs both live under it and will silently replay the previous (possibly broken) compile otherwise. This prevents two common mistakes: assuming new tests/*.py files are automatically gated on PR, and assuming a fresh re-run will pick up a codegen fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…228) The second clause of the index-cast guard at mlir_ops.py:268 compared the whole [tile_size, dtype] list against the string "index", which is never true. With i64 on the lhs and index on the rhs the code fell through to the same-bit-width branch (MLIR_TO_BIT["i64"] == MLIR_TO_BIT["index"] == 64) and emitted arith.cmpi between vector<Nxi64> and vector<Nxindex>, which mlir-opt rejects. This blocked any MoE model whose (i64_buf == arange) expert-mask pattern landed with that operand orientation -- first observed on deepseek_v3. Fix replaces the dead clause with op_type2[1] == "index" so the operand2-side index_cast at lines 285-288 is reachable, normalizing the rhs to i64 before the cmpi. Add tests/test_expert_mask.py as a focused regression covering the (expert_idx_i64.unsqueeze(-1) == arange(N)) -> torch.where pattern.
Adds three top-level jobs (test_eager, test_exponent, test_sort) and two steps inside the test_fusion job (test_attention_fusion, test_matmul_vector). All five were verified locally before registration. Files in tests/ that remain intentionally out of CI: - test_gqa.py, test_gqa_decode.py: WIP GQA SDPA path, tracked by issue #198 - test_sdpa.py: overlaps with in-flight SDPA template work; ambiguous about which backend it actually exercises - test_topk.py: sort-family coverage now provided by test_sort (stable + unstable + duplicate-key); revisit if topk-specific shapes need gating - test_group_conv.py: not run locally yet (stress config); follow-up after runtime cost is understood - test_vectorops.py: imports from other tests (test_add, test_activation, test_reduce, test_layernorm, test_softmax) which is fragile; needs an independent helper extraction before going into CI Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ovided (issue #236) decompose_native_multi_head_attention referenced attn_bias before it was defined, so any masked-attention call raised NameError. The same block also computed attn_bias but never folded it into scores before softmax, so the mask would have had no effect even if attn_bias existed. Initialize attn_bias as zeros_like(scores), build the bias from the boolean/additive mask, and add it to scores ahead of softmax. Resolves issue #236.
…sue #237) 1. mlir_codegen_backend.py:909 - NotImplementedError was constructed but not raised, so kernels with multiple reduction axes silently fell through to incorrect codegen. Add the missing raise. 2. mlir_codegen_backend.py:171-179 - codegen_sram_plan_prefix called buf.get_size() before checking buf is None, so a None entry in graph_inputs would AttributeError. Reorder the None check first. 3. mlir_common.py - CSEProxy had two identical static check_bounds declarations; only the latter survived class-body evaluation. Drop the duplicate so future drift cannot hide which one is authoritative. 4. extension_codecache.py:274 - CustomAsyncCompile.mlir passed valdiation_wrapper_name (typo) carrying self.validation_binary_name to MLIRCodeCache.load. The misspelled kwarg was absorbed by **kwargs and validation_wrapper_name silently fell back to its default. Use the correct keyword and value. Resolves issue #237.
CONFIG_TORCHSIM_TOG_HOST_{CC,CFLAGS,LDFLAGS} were added in 54ccd4c but
no caller ever consumed them; MLIRCodeCache._load_library is still
`pass`. Issue #239 flagged the `if True:` shortcut in
_default_tog_host_cflags, but the helper's return value is never read,
so no .so is actually compiled with `-Og` as the issue assumed —
removing the whole block is the honest fix instead of restoring the
env-var gate around code nothing calls. If a host-side TOG .so compile
path lands later, the gate can be reintroduced at the real call site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
) `extension_config.CONFIG_TORCHSIM_DUMP_PATH` was resolved through the module-level `__getattr__`, which read `TORCHSIM_DUMP_PATH` and *mutated* `os.environ["TORCHINDUCTOR_CACHE_DIR"]` as a side effect on every attribute access. Generic attribute reads should not change process state, and the four codegen call sites trigger this on every compile. The dynamic re-derivation is load-bearing for the Jupyter tutorials (tutorial/session1/CompilerOptimization.ipynb, Mapping.ipynb) which flip `TORCHSIM_DUMP_PATH` between cells to compare fusion ON/OFF in separate output dirs, so the issue's "move to module top-level" fix would silently break them. Instead expose the side effect as an explicit `get_dump_path()` function — same behavior at the codegen call sites, but readers can no longer trigger an env mutation by accident, and the function name signals "this writes to os.environ". Also fixes the unrelated bug noted at the end of #240: `__getattr__` fell through to an implicit `None` return for unknown names. Now it raises `AttributeError` per PEP 562, so typos like `extension_config.CONFIG_NONEXISTENT` surface immediately instead of becoming downstream `NoneType` errors. Verified locally: `get_dump_path()` syncs `TORCHINDUCTOR_CACHE_DIR`, follows dynamic env changes between calls, leaves env untouched on unrelated attribute reads, and unknown attrs raise `AttributeError`. Closes #240. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…sion
The guard at mlir_codegen_backend.py:909-910
if len(reductions.loops) > 1:
raise NotImplementedError("Not support multiple reduction axis..")
was a leftover from when the codegen was deliberately limited to a
single reduction axis, but multi-axis support was restored months ago
and the guard was never removed.
Timeline:
- 8eacf27 ("Optimize reduce/elementwise code", 2025): replaced
`for reduction in reductions.loops` with `reductions.loops[0]`,
added this guard (no `raise` — silently dead).
- c61f67d ("Support multi reduction dim + Add Diffusion model test",
2025-08-14): restored the `for reduction_loop in reductions.loops`
iteration explicitly to support multi-axis reduction. test_diffusion
was added in this same commit and passed. The guard was left in
place — harmless because still `raise`-less.
- 5045837 ("Fix four codegen correctness issues surfaced by review",
2026-05-26, issue #237): added the missing `raise`, framing it as a
"silently fell through to incorrect codegen" fix. In fact the
codegen below the guard (which iterates all reduction loops) was
correct — c61f67d had made it so — and adding `raise` broke the
very test (test_diffusion) that c61f67d added to validate multi-axis
support.
Right fix is to delete the guard, not add `raise` to it. The other
three items in #237 are unrelated and stand.
Verified: file py_compiles. CI test_diffusion is the empirical proof
that the post-guard codegen handles multi-axis correctly within the
allclose tolerance the test uses.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…s.py
Replaces 48 near-duplicate ~12-line `test_result(name, out, ...)` defs
(two signature variants, three slightly divergent bodies — one had a
'pass message only' bug) with a single canonical helper that prints
framed pass/fail messages and exits 1 on mismatch. Positional-argument
compatible, so caller sites are unchanged.
Each migrated file replaces its local def with:
import os, sys
sys.path.insert(0, os.path.join(
os.environ.get("TORCHSIM_DIR", default="/workspace/PyTorchSim"), "tests"))
from _pytorchsim_utils import test_result
The module name is deliberately unique rather than `tests._utils`:
ultralytics ships its own top-level `tests` package in site-packages,
which shadows any generic `tests` import. `insert(0, .../tests)` puts
the repo's tests dir ahead of site-packages so the helper resolves
regardless of installed packages.
Net diff: 50 files changed, 220 insertions(+), 729 deletions(-).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tests/ was a flat mix of op-level files, single-file model tests, and
inconsistently-cased model directories. Adds a hierarchy:
tests/
_pytorchsim_utils.py
ops/
elementwise/ reduce/ gemm/ conv/ attention/
view/ sort/ sparsity/ misc/ fusion/
models/
DeepSeek/ Diffusion/ Llama/ MLP/ MoE/
MobileNet/ Mixtral8x7B/ Yolov5/
test_mlp.py test_resnet.py test_single_perceptron.py
test_transformer.py test_vit.py
system/
test_eager.py test_hetro.py test_scheduler.py
test_stonne.py test_vectorops.py
Mixtral_8x7B → Mixtral8x7B for consistency with the other PascalCase
model dirs. Existing single-file model dirs are kept as dirs (they
may grow companion files like the Mixtral model.py).
All file moves use `git mv` to preserve history. External path
references rewritten across .github/workflows/pytorchsim_test.yml,
README.md, CLAUDE.md, .github/ISSUE_TEMPLATE/bug_report.md, and
scripts/{sparsity,stonne}_experiment/.
Cross-test imports drop the `tests.` prefix (the prior commit puts
`<repo>/tests` on sys.path[0]). `__init__.py` added to each new subdir
so e.g. `from ops.elementwise.test_add import test_vectoradd` resolves.
Sample-verified locally on tests/ops/elementwise/test_add.py and
tests/ops/fusion/test_matmul_vector.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PyTorchSim never executes CUDA: the npu PrivateUse1 backend (third_party/openreg) simulates CUDA on the CPU and forces math-only SDPA, and flash-attn is never imported. The only thing that required CUDA was find_package(Torch) inheriting it from the CUDA-built torch wheel; installing the CPU wheels removes it, so the whole CUDA base image can go. Dockerfile.base: - Base image ubuntu:22.04 instead of the cuda-cudnn-devel image, kept on 22.04 to match the prebuilt gem5/llvm/spike ABIs (libpython3.11, libprotobuf.so.23). - Python 3.11 via deadsnakes and CPU torch/torchvision wheels. - Remove flash-attn (unused, needed CUDA/nvcc to build). - Install psutil and the other yolov5 hub runtime deps (pyyaml, requests, tqdm, py-cpuinfo) that the old fat base provided implicitly; ultralytics is installed with --no-deps. - Remove duplicate onnx/matplotlib/conan/ninja installs. - Fix the RISC-V toolchain step to download and extract the elf toolchain once (the glibc tarball was downloaded but unused, the elf tarball was extracted twice). - Drop conda/nvidia paths from LD_LIBRARY_PATH. thirdparty/github-releases.json: pytorch_image -> ubuntu:22.04. DeepSeek: its remote modeling file (trust_remote_code) imports flash_attn, which transformers check_imports requires installed even though flash attention is never executed on the npu. Register an import shim in tests/models/DeepSeek/test_deepseek_v3_base.py that satisfies the static import check while leaving is_flash_attn_2_available() False. CI runners: the CPU-only image (~3.7 GB compressed) is small enough to pull on GitHub-hosted runners, so the base build, app build, and the op/model test jobs run on ubuntu-latest. Only test_deepseek (largest model), test_diffusion (UNet2D simulation OOMs the hosted runner), and test_accuracy (accuracy + speedup) stay on self-hosted.
The tests/ reorg (tests -> tests/ops, tests/models) left experiments/BERT.py importing EncoderBlock from the old paths (tests.Fusion.test_transformer_fusion, tests.test_transformer), so every BERT size raised ModuleNotFoundError. Put tests/ on sys.path and import from the new locations (ops.fusion.test_transformer_fusion, models.test_transformer), matching how the test files import their siblings. run_cycle.sh ran each model as 'python3 ... | tee log' under 'set -e' only, so a model crash was masked by tee's exit code and the job stayed green with empty logs (this is why BERT failures went unnoticed). Add 'set -o pipefail' so a failing model aborts the run. Verified: BERT --size base now runs end to end (TOGSim, 329573 cycles). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run the MLIR -> LLVM pipeline in-process via the bindings PassManager. Add Python out-of-line passes: lower_to_llvm, lower_dma_to_gemmini, lower_vlane_idx. Auto-resolve the bindings path from TORCHSIM_LLVM_PATH and ship the bindings in the LLVM artifact. Add op_coverage tooling and the bindings smoke test. Bump the LLVM pin and rebuild the thirdparty base image.
Emit togsim.transfer for >4D DMA and decompose it to a <=4D customized memref.dma_start: unit-collapse fast path, unrolled-subview peel for >4 effective dims. Fix #258 by emitting affine.apply (not arith.addi) for the peeled DRAM offset so the TOG pass can walk the loop index through it.
Split a loop axis so aligned FloorDiv/ModularIndexing collapse to per-axis affine indices. Mixed-radix split over a divisibility chain; integer-typed split symbols; r-prefix innermost reduce dims. Reindex the collapsed LoopBody instead of re-tracing; fold residual floor/mod via tensor range info. Shared boundary helpers, rank guard, and an uncovered floor/mod ledger. Enabled by default with the recompile fallback instrumented.
… floor/mod Insert a copy to relayout an operand whose floor/mod cannot be removed by axis-split: incompatible-radix shared-axis access and cross-axis multi-variable arguments. Enabled by default alongside axis-split.
Port the analysis and IR-mutation halves of the C++ test-tile-operation-graph pass to Python, wire build_tog into the gem5 path, and drop the C++ pass. Node-id counter is thread-local for concurrent compilation.
…seek seed Add tests/ops/view/test_floormod_axis_split.py covering axis-split and graph-copy patterns. Seed the global RNG in the deepseek base test so config-random weights are deterministic.
…cal SRAM offset Rewrite the >4D peel to mirror the C++ -dma-fine-grained subtile loop: wrap the outer dims in an affine.for nest (marked inner_loop so build_tog/TOG registers the induction var) and emit one <=4D memref.dma_start per iteration. The slice SRAM offset is the lane-banked physical offset -- split-outer dims rescaled by the lane coeff (stride/old_size*new_size, the MVIN block_stride / buildSramAffineMap rule) -- delivered as the last SRAM index operand. The previous unrolled subview carried the offset in the subview, which extract_aligned_pointer_as_index strips in the gemmini lowering, so every slice aliased the same spad location (pixel_shuffle MISMATCH). The DRAM offset folds with the original index into one affine.apply so processDramIndices can walk the loop index (#258). Thread vectorlane (systolic-array size) through run_python_passes into the pass for the rescale's nr_outerloop. Drop the axis-split rank guard now that >4D is peeled correctly, and register tests/ops/view/test_floormod_axis_split.py in the CI allowlist. Validated end-to-end (Gem5+Spike+TOGSim): pixel_shuffle (>4D peel) and the full floor/mod suite pass; elementwise/gemm/conv2d/reduce/softmax/MLP regress clean.
Route every MVIN/MVOUT -- both the MLIRKernel load/store backend path and the template path (gemm/conv/bmm/maxpool/cat) -- through emit_transfer, so a single decompose-transfer pass lowers all DMAs to memref.dma_start. This drops the get_dma_code emitter, the _dma_needs_transfer instance flag, and format_dma_op_attributes. togsim.transfer now also carries subtile_size and async, which decompose propagates onto the lowered dma_start (subtile filtered to the kept axes when unit dims collapse). For <=4D tiles decompose emits the descriptor directly on the original SRAM buffer (no collapse_shape) so the C++ -dma-fine-grained subtile split, which walks the SRAM operand, sees a direct buffer as before. Validated end-to-end (Spike + TOGSim) on elementwise, gemm (matmul/addmm), bmm, conv2d, group_conv, pool, cat, reduce, softmax, layernorm, batchnorm. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reflect a6b7ebb: the find_split_plan rank guard is gone (>4D index now lowers through the decompose-transfer affine.for peel, pixel_shuffle end-to-end), and the decompose-transfer peel <-> TOG incompatibility is resolved. Move it from Known-issues to Done; drop the >4D rank-guard caveat and the high-rank next-step. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Port mlir/test/lib/Analysis/TestDmaFineGrained.cpp to a Python out-of-line pass (passes/dma_fine_grained.py): split the matmul MVIN DMAs (input/weight/bias) into subtile affine.for nests and fuse the input/weight nests, replacing the C++ -dma-fine-grained pass. The MLIR Python bindings expose no IRMapping, so the fused nest is built directly (each DMA emitted with the fused induction vars) instead of cloning bodies -- structurally equivalent, not byte-exact SSA text. Pipeline: the single mlir-opt invocation is split around the Python pass (loop-padding -> run_fine_grained in place -> pytorchsim-to-vcix) in both the functional and gem5 paths (extension_codecache); vectorlane (systolic-array size) is threaded in for the lane-banked SRAM offset rescale. Validated against mlir-opt -dma-fine-grained on rank 2/3/4 fixtures (matmul / bmm / conv: same vcix dma_start and line counts) and end-to-end (Gem5+Spike+TOGSim): gemm/bmm/conv2d plus the resnet/transformer/vit/mlp models pass. Docs: dma-transfer-lowering.md -- >4D peel is affine.for + lane-banked physical SRAM offset via the last index operand; dma_fine_grained / build_tog are now Python passes; the #258 appendix is marked resolved.
…ings)
Port mlir/test/lib/Conversion/PyTorchSimToVCIX/TestPyTorchSimToVCIXConversion.cpp to
a Python out-of-line pass (passes/lower_to_vcix.py): lower linalg.matmul (gemm and
conv2d) and the transcendental math ops (exp/erf/tanh/sin/cos) to VCIX dialect ops
(RISC-V vector custom instructions), replacing the C++ -test-pytorchsim-to-vcix.
The C++ pass is a dialect conversion (applyPartialConversion); the bindings expose no
conversion framework, so each matchAndRewrite is reimplemented as imperative IR
rewriting. The VCIX dialect is not in the Python bindings, so vcix ops are created as
unregistered generic ops -- mlir-opt / mlir-translate (vcix registered) re-parse the
{}-attr generic form fine, and run_standard_lowering already consumes vcix output via
allow_unregistered_dialects, so this matches the existing pipeline.
Pipeline: the vcix mlir-opt invocation is dropped; run_to_vcix runs in-process after
the Python fine-grained pass and before the standard lowering (both functional and
gem5 paths in extension_codecache). mlir-opt now runs only -test-loop-padding.
Validated structurally against mlir-opt -test-pytorchsim-to-vcix (non-constant ops
byte-identical including the dma_wait tag maps, on gemm and conv2d fixtures) and
numerically end-to-end (Gem5+Spike+TOGSim allclose): gemm/bmm/conv2d (incl. large
N/K), softmax, exp/erf/sin/cos, and the resnet18/vit/transformer/mlp models.
dma-fine-grained and pytorchsim-to-vcix are now Python passes (dma_fine_grained, lower_to_vcix); update the docstring listing -- only test-loop-padding still runs in mlir-opt.
axis-split + graph-copy (on by default) linearize aligned floor/mod at the scheduling layer, so the index reaching get_dma_info is affine and the FloorDiv/ModularIndexing tile-divisibility branches there are never entered (measured: 0 entries across elementwise, gemm, bmm, conv, cat, floor/mod, reduce, attention). Remove those dead branches and their orphans: - the FloorDiv and ModularIndexing tile-forcing + RecompileSignal blocks - the implicit-ModularIndexing index rewrite and implicit_local_dims - the dead ModularIndexing branch in the dram_stride computation - is_modular_indexing, the write-only implicit_dim_size, unused import sys Kept: the non-floor/mod recompile paths (index-divisibility, indirect access, non-power-of-2 vec size), RecompileSignal, and the retry loop. The upstream implicit_dim_ops tile-forcing is left untouched (separate change). Validated end-to-end (Spike + TOGSim): elementwise, gemm, bmm, conv2d, group_conv, pool, cat, floor/mod suite, reduce, softmax, layernorm, batchnorm, gqa -- all pass, 0 recompiles. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…o CONFIG/2/3/4) Instead of packing the transfer params into four CONFIG asm instructions, build a 144-byte DMA descriptor as an 18xi64 memref.global (.data, non-constant so runtime fields can be written) and emit a single CONFIG_DESC that hands its address to the MVIN/MVOUT (which read the struct, matching the Spike-side torchsim_config_desc). This is the vehicle for the masked-DMA low/high clamp: dim_low/dim_high/fill are descriptor fields (currently defaulted, so behavior is unchanged), not new special registers. - _pack_desc encodes the fields into the struct's i64 slots (dim_size/low/high/mm_stride/ spad_stride/element_size/vlane/config_type/flags/indirect stride+esize). - descriptor globals are deduped by content; the indirect offset spad address is a runtime value, stored into slot 15 before the transfer (flags.bit0 = indirect). Requires the Spike torchsim_config_desc instruction (riscv-isa-sim). Validated functional (Spike) + trace (TOGSim): add, matmul (subtile+async), softmax, gather/scatter (indirect runtime store), constant_pad all pass.
Clamp each tile axis to its real DRAM extent in the DMA descriptor, so a tile that runs past the extent is filled with the consuming reduction's identity instead of MAC-ing garbage, and a store skips the ragged tail instead of writing the systolic pad region. Covers pointwise, pad, reduction, gather/scatter, matmul and conv, plus the fused-epilogue output store. Adds a golden unit test for the lower_transfer_to_gemmini lowering and a non-dividing regression suite to CI.
…ixel_shuffle trace) Both the Gemmini descriptor and the TOGSim DMA model cap at 4 tile dims, but a logical tile can exceed 4D (e.g. pixel_shuffle splits two spatial axes -> a 5D tile [1,1,2,4,2]). lower_transfer_to_gemmini reduces >4D as part of emitting Gemmini asm, but that runs only on the Spike/gem5 lowering path. The trace producer (build_tog) reads togsim.transfer BEFORE that, so it emitted a 5D DMA and TOGSim rejected it: "issued tile is not supported format.. tile.size: 5". Exposed by the pixel_shuffle case newly added to the wrapper1 CI suite. Add a peel_transfer POST_OPT pass that reduces every >4D togsim.transfer up front (before build_tog) while KEEPING it a togsim.transfer, so both consumers see <=4D. It mirrors lower_transfer_to_gemmini's two reductions: collapse unit dims when the effective rank is <=4 (memref.collapse_shape), else peel the outer effective dims into an affine.for nest with the lane-banked physical SRAM offset. The gemmini lowering keeps its own reduction as a now-defensive no-op. Validated on 128x128: test_floormod_axis_split (pixel_shuffle + 9 others), test_masked_nondividing, test_matmul, test_conv2d all pass.
Three MVIN over-reads on shapes whose tiles do not divide the extent: a cat concat-dim split, a vlane index_expr row stride on multi-row splits, and a 1-D reduction tile that exceeded its extent.
torch.sort's frontend lowering exposes the sorted VALUES as the sole scheduler-visible template output; the INDICES buffer is written in place by the same kernel (make_inplace) but is not registered as an output. For argsort the values output is dead, so Inductor DCEs the whole sort kernel and the indices buffer is returned uninitialized (all-zeros), silently corrupting any argsort result (e.g. DeepSeek MoE expert routing). Register the in-place indices write as a MutationOutput owned by the sort operation so the scheduler keeps the kernel alive whenever indices is consumed. OperationBuffer.get_outputs() hardcodes [self], so add a small MLIRTemplateBuffer subclass that can own extra MutationOutputs and have MLIRTemplateCaller.output_node build it; the sort lowering attaches the indices mutation via add_mutation_output. For templates with no mutation this is behaviorally identical to the base (get_outputs == [self]). Then gate the values MVOUT on the values output being live: in argsort it is pruned from the kernel args, so emitting its MVOUT referenced an undeclared %YV. The values scratchpad is still produced internally for the bitonic compare; only the dead DRAM store is skipped. Verified: t.argsort() now returns correct indices (was all-zeros); torch.sort with both outputs live is unchanged; test_sort, test_matmul, test_cat, test_indirect_access unaffected by the output_node change. Note: x[argsort] end to end additionally needs the indirect-gather codegen fix (separate).
…oad clamp A bare indirect index (x[idx]) left the loop-local symbol in the DMA base offset. Separately, reverse-engineering per-dim padding from a single flat offset is ill-posed -- under channels_last the stride-1 axis absorbs the remainder and yields an empty [low, high) box that zeroed the whole load.
codegen_epilogue_body() decided whether to emit the template buffer's MVOUT
by asking "did a fused epilogue already store something?":
if len(self.stores._lines) == 0:
template_store()
That is a proxy for the real question, "does anything outside this kernel still
read the template buffer?", and it is wrong in two of the four cases:
- epilogue + live template buffer -> store skipped, buffer never written
- no epilogue + dead template buffer -> store emitted for a pruned DRAM arg
The first case corrupts DeepSeek-V3's MoE gate. The gate is
sigmoid(mm(hidden, gate_w) + bias), so add+sigmoid fuse onto the GEMM template,
but the raw mm result is also read by a later gather kernel. Its buffer was
declared as a kernel output and never stored, so the gather read an
uninitialized buffer (all zeros), scrambling expert routing. Every config was
affected; only 8x8 crossed the test's loose atol=2e-1 (final Max abs diff
0.2546) while 32x32 landed just under it and passed.
Ask the real question instead. Inductor already answers it:
Kernel.remove_kernel_local_buffers() drops a buffer only when
Scheduler.can_buffer_be_removed_through_fusion() finds every user inside this
fused kernel. A buffer that survived removal therefore has an outside user and
must be stored. Record the template buffer's name in def_kernel() and
def_conv_kernel() (conv has its own implementation) and gate template_store()
on it, in both the main and the reduction_fusion branches. The proxy is gone.
Use the kernel-local self.removed_buffers, not V.graph.removed_buffers: the
former is what remove_buffer() populates, and the latter is still empty when
the store hook runs.
Note this makes a previously missing MVOUT issue, so DRAM traffic and cycle
counts rise for any model with a live template buffer under a fused epilogue.
That is the correct cost, not a regression, but reported cycles will shift.
Verified on the 8x8 config: test_deepseek_v3_base goes from Max abs diff 0.2546
to passing; a dead template buffer (relu(mm)) still prunes its DRAM arg and
emits a single MVOUT; the trace/TOGSim timing path runs with the extra store.
Regressions pass: test_bmm_reduction, test_matmul, test_bmm, test_conv2d,
test_cnn, test_group_conv, test_pool, test_reduce, test_softmax, test_layernorm,
test_mlp.
Lower the remaining pointwise ops, route math.log/atan to VCIX in the Python pass, and add tests/ops/elementwise/test_pointwise.py.
Bump the third-party pins to the releases shipping the new pointwise instructions (spike torchsim_vlog/vatan, gem5 CustomVlog/Vatan + decode).
Make int max/min reductions dtype-aware, lower torch.roll to narrow+cat and realize it so the consumer reads a plain affine index, flatten nested single-variable floor/mod in axis-split, define empty_strided_cpu in the generated wrapper header, and place the reduction axis outermost in the per-lane tile layout.
SwinV2 batch>1 used to fail codegen with "Unlinearized floor/mod in DMA index" because the shifted-window path (torch.roll composed with window partition/reverse) produced views axis-split could not linearize. With the roll->slice+cat lowering, the nested/shifted mod handling, and the reduction-axis layout fix, the whole backbone now compiles and matches CPU end to end (max diff ~5e-6 for image 64 / window 8 / batch 2). Add tests/models/test_swinv2.py, wire it as a self-hosted CI job next to the other model tests, and list SwinV2 in the README model coverage.
conv_multi_tile_mapping (used for batch > 1 convs) collects every tile that fits SPAD into tile_candidates and returns them sorted, but it guarded the "no mapping" error on max_used_spad_size instead of on tile_candidates. max_used_spad_size is only bumped when a candidate also satisfies max_k_h_w <= k_h, and max_k_h_w is initialized to K_W, so it only ever fires for the full-kernel (k_h == K_H) tile. When that tile overflows SPAD -- e.g. a CLIP ViT-B/32 patch conv (3->768, 32x32 stride 32) at batch > 1 -- the guard raised "Cannot find a valid mapping" even though smaller-k_h tiles fit and were already in tile_candidates. The mapping variable it tracks is never returned. Guard on `not tile_candidates` instead, so the conv is rejected only when no tile fits at all. The returned candidate list is unchanged, so convs that already worked are unaffected. Adds a batched patch-conv case to test_conv2d.py. Fixes issue #252.
With the conv-mapping fix, CLIP's vision transformer runs with batch > 1 and matches CPU end to end (max diff ~1e-5). Add tests/models/test_clip.py (CLIPVisionModel, batch 2, patch 32), wire it as a self-hosted CI job, and list CLIP in the README model coverage.
Instruction::_deps is a std::set<std::shared_ptr<Instruction>>, whose default comparator orders by pointer value. That only looked deterministic because the whole TileGraph was allocated up front in one monotonically growing run, so addresses happened to track creation order and fire() released dependents in creation order. It is not a property of the code. Running the same binary on the same trace under a different allocator changes the reported cycle count: kernel glibc malloc tcmalloc 677alforuj6 7852 7846 6lxncgaxtnh 3490 3491 7ascx6t7uu7 2888 2889 Order the sets by _global_inst_id instead. The id is intrinsic to the instruction, so the release order -- and the issue order and the cycle count -- no longer depend on the heap layout. All three allocators above now agree. Five of twelve real kernels shift by 1..6 cycles; a two-tenant model shifts by 416 of 1.4M (0.03%). Those numbers were allocator artefacts, not the model. This is also a prerequisite for building the TileGraph on demand, which frees and reuses tiles and would otherwise perturb the release order. (cherry picked from commit 1d067ef)
sram_finalize() tags each buffer version's last reader while walking the version map, which is ordered by version id, so an instruction that frees several versions receives them ascending. Make that an invariant of add_sram_release instead of an accident of the caller's iteration order. No behaviour change today. It lets the builder tag a version the moment it closes -- in buffer order -- and still produce a byte-identical graph, which the on-demand construction that follows relies on. (cherry picked from commit 7a52bc9)
…O(K) link()'s is_mm_accum UNION branch rescanned all of writers(b) on every one of the K matmuls that accumulate into b, to add a dep edge to the non-MATMUL producer (the init/bias that seeded the accumulator). But a UNION only ever appends MATMULs -- that producer is fixed until the next REPLACE -- so the rescan found the same one instruction every time and skipped everything else. Measured with a counter on an 8x8 conv whose K_tiles is 784 (50176 accumulating matmuls): the scan ran 1,258,840,576 times = N(N+1)/2, and only 50176 of those iterations produced an add_dep. 99.996 percent of the work was wasted. The real CI case (test_conv2d batch2 128->512 14x14 k7) has N=2458624, i.e. about 3e12 iterations, which is why it never finished. Replace the bare writers(b) vector with a BufferWriters struct that keeps the initializer alongside the producer set, so the UNION reads it in O(1) instead of rescanning. Both mutations (REPLACE, JOIN) are now methods, which is what keeps the two consistent -- the old vector let any caller assign writers[b] directly and silently desync a parallel map. The scan drops to 50176 on that case (25088x fewer iterations) and the set of add_dep edges is unchanged. Total execution cycles are identical on all three cases checked (33000, 129461, 202082), before and after. (cherry picked from commit 83f9f33)
core_trace_log::trace_instruction_line() takes formatted strings, so its arguments were built at every call site whether or not spdlog would keep them: each issued instruction paid a TraceLogTag::pad15() and a fmt::format over its tile dims and strides, which spdlog::trace() then dropped. The same holds for two spdlog::trace() calls in DMA::cycle that format the instruction's addr_name. Guard them with core_trace_log::trace_enabled(). Output is unchanged when the trace is on; when it is off (the default) an 8x8 conv2d simulation drops from 77.59s to 76.84s of CPU. Modest -- the string work was never the bottleneck -- but it is pure waste. (cherry picked from commit a220fb4)
…scan
Core's issue scan walks a tile's ready list every cycle and issues at most one
instruction. For a PRELOAD it counted the matmuls subscribed to its ISSUE event
by iterating the dependency set -- to size the weight-slot refcount -- before
asking whether a weight slot was even free. A preload that stalls waiting for a
slot is revisited on the next cycle, and the next, and recounted every time. One
preload's ISSUE set can hold hundreds of thousands of matmuls.
Measured on an 8x8 conv2d (CM16): that count was 83% of the whole simulation
(Core.cc:358-359 plus the std::set iteration inlined into it), and
matmul_consumers() was entered 762,064,838 times to perform 25,088 counts.
Two fixes:
* cache the count on the Instruction -- it is a property of the graph and
cannot change once the tile is built;
* ask pick_free_weight_sa() first, so a preload that cannot issue costs a
compare instead of a walk. Safe to reorder: try_occupy_sram() is a no-op for
a preload (the virtual SA-weights buffer is untracked), so nothing has been
reserved that would need giving back, and the call count -- hence the
round-robin cursor it advances -- is unchanged.
CM16 (conv, 803k cycles) 78.5s -> 29.3s (2.80x)
MM16 (gemm, 803k cycles) 55.6s -> 38.9s (1.43x)
MM36 (gemm, 9.8M cycles) ~1h56m -> 5m51s
Cycle counts unchanged: verified on 297 kernels.
The scan itself is now the top cost -- a stalled preload is still visited once
per cycle. Making the ready list resource-aware is the next step.
(cherry picked from commit e3707ff)
… demand togsim_dispatch(ctx, fn, iv, n) hands the runtime the work-item's function pointer and its induction variables, and the tile body reads nothing but ctx and iv. So running togsim_kernel records (fn, iv, core) per dispatch WITHOUT running any body, and any single work-item can be replayed later on its own. That is now all togsim_dispatch does: register the work-item and return. There is no mode to select -- LazyProducer::open() runs the kernel once (which only walks its outer loop nest), and run_item(i) replays work-item i, returning its record stream in a reused buffer. No whole-kernel stream is ever materialized. run_producer(), the eager whole-kernel run, becomes every work-item's records concatenated. That is an identical stream: a producer emits nothing outside a dispatch (checked: 0 records). It stays only so its one caller -- the TileGraph builder -- keeps compiling; the next commit switches that caller over and deletes it.
The builder materialized every dispatch tile up front and handed the finished
graph to the Simulator, even though a Core only ever holds one or two tiles
(Core::max_concurrent). Peak memory scaled with the dispatch count: an 8x8
conv2d(x[2,128,14,14], w[512,128,7,7], pad=3) -- 49 dispatch tiles, 17.7M
instructions, 37.0M dependency edges -- needed 12.7 GiB and was SIGKILLed in CI,
while the identical torch.mm needed 1.8 GiB.
Build a tile only when a Core asks for work:
* trace_to_tilegraph now takes the producer .so directly. BuildState::index()
records each dispatch as a WorkItem and collects buffer footprints; it makes
no Instruction.
* BuildState::build_one_tile() materializes the next work-item's tile on
demand, wired to the graph via TileGraph::set_tile_source. A finished
subgraph is dropped instead of parked in a keep-everything vector, which is
what releases a tile's Instructions once the Core is done.
* sram_schedule() precomputes each buffer version's last reader in a pass that
allocates nothing, so the builder no longer retains every reader of every
version to end-of-stream (7.5M shared_ptrs on the conv) just to free them.
Sound because a dispatch tile is dependency-closed: flush() resets
writers/seeds/tag maps at every tile boundary, so no dependency edge crosses
tiles (measured: 0 of 37.0M). Buffer versions do cross tiles and their
bookkeeping is kept; only where it lives moved.
Peak graph-build memory: MM16 232->71 MiB, MM36 1.83 GiB->477 MiB (3.9x),
CI36 12.7 GiB->537 MiB (24.3x), and it no longer grows with the dispatch count.
Cycle counts unchanged (verified on the 2-core 2-partition config and others).
Deletes run_producer()/RunResult with its last caller: nothing materializes a
whole-kernel record stream any more.
Instruction's constructor took its five vectors by value and copied them into the members, allocating a second buffer per vector for every DMA and barrier. And prepare_tag_key() pushed two elements onto an empty vector, allocating at capacity 1 and immediately reallocating at 2. Move instead of copy, and reserve the exact tag-key size. These are millions of allocations on the graph-build path that never had to happen. No behaviour change.
…ssue Core's issue scan walked the ready queue from the front every cycle and issued the first instruction it could. The ones it walked past are blocked -- and only ever on one of two things: free spad bytes (try_occupy_sram) or a free SA weight slot (pick_free_weight_sa). Every other path in the scan issues. So the scan re-walked the same blocked prefix on its way to every single issue. Measured on an 8x8 conv (the CI wrapper3 config, where the small array keeps preloads waiting on weight slots so the prefix grows to ~6000): case steps walked issues steps per issue CM16 762,271,822 232,072 3284 MM16 1,070,436,422 302,112 3543 Give each Tile a cursor at the last instruction the scan found blocked and let the next scan resume after it. Blocked stays blocked until a resource GROWS, so the cursor is keyed on the resource levels (_sram_used, _weight_free) at the time, not on a free event: a freed weight slot that the woken preload immediately takes leaves the rest of the prefix blocked, and the cursor stays good. It is dropped only when its own instruction is erased. New ready instructions are appended, so they always land after the cursor. case steps walked steps per issue CM16 257,156 1.11 MM16 402,460 1.33 The scan issues the same instruction on the same cycle -- it just stops re-deriving what it already knew. Total execution cycles are identical on every case checked: CM16 803,796, MM16 803,345, MM36 9,835,778, CI36 39,449,132. The 8x8 conv2d of tests/ops/conv/test_conv2d.py (CI36) goes from 4h36m to 7m19s. That job currently fails on develop (SIGKILL, out of memory) and still failed after the on-demand TileGraph (3h job limit, simulation unfinished).
A conv input node can be a ReinterpretView. call_kernel passes template args
by buffer *name*, so the wrapper receives the base buffer, whose rank/shape may
differ from the view codegen assumed. The wrapper then did
padded_shape = list(X.shape)
padded_shape[3] += 2 * PADDING_W # IndexError on a 3D base buffer
This fires whenever the view is free: a contiguous (B, N, C) buffer already is
the channels_last layout of (B, C, H, W), so inductor inserts no materializing
copy. That is exactly the transformer (B, N, C) -> (B, C, H, W) pattern, e.g.
SegFormer efficient attention.
Stop inferring geometry from the runtime tensor. Bake the size/stride/offset the
codegen used into the wrapper and rebuild the logical NCHW inputs from it, so the
wrapper is correct whether it is handed the base buffer, an already reinterpreted
view, or a materialized copy (reinterpret is idempotent in all three cases). The
conv templates were the only place reading .shape at runtime; everything else
already derives geometry from the node layout at codegen time.
No change to call_kernel, mlir_argdefs, the MLIR signature, or arg_attributes.
Also allocate the padding buffer directly on the input device with the input
dtype: torch.zeros() defaulted to f32 and to(device=...) round-tripped via CPU.
Covers a conv fed by a free ReinterpretView of a lower-rank buffer. Fails with IndexError on the pre-fix wrapper and passes after it.
ConvNextV2 with batch > 1 died in codegen with "Index names length
mismatch: 2 != 3". The scheduler had approved fusing a reduction epilogue
into a GEMM whose 2-D (M rows x N cols) frame cannot express it, and
set_ranges then found more loop ranges than the template's coordinate map
has entries.
It approved it because the eligibility check read the reduction's stride
out of the node's repr:
stride = [i.strip()[:-1].split(",")[-1].strip()
for i in str(node).split("\n") if "r0" in i][1]
The intent (reject a reduction over the contiguous, stride-1 axis, which
the GEMM reduction template cannot codegen) was right, but picking the
second line that mentions "r0" lands on the wrong index expression once
the node has more than two dimensions -- so ConvNextV2's channels-first
LayerNorm mean was let through. Reading the stride off the LoopBody's
index expression instead makes it decline for the right reason. Note a
MemoryDep's index cannot be used here: it is normalized to a flat
contiguous access and no longer carries the per-axis strides.
Fixing that exposed the deeper problem: only a template knows its output
coordinate frame, yet can_fuse_horizontal hardcoded a case per template
kind and re-guessed that frame. It also mutated nodes (revert_group) from
inside a predicate the scheduler calls speculatively for every candidate
pair, carried a dead isinstance(MaxPoolTemplate) special case, and
declined without ever saying why.
So the decision moves to the templates, following how Inductor's own
CUTLASS and CPP backends do it:
- MLIRTemplate.try_fuse_epilogue / try_fuse_prologue own every condition
that depends on the frame -- including the config gates -- and return a
FusionPlan or None. They are pure; a node mutation the fusion needs is
deferred into the plan's remap and applied from MLIRScheduling.fuse(),
the commit hook Inductor calls once it really fuses. The plan is
recomputed there rather than cached, since can_fuse runs many times per
pair and node identities change as fusions land.
- The scheduler keeps only graph facts: which node carries the template,
the direction, and that this is a single-node to single-node fusion.
Case 1 (pointwise) and Case 2 (reduction) collapse into one delegation.
- Every decline logs a reason, like Inductor's WhyNoFuseNames. Declining
is a normal answer: upstream declines reduction epilogues outright.
- A template declares REDUCTION_EPILOGUE_ALIASING, the coordinate map its
reduction-epilogue codegen addresses. render() consumes it, and its
length is exactly what set_ranges asserts against -- so the fusion gate
is that assert, raised as a decline instead of a crash. Templates
without one cannot absorb a reduction epilogue, which retires the
support_reduction_fusion flag. GEMM's map has two entries, BMM's three.
When the rows do not fit, LoopBody.merge_loops() (which returns a new
body, so this stays pure) says whether merging them would; if it would,
the merge becomes the plan's remap.
tests/ops/fusion covers pointwise, reduction, prologue, conv and
attention fusion and is unchanged, as are the kernels each of them fuses.
Adds a matmul whose reduction is over the contiguous axis: it must not
fuse, and wrongly fusing it returns wrong values rather than failing to
compile.
ConvNextV2 batch > 1 no longer hits the assertion; it now stops at an
unrelated SPAD overflow, so issue #255's report is not fully closed.
SpadOverflowError was raised with no arguments and the only diagnostic was a logger.debug line that is off by default, so a failing compile said nothing beyond "SPAD overflow occurred." and gave no way to tell which kernel, which tiling, or by how much. load() already has the measured usage, the budget, the tile size and the kernel's origins in scope. Put them in the exception message.
Spad globals follow a fixed convention: the MLIR memref carries the full tile shape (all lanes), the .spad C header that Spike links declares the per-lane slice, and the gem5 header declares the full tile. codegen_spad_buffer() emits exactly that, tile_numel_per_lane and tile_numel_per_lane * vector_lane. index_expr() had the two headers swapped. The iota scratch buffer it allocates for vectorized index arithmetic declared compute_vec_size * vector_lane entries in the .spad header and compute_vec_size entries in the gem5 header. So the buffer Spike sees is vector_lane times too large, and the one gem5 sees is one element too small: the initializer stores two elements per iteration, so its last iteration writes one slot past compute_vec_size. The oversize side is what breaks compiles. Since the spad guard tightened to spad/2 for double buffering, a kernel with compute_vec_size 64 spends 64 KB of its 64 KB budget on an iota that only ever uses 64 entries. ConvNextV2 with batch > 1 fails there: its channels-first LayerNorm var_mean kernel needs 81984 bytes/lane, of which 65536 is the iota and only 16448 is real data. Declare compute_vec_size + 1 entries per lane, and vector_lane times that for gem5. The var_mean kernel now measures 16968 bytes/lane.
Add tests/models/test_convnextv2.py, a self-hosted CI job for it, and a model coverage row. batch=2 with depths=[1,1,1,1] and image_size=64 now matches CPU end to end, max abs diff 5.0e-06. The test keeps the runner on self-hosted like the other model tests: the depthwise 7x7 conv lowers to one gather kernel per tap, so the graph is wide.
axis-split's affine-only contract (docs/axis-split-scheduling.md) linearizes aligned FloorDiv/ModularIndexing into per-axis affine indices upstream, so the load/store index reaching codegen is already pure affine. The convert_index method (which lowered floor/mod view indices to affine.apply with a constant divisor and a single free symbol) and the helper that guarded it are therefore dead -- a tripwire at every floor/mod branch never fired across the view/op test suite. Remove convert_index entirely and inline its only live behaviour at the call sites in _convert_sympy_to_mlir_expr and parse_index_list. A residual FloorDiv/ModularIndexing now fails loudly via the whole-expression guard in each of those functions (mirroring the DMA-index assert) instead of silently mis-lowering. Also drop the assumption-stripping Symbol(str(...)) + expr.replace round-trip in the affine-map builder: it only mattered when convert_index transformed the term, and is now a verified no-op (indices and the affine string depend only on symbol names). Drop the now-unused re import. Verified: the full view/reshape/transpose suite (test_floormod_axis_split incl. group_norm/repeat/repeat_interleave/mixed-radix/pixel_shuffle, transpose2D/3D, view3D_2D, cat) plus add/matmul/reduce/softmax/layernorm/batchnorm pass unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SfwHCV7TaX4s9xkn8i7anG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.