[TOGSim] Build the TileGraph on demand, one dispatch tile at a time#301
Merged
Conversation
YWHyuk
force-pushed
the
togsim/on-demand-tilegraph
branch
3 times, most recently
from
July 13, 2026 10:57
1626afb to
bb295f7
Compare
YWHyuk
commented
Jul 13, 2026
| // (fn, iv, core) and returns WITHOUT running the tile body, so only the | ||
| // kernel's outer loop nest runs and no records are emitted. run_item() replays | ||
| // an individual work-item later. | ||
| _ctx->index = &_items; |
Collaborator
Author
There was a problem hiding this comment.
This mechanism looks a little bit awkward
YWHyuk
force-pushed
the
togsim/on-demand-tilegraph
branch
2 times, most recently
from
July 13, 2026 12:42
cdd5385 to
602f7f9
Compare
Instruction::_deps is a std::set<std::shared_ptr<Instruction>>, whose default comparator orders by pointer value. That only looked deterministic because the whole TileGraph was allocated up front in one monotonically growing run, so addresses happened to track creation order and fire() released dependents in creation order. It is not a property of the code. Running the same binary on the same trace under a different allocator changes the reported cycle count: kernel glibc malloc tcmalloc 677alforuj6 7852 7846 6lxncgaxtnh 3490 3491 7ascx6t7uu7 2888 2889 Order the sets by _global_inst_id instead. The id is intrinsic to the instruction, so the release order -- and the issue order and the cycle count -- no longer depend on the heap layout. All three allocators above now agree. Five of twelve real kernels shift by 1..6 cycles; a two-tenant model shifts by 416 of 1.4M (0.03%). Those numbers were allocator artefacts, not the model. This is also a prerequisite for building the TileGraph on demand, which frees and reuses tiles and would otherwise perturb the release order. (cherry picked from commit 1d067ef)
sram_finalize() tags each buffer version's last reader while walking the version map, which is ordered by version id, so an instruction that frees several versions receives them ascending. Make that an invariant of add_sram_release instead of an accident of the caller's iteration order. No behaviour change today. It lets the builder tag a version the moment it closes -- in buffer order -- and still produce a byte-identical graph, which the on-demand construction that follows relies on. (cherry picked from commit 7a52bc9)
…O(K) link()'s is_mm_accum UNION branch rescanned all of writers(b) on every one of the K matmuls that accumulate into b, to add a dep edge to the non-MATMUL producer (the init/bias that seeded the accumulator). But a UNION only ever appends MATMULs -- that producer is fixed until the next REPLACE -- so the rescan found the same one instruction every time and skipped everything else. Measured with a counter on an 8x8 conv whose K_tiles is 784 (50176 accumulating matmuls): the scan ran 1,258,840,576 times = N(N+1)/2, and only 50176 of those iterations produced an add_dep. 99.996 percent of the work was wasted. The real CI case (test_conv2d batch2 128->512 14x14 k7) has N=2458624, i.e. about 3e12 iterations, which is why it never finished. Replace the bare writers(b) vector with a BufferWriters struct that keeps the initializer alongside the producer set, so the UNION reads it in O(1) instead of rescanning. Both mutations (REPLACE, JOIN) are now methods, which is what keeps the two consistent -- the old vector let any caller assign writers[b] directly and silently desync a parallel map. The scan drops to 50176 on that case (25088x fewer iterations) and the set of add_dep edges is unchanged. Total execution cycles are identical on all three cases checked (33000, 129461, 202082), before and after. (cherry picked from commit 83f9f33)
core_trace_log::trace_instruction_line() takes formatted strings, so its arguments were built at every call site whether or not spdlog would keep them: each issued instruction paid a TraceLogTag::pad15() and a fmt::format over its tile dims and strides, which spdlog::trace() then dropped. The same holds for two spdlog::trace() calls in DMA::cycle that format the instruction's addr_name. Guard them with core_trace_log::trace_enabled(). Output is unchanged when the trace is on; when it is off (the default) an 8x8 conv2d simulation drops from 77.59s to 76.84s of CPU. Modest -- the string work was never the bottleneck -- but it is pure waste. (cherry picked from commit a220fb4)
…scan
Core's issue scan walks a tile's ready list every cycle and issues at most one
instruction. For a PRELOAD it counted the matmuls subscribed to its ISSUE event
by iterating the dependency set -- to size the weight-slot refcount -- before
asking whether a weight slot was even free. A preload that stalls waiting for a
slot is revisited on the next cycle, and the next, and recounted every time. One
preload's ISSUE set can hold hundreds of thousands of matmuls.
Measured on an 8x8 conv2d (CM16): that count was 83% of the whole simulation
(Core.cc:358-359 plus the std::set iteration inlined into it), and
matmul_consumers() was entered 762,064,838 times to perform 25,088 counts.
Two fixes:
* cache the count on the Instruction -- it is a property of the graph and
cannot change once the tile is built;
* ask pick_free_weight_sa() first, so a preload that cannot issue costs a
compare instead of a walk. Safe to reorder: try_occupy_sram() is a no-op for
a preload (the virtual SA-weights buffer is untracked), so nothing has been
reserved that would need giving back, and the call count -- hence the
round-robin cursor it advances -- is unchanged.
CM16 (conv, 803k cycles) 78.5s -> 29.3s (2.80x)
MM16 (gemm, 803k cycles) 55.6s -> 38.9s (1.43x)
MM36 (gemm, 9.8M cycles) ~1h56m -> 5m51s
Cycle counts unchanged: verified on 297 kernels.
The scan itself is now the top cost -- a stalled preload is still visited once
per cycle. Making the ready list resource-aware is the next step.
(cherry picked from commit e3707ff)
… demand togsim_dispatch(ctx, fn, iv, n) hands the runtime the work-item's function pointer and its induction variables, and the tile body reads nothing but ctx and iv. So running togsim_kernel records (fn, iv, core) per dispatch WITHOUT running any body, and any single work-item can be replayed later on its own. That is now all togsim_dispatch does: register the work-item and return. There is no mode to select -- LazyProducer::open() runs the kernel once (which only walks its outer loop nest), and run_item(i) replays work-item i, returning its record stream in a reused buffer. No whole-kernel stream is ever materialized. run_producer(), the eager whole-kernel run, becomes every work-item's records concatenated. That is an identical stream: a producer emits nothing outside a dispatch (checked: 0 records). It stays only so its one caller -- the TileGraph builder -- keeps compiling; the next commit switches that caller over and deletes it.
The builder materialized every dispatch tile up front and handed the finished
graph to the Simulator, even though a Core only ever holds one or two tiles
(Core::max_concurrent). Peak memory scaled with the dispatch count: an 8x8
conv2d(x[2,128,14,14], w[512,128,7,7], pad=3) -- 49 dispatch tiles, 17.7M
instructions, 37.0M dependency edges -- needed 12.7 GiB and was SIGKILLed in CI,
while the identical torch.mm needed 1.8 GiB.
Build a tile only when a Core asks for work:
* trace_to_tilegraph now takes the producer .so directly. BuildState::index()
records each dispatch as a WorkItem and collects buffer footprints; it makes
no Instruction.
* BuildState::build_one_tile() materializes the next work-item's tile on
demand, wired to the graph via TileGraph::set_tile_source. A finished
subgraph is dropped instead of parked in a keep-everything vector, which is
what releases a tile's Instructions once the Core is done.
* sram_schedule() precomputes each buffer version's last reader in a pass that
allocates nothing, so the builder no longer retains every reader of every
version to end-of-stream (7.5M shared_ptrs on the conv) just to free them.
Sound because a dispatch tile is dependency-closed: flush() resets
writers/seeds/tag maps at every tile boundary, so no dependency edge crosses
tiles (measured: 0 of 37.0M). Buffer versions do cross tiles and their
bookkeeping is kept; only where it lives moved.
Peak graph-build memory: MM16 232->71 MiB, MM36 1.83 GiB->477 MiB (3.9x),
CI36 12.7 GiB->537 MiB (24.3x), and it no longer grows with the dispatch count.
Cycle counts unchanged (verified on the 2-core 2-partition config and others).
Deletes run_producer()/RunResult with its last caller: nothing materializes a
whole-kernel record stream any more.
Instruction's constructor took its five vectors by value and copied them into the members, allocating a second buffer per vector for every DMA and barrier. And prepare_tag_key() pushed two elements onto an empty vector, allocating at capacity 1 and immediately reallocating at 2. Move instead of copy, and reserve the exact tag-key size. These are millions of allocations on the graph-build path that never had to happen. No behaviour change.
YWHyuk
force-pushed
the
togsim/on-demand-tilegraph
branch
from
July 13, 2026 12:57
602f7f9 to
bb2080a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Build the TileGraph one dispatch tile at a time instead of all at once, and fix
the hot loops that a large conv then exposed. A big 8x8 conv2d that used to be
SIGKILLed in CI (peak 12.7 GiB, never finished) now completes at ~1.7 GiB, and
the simulation is materially faster. Cycle counts are unchanged.
Eight commits, each buildable and cycle-verified on its own.
The problem
A conv is a GEMM: M = batchHW, N = C_out, K = C_inkhkw. So the CI conv2d
(x[2,128,14,14], w[512,128,7,7], pad=3) is the identical 392x512x6272 multiply
as its matmul twin. Yet the conv had two problems:
17.7M instructions, 37.0M dependency edges), even though a Core only ever
holds one or two tiles. The graph hit 12.7 GiB and the OOM killer took it.
On-demand construction
togsim_dispatch(ctx, fn, iv, n) hands the runtime the work-item's function
pointer and its induction variables, and the tile body reads nothing but ctx and
iv. So one indexing pass records (fn, iv, core) per dispatch WITHOUT running the
body, and any single work-item can be replayed on its own later.
and returns its records.
for work, via TileGraph::set_tile_source. A finished subgraph is dropped
instead of parked in a keep-everything vector -- that is what releases a
tile's Instructions.
allocates nothing, so the builder no longer retains every reader of every
version to end-of-stream (7.5M shared_ptrs on the conv) just to free them.
Sound because a dispatch tile is dependency-closed: flush() resets writers/seeds/
tag maps at every tile boundary, so no dependency edge crosses tiles (measured:
0 of 37.0M). Buffer versions DO cross tiles and their bookkeeping is kept; only
where it lives moved.
Graph-build peak RSS
MM16 232 MiB -> 71 MiB
MM36 1.83 GiB -> 477 MiB (3.9x)
CI36 12.7 GiB -> ~0.5 GiB (24x)
Peak no longer scales with the dispatch count, so a bigger conv cannot run out.
Speed
A CPU profile of a conv simulation pointed at one loop: the issue scan counts the
matmuls reusing a preload's weight by iterating its dependency set, every cycle
the preload stalls waiting for a slot. One preload's set can hold hundreds of
thousands of matmuls. On CM16 that was 83% of the whole simulation;
matmul_consumers() was entered 762,064,838 times to perform 25,088 counts. Cache
the count (a graph property) and check the weight slot first.
CM16 (conv, 803k cycles) 78.5s -> 29.3s CPU (2.8x)
MM16 (gemm, 803k cycles) 55.6s -> 38.9s CPU (1.4x)
MM36 (gemm, 9.8M cycles) ~1h56m -> 5m51s wall
Two smaller cleanups are included: the instruction vectors are moved rather than
copied, and the instruction trace strings are no longer formatted when the trace
is off.
A latent bug this surfaced
Instruction::_deps was a std::set<shared_ptr>, ordered by pointer value. That
only looked deterministic because the whole graph was allocated in one
monotonically growing run, so addresses tracked creation order by accident.
Running the same binary on the same trace under a different allocator moved the
cycle count (677alforuj6: glibc 7852 vs tcmalloc 7846). Ordering the set by the
intrinsic _global_inst_id makes the release order -- and the cycle count --
independent of heap layout. It is also a prerequisite for on-demand construction,
which frees and reuses tiles.
Verification
2-partition config: CM16 803,796 / MM16 803,345 / MM36 9,835,778.
MACs / (882 arrays) = 39,335,938.
Not addressed here
The upstream cause of the conv's size is untouched: batch=2 pads to TILE_M=8, so
the conv still emits 4x the instructions of its matmul twin. That is a frontend
mapping fix, separate from this simulator-side work. The issue scan is now the
top remaining cost (a stalled preload is still visited once per cycle); making
the ready list resource-aware would need its own cycle-equivalence proof.
The producer is replayed three times (index, version schedule, build). It is a
pure emitter so this is cheap and memory stays O(tiles in flight), but the two
dry passes cannot be merged: the version schedule needs every buffer's size,
which is only known after all DMA records have been seen.
🤖 Generated with Claude Code
https://claude.ai/code/session_014PxddUzqSZzZBcKTC8uahV