Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
6a4c233
Add dual memory planner support
FangRui0 Jul 9, 2026
6f00019
Implement modern memplan five-gate reuse checks
FangRui0 Jul 13, 2026
c789dd5
Fix cross-root local overlap sync analysis build
FangRui0 Jul 13, 2026
e2d8d08
Fix tile-native memplan CI regressions
FangRui0 Jul 14, 2026
73d966d
Avoid treating DPS outputs as ignored A5 tmp
FangRui0 Jul 14, 2026
e0a7f1a
Fix tile subview materialization for strided windows
FangRui0 Jul 14, 2026
9a16d4b
Treat tile bitcast as memory-planning alias
FangRui0 Jul 14, 2026
ebc9a29
Lower PTO castptr in EmitC backend
FangRui0 Jul 14, 2026
f9fd624
Fix EmitC castptr ptr/int lowering
FangRui0 Jul 16, 2026
d3faad7
Fix flaky plan memory lit checks
FangRui0 Jul 16, 2026
8ffa703
Keep multi-tile buffers native through memory planning
FangRui0 Jul 17, 2026
3ab2d27
Keep declared tile handles native
FangRui0 Jul 17, 2026
9bf81ef
Keep tile reshape views native
FangRui0 Jul 17, 2026
72e6f15
Keep tile bitcast views native
FangRui0 Jul 17, 2026
e9af390
Keep tile valid-shape metadata native
FangRui0 Jul 17, 2026
86a4205
Keep tile subviews native through sync
FangRui0 Jul 17, 2026
dfb80fb
Keep tile address extraction native
FangRui0 Jul 17, 2026
c321132
Keep GM tensor views native through sync
FangRui0 Jul 18, 2026
3bd0577
Keep simple unary ops tile-native
FangRui0 Jul 18, 2026
142422d
Keep transcendental unary ops tile-native
FangRui0 Jul 18, 2026
d697880
Keep TLRelu tile-native
FangRui0 Jul 18, 2026
a40aa1b
Keep TMax and TMin tile-native
FangRui0 Jul 18, 2026
9fbd807
Keep TMaxS and TMinS tile-native
FangRui0 Jul 18, 2026
949fd3e
Keep TAdd and TMul tile-native
FangRui0 Jul 18, 2026
c9f3234
Keep TAddS and TMulS tile-native
FangRui0 Jul 18, 2026
415886d
Make VPTO cache control checks order independent
FangRui0 Jul 18, 2026
dfcbe1b
Keep TAnd and TOr tile-native
FangRui0 Jul 18, 2026
af9e591
Keep TAndS and TOrS tile-native
FangRui0 Jul 18, 2026
d277e8c
Keep TCmp and TCmpS tile-native
FangRui0 Jul 18, 2026
cb70d49
Keep TDiv and TDivS tile-native
FangRui0 Jul 18, 2026
0a493ab
Keep TConcat ops tile-native
FangRui0 Jul 18, 2026
5e6424b
Keep TPart arithmetic ops tile-native
FangRui0 Jul 18, 2026
91beee9
Keep TColExpand ops tile-native
FangRui0 Jul 18, 2026
961cfda
Keep TExpands tile-native
FangRui0 Jul 18, 2026
78c5aad
Keep tile scalar access ops tile-native
FangRui0 Jul 18, 2026
cf8a9aa
Keep TExtract and TInsert tile-native
FangRui0 Jul 18, 2026
32ac9ef
Keep TExtractFP and TInsertFP tile-native
FangRui0 Jul 18, 2026
0d9372b
Keep carry add ops tile-native
FangRui0 Jul 18, 2026
ca65b37
Keep TMovFP tile-native
FangRui0 Jul 18, 2026
018e097
Keep TFillPad ops tile-native
FangRui0 Jul 18, 2026
a73aafa
Keep TLoad and TStore local tiles native
FangRui0 Jul 18, 2026
8115ef4
Keep TColMax and TColMin tile-native
FangRui0 Jul 18, 2026
9a5b53d
Keep TColSum tile-native
FangRui0 Jul 18, 2026
88ab4c5
Keep TCvt tile-native
FangRui0 Jul 18, 2026
738a59e
Keep gather ops tile-native
FangRui0 Jul 18, 2026
7f9f998
Keep TCI tile-native
FangRui0 Jul 18, 2026
ba29931
Keep TPrint tile-native
FangRui0 Jul 18, 2026
d834495
Keep TQuant tile-native
FangRui0 Jul 18, 2026
9b218cd
Keep TTrans tile-native
FangRui0 Jul 18, 2026
143ec49
Keep TMrgSort tile-native
FangRui0 Jul 18, 2026
5673c6e
Keep TAssign tile-native
FangRui0 Jul 18, 2026
5874cfa
Keep MGather and MScatter tile-native
FangRui0 Jul 18, 2026
bd19835
Keep subtraction ops tile-native
FangRui0 Jul 18, 2026
a668900
Keep basic floating-point ops tile-native
FangRui0 Jul 18, 2026
d071668
Keep shift ops tile-native
FangRui0 Jul 18, 2026
7a030f0
Keep TPartMax and TPartMin tile-native
FangRui0 Jul 18, 2026
8ea2035
Keep remaining TColExpand ops tile-native
FangRui0 Jul 19, 2026
f853e9d
Keep TRowExpand ops tile-native
FangRui0 Jul 19, 2026
b208736
Keep reduction ops tile-native
FangRui0 Jul 19, 2026
2259172
Keep arg-reduction ops tile-native
FangRui0 Jul 19, 2026
e2f682d
Keep pow and rsqrt ops tile-native
FangRui0 Jul 19, 2026
e3e619a
Keep scratch elementwise ops tile-native
FangRui0 Jul 19, 2026
1dfd65d
Keep select ops tile-native
FangRui0 Jul 19, 2026
fa6fe53
Keep XOR ops tile-native
FangRui0 Jul 19, 2026
6c583db
Keep TSort32 tile-native
FangRui0 Jul 19, 2026
efa96b3
Keep TQuantMx tile-native
FangRui0 Jul 19, 2026
0e5f0e1
Keep TAxpy and TDequant tile-native
FangRui0 Jul 19, 2026
bd65abe
Keep tile movement metadata ops native
FangRui0 Jul 19, 2026
18e32ae
Keep TMov tile-native
FangRui0 Jul 19, 2026
920f18d
Keep TStoreFP tile-native
FangRui0 Jul 19, 2026
cc8e539
Keep vector specialty ops tile-native
FangRui0 Jul 19, 2026
d399f36
Keep matmul ops tile-native
FangRui0 Jul 19, 2026
9dd2216
Fix matmul tile-native type checks
FangRui0 Jul 20, 2026
6f2ae90
Keep gemv ops tile-native
FangRui0 Jul 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
655 changes: 655 additions & 0 deletions docs/designs/ptoas-largest-first-fit-four-gates-memplan-design.md

Large diffs are not rendered by default.

377 changes: 377 additions & 0 deletions docs/designs/ptoas-tile-native-mainline-op-migration.md

Large diffs are not rendered by default.

13 changes: 0 additions & 13 deletions include/PTO/IR/PTOAttrs.td
Original file line number Diff line number Diff line change
Expand Up @@ -600,19 +600,6 @@ def PTO_EventAttr : PTO_Attr<"Event", "event"> {
}];
}

//===----------------------------------------------------------------------===//
// Plan Memory Mode
//===----------------------------------------------------------------------===//

def PTO_LOCALMEMPLAN : I32EnumAttrCase<"LOCAL_MEM_PLAN", 0>;
def PTO_GLOBALWORKSPACEPLAN : I32EnumAttrCase<"GLOBAL_WORKSPACE_PLAN", 1>;

def PTO_MemPlanModeEnum : PTO_I32Enum<
"MemPlanMode", "Mem Plan Mode", [
PTO_LOCALMEMPLAN,
PTO_GLOBALWORKSPACEPLAN
]>;

//===----------------------------------------------------------------------===//
// Pad Mode for LoadOp
//===----------------------------------------------------------------------===//
Expand Down
11 changes: 8 additions & 3 deletions include/PTO/IR/PTOMultiBuffer.h
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,10 @@
//===- PTOMultiBuffer.h - Shared constants for multi-buffer ----*- C++ -*-===//
//
// Shared constants for the multi-buffer expression scheme:
// - `kPtoMultiBufferAttrName` is the memref-level attribute name written by
// PTOViewToMemref when lowering an `alloc_multi_tile` op. PlanMemory and
// downstream passes read it to reserve N physical slots.
// - `kPtoMultiBufferAttrName` is the legacy memref-level slot-count
// attribute.
// - `kPtoMultiBufferAddrsAttrName` is the internal address list written
// directly on tile-native `pto.alloc_multi_tile` by PlanMemory.
// - `kPtoMultiBufferMaxNum` is the upper bound on the slot count N. It is
// kept in lock-step with the InsertSync `MAX_MULTI_BUFFER_NUM`.
//
Expand All @@ -29,6 +30,10 @@ namespace pto {
inline constexpr llvm::StringLiteral kPtoMultiBufferAttrName =
"pto.multi_buffer";

/// Internal DenseI64ArrayAttr containing one byte address per physical slot.
inline constexpr llvm::StringLiteral kPtoMultiBufferAddrsAttrName =
"pto.multi_buffer_addrs";

/// Upper bound for N; must stay consistent with `MAX_MULTI_BUFFER_NUM` in
/// insert-sync.
inline constexpr unsigned kPtoMultiBufferMaxNum = 16;
Expand Down
40 changes: 18 additions & 22 deletions include/PTO/IR/PTOOps.td
Original file line number Diff line number Diff line change
Expand Up @@ -343,19 +343,18 @@ def AllocTileOp : PTO_Op<"alloc_tile", [AttrSizedOperandSegments]> {
//===----------------------------------------------------------------------===//
//
// `pto.alloc_multi_tile` declares an N-slot logical tile buffer. The actual
// physical allocation of N slots is performed by PTOPlanMemory once the op
// has been lowered to `memref.alloc {pto.multi_buffer = N : i32}` by
// PTOViewToMemref.
// physical allocation of N slots is performed directly by PTOPlanMemory.
// The op remains tile-native until PTOResolveBufferSelect materializes an
// addressed `pto.alloc_tile` for each selected slot.
//
// `pto.multi_tile_get` is the *only* way to consume a `multi_tile_buf`: it
// returns a regular `tile_buf` view onto the chosen slot. The slot index
// may be a constant or any SSA `index` value; PTOAS does not synthesize
// `iv mod N` automatically -- the user expression IS the slot selector.
//
// `pto.slot_marker` is an internal op materialized by PTOViewToMemref to
// thread the slot SSA from `multi_tile_get` through the memref layer down
// to PlanMemory / sync analysis. It is not intended for direct frontend
// use.
// `pto.slot_marker` remains as an internal compatibility op for legacy
// memref-based multi-buffer IR. PTOViewToMemref does not create it for
// tile-native `pto.multi_tile_get`.

def AllocMultiTileOp : PTO_Op<"alloc_multi_tile", [AttrSizedOperandSegments]> {
let summary = "Allocate an N-slot multi-buffer tile";
Expand Down Expand Up @@ -415,9 +414,9 @@ def MultiTileGetOp : PTO_Op<"multi_tile_get", [
sync analysis treats this use as a slot-indexed dynamic access; ptoas
does NOT rewrite the user expression into `iv mod N`.

This op is metadata-only (no data movement); the lowering merely
annotates the underlying memref view with the slot index for PlanMemory
/ sync / EnableBufferSelect to consume.
This op is metadata-only (no data movement). PlanMemory preserves it,
sync analysis consumes its slot index directly, and
PTOResolveBufferSelect replaces it with an addressed `pto.alloc_tile`.
}];

let arguments = (ins
Expand Down Expand Up @@ -560,7 +559,7 @@ def SetValidShapeOp : PTO_Op<"set_validshape", [
}];

let arguments = (ins
TileBufOrMemRef:$source,
TileBufType:$source,
Index:$valid_row,
Index:$valid_col
);
Expand All @@ -584,7 +583,7 @@ def GetValidShapeOp : PTO_Op<"get_validshape", [
}];

let arguments = (ins
TileBufOrMemRef:$source
TileBufType:$source
);

let results = (outs
Expand Down Expand Up @@ -1556,11 +1555,10 @@ def SlotMarkerOp : PTO_Op<"slot_marker", [
]> {
let summary = "Tag a memref view as referring to one slot of a multi_tile_buf";
let description = [{
Internal op materialized by `PTOViewToMemref` while lowering
`pto.multi_tile_get`. It carries the slot SSA index through the memref
layer so that PlanMemory, sync analysis (InsertSync / GraphSyncSolver),
and the buffer-select lowering pass can identify which physical slot
this memref reference touches.
Internal compatibility op for legacy memref-based multi-buffer IR. It
carries a slot SSA index so sync analysis (InsertSync / GraphSyncSolver)
and the buffer-select lowering pass can identify which physical slot a
memref reference touches.

The op is metadata-only (no data movement, no extra address arithmetic);
its result memref aliases the source memref byte-for-byte. Frontends do
Expand Down Expand Up @@ -2634,12 +2632,10 @@ def LocalArraySetOp : PTO_Op<"local_array_set"> {
}

def DeclareTileMemRefOp : PTO_Op<"declare_tile_memref"> {
let summary = "Internal memref placeholder for a tile whose address is assigned later";
let summary = "Legacy memref placeholder for a runtime-bound tile";
let description = [{
Internal lowering op used by PTOViewToMemref. This op does not allocate
storage; it only provides a memref-typed SSA handle so later passes can
attach tile metadata through pto.bind_tile before the address is filled by
pipe operations such as pto.tpop.
Compatibility op for legacy memref-based IR. New PTOViewToMemref pipelines
preserve `pto.declare_tile` directly instead of producing this op.
}];

let results = (outs AnyMemRef:$result);
Expand Down
13 changes: 6 additions & 7 deletions include/PTO/IR/VPTOOps.td
Original file line number Diff line number Diff line change
Expand Up @@ -220,11 +220,10 @@ def TensorViewAddrOp : PTO_Op<"tensor_view_addr", [Pure]> {
typed PTO pointer, depending on the requested destination type.

In authoring-form IR this op preserves the descriptor-style surface;
during view-to-memref lowering it collapses to the underlying memref value
or to a memref-derived pointer.
tile-native inputs remain unchanged through memory planning and sync.

This op may also accept a memref operand after earlier view lowering, in
which case it behaves as an identity marker and is removed by lowering.
This op may also accept a legacy memref operand, in which case it behaves
as an identity marker and is removed by compatibility lowering.
}];

let arguments = (ins AnyTypeOf<[TensorViewType, PartitionTensorViewType, AnyMemRef],
Expand All @@ -247,9 +246,9 @@ def TileBufAddrOp : PTO_Op<"tile_buf_addr", [Pure]> {
shape and address space.

This op is emitted by TileLang DSL templates and resolved by the
FoldTileBufIntrinsics pass after inlining. Hand-written `.pto` may also
use it directly on the memref result of `pto.bind_tile` / lowered
`pto.alloc_tile`.
FoldTileBufIntrinsics pass after inlining. Tile-native uses remain in PTO
IR through memory planning and sync. Hand-written legacy `.pto` may also
use it directly on the memref result of `pto.bind_tile`.
}];

let arguments = (ins AnyTypeOf<[TileBufType, AnyMemRef],
Expand Down
5 changes: 5 additions & 0 deletions include/PTO/Transforms/InsertSync/PTOIRTranslator.h
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,8 @@ class PTOIRTranslator {
// --- 内存/Alias 分析 ---
void UpdateKernelArgMemInfo();
LogicalResult UpdateAllocTileOpMemInfo(pto::AllocTileOp op);
LogicalResult UpdateAllocMultiTileOpMemInfo(pto::AllocMultiTileOp op);
LogicalResult UpdateDeclareTileOpMemInfo(pto::DeclareTileOp op);
LogicalResult UpdateDeclareGlobalOpMemInfo(pto::DeclareGlobalOp op);
LogicalResult UpdateDeclareTileMemRefOpMemInfo(pto::DeclareTileMemRefOp op);
LogicalResult UpdatePointerCastOpMemInfo(pto::PointerCastOp op);
Expand All @@ -81,6 +83,9 @@ class PTOIRTranslator {
void UpdateMemrefSubViewAliasBufferInfo(memref::SubViewOp op);
void UpdateTileSubViewAliasBufferInfo(pto::SubViewOp op);
void UpdateSlotMarkerAliasBufferInfo(pto::SlotMarkerOp op);
void UpdateMultiTileGetAliasBufferInfo(pto::MultiTileGetOp op);
void UpdateSlotSelectedAliasBufferInfo(Value result, Value source,
Value slot);

// --- 控制流处理 (SCF) ---
void UpdateForOpInfo(scf::ForOp forOp);
Expand Down
6 changes: 3 additions & 3 deletions include/PTO/Transforms/Passes.h
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,10 @@ std::unique_ptr<Pass> createConvertToPTOOpPass();
/// PTO Ops.
std::unique_ptr<Pass> createInferPTOMemScopePass();

/// Create a pass to plan memory.
std::unique_ptr<Pass>
createPlanMemoryPass(const PlanMemoryOptions &planMemoryOption = {});

createPlanMemoryPass(const PlanMemoryOptions &options = {});
std::unique_ptr<Pass>
createPlanMemoryModernPass(const PlanMemoryOptions &options);
std::unique_ptr<Pass> createPTORemoveRedundantBarrierPass();
std::unique_ptr<Pass> createPTOViewToMemrefPass();
std::unique_ptr<Pass> createPTOValidateIntToPtrUsesPass();
Expand Down
70 changes: 27 additions & 43 deletions include/PTO/Transforms/Passes.td
Original file line number Diff line number Diff line change
Expand Up @@ -193,37 +193,26 @@ def PTORematerializeFixpipeVectorQuant
}

def PlanMemory : Pass<"pto-plan-memory", "ModuleOp"> {
let summary = "Plan memory for PTO Ops";
let summary = "Plan memory for PTO Ops";
let constructor = "mlir::pto::createPlanMemoryPass()";
let dependentDialects = ["pto::PTODialect", ];

let options = [
Option<"memMode", "mem-plan-mode", "pto::MemPlanMode",
"pto::MemPlanMode::LOCAL_MEM_PLAN",
"plan mem mode (default is LOCAL_MEM_PLAN)",
[{::llvm::cl::values(
clEnumValN(pto::MemPlanMode::LOCAL_MEM_PLAN, "local-mem-plan",
"plan mem mode is for memref.alloc"),
clEnumValN(
pto::MemPlanMode::GLOBAL_WORKSPACE_PLAN,
"global-work-space-plan",
"plan mem mode is for memref_ext.alloc_workspace"))}]>,
Option<"enableGlobalReuse", "enable-global-workspace-reuse", "bool",
/*default=*/"false",
"Enable global workspace reuse ,default : false">,
Option<"enablePrintMemoryAllocatedSize", "enable-print-memory-allocated-size", "bool",
/*default=*/"false",
"Enable print memory allocated size, default : false">,
Option<"restrictInplaceAsISA", "restrict-inplace-as-isa", "bool",
/*default=*/"false",
"restrict memory inplace as isa, default : false">,
Option<"orderBySize", "order-by-size", "bool",
/*default=*/"false",
"Process buffers largest-first (first-fit-decreasing order) during "
"local memory planning instead of the default DMA-first order. "
"Decreasing-size order packs heterogeneous-size buffers tighter "
"(matches XLA/TVM/SOMAS). default : false">,
Option<"memMode", "mem-mode", "std::string", "\"local\"",
"Planning mode. Supported value today: local">,
Option<"orderBySize", "order-by-size", "bool", "false",
"Plan larger buffers first inside one AddressSpace before applying "
"SPEC_LEVEL_0 first-fit reuse">
];

let dependentDialects = [
"mlir::pto::PTODialect",
"mlir::memref::MemRefDialect",
"mlir::arith::ArithDialect",
"mlir::func::FuncDialect",
"mlir::scf::SCFDialect"
];
}

def PTOLoweringSyncToPipe : Pass<"pto-lowering-sync-to-pipe", "func::FuncOp"> {
let summary = "Lower high-level sync ops to low-level pipe ops";
let description = [{
Expand Down Expand Up @@ -752,25 +741,20 @@ def PTOMaterializeTileHandles : Pass<"pto-materialize-tile-handles", "ModuleOp">
}

def PTOResolveBufferSelect : Pass<"pto-resolve-buffer-select", "ModuleOp"> {
let summary = "Lower pto.slot_marker to single-slot pointer_cast (multi-buffer)";
let summary = "Resolve multi-buffer slot selections to addressed tile handles";
let description = [{
Consumes `pto.slot_marker %src[%k]` ops written by PTOViewToMemref to
thread multi-buffer slot selection through the memref layer.

The op is replaced by a fresh single-address `pto.pointer_cast` that
refers to the chosen physical slot:
Primarily consumes tile-native `pto.multi_tile_get %src[%k]` operations.
The source `pto.alloc_multi_tile` carries either planner-assigned physical
slot addresses or a level3 base address. Each selection is replaced by a
`pto.alloc_tile` carrying the selected address; constant slots select one
address directly, while dynamic slots use an N-way `arith.select` chain.

- constant slot `k`: a single `pto.pointer_cast(addrK)` is emitted at
the use site, using slot k from the underlying multi-address
pointer_cast created by PTOPlanMemory / AllocToPointerCast.
- dynamic slot `%k`: per-slot single-address `pto.pointer_cast`s are
created and an N-way `arith.select` chain picks one according to the
user-supplied SSA. This pass does NOT rewrite `%k` into `iv mod N`;
the frontend expression is the slot selector.
The pass also retains compatibility with legacy memref-based
`pto.slot_marker` IR, where the selection is materialized as a
single-address `pto.pointer_cast`.

The original multi-address `pto.pointer_cast` is kept in IR as the
"alloc anchor" so future sync passes can recognize the multi-buffer
geometry.
This pass does not rewrite `%k` into `iv mod N`; the frontend expression
remains the slot selector.
}];

let constructor = "mlir::pto::createPTOResolveBufferSelectPass()";
Expand Down
Loading
Loading