diff --git a/docs/designs/ptoas-largest-first-fit-four-gates-memplan-design.md b/docs/designs/ptoas-largest-first-fit-four-gates-memplan-design.md new file mode 100644 index 0000000000..3bd87e6b03 --- /dev/null +++ b/docs/designs/ptoas-largest-first-fit-four-gates-memplan-design.md @@ -0,0 +1,655 @@ +# PTOAS Largest-First-Fit 与四道冲突闸门内存规划设计 + +## 总体方案 + +PTOAS 当前保留两套 local memory planner： + +- legacy memplan：默认启用，保留旧版 SPEC_LEVEL_0/1/2 三级投机复用，内存不够时回滚的策略。 +- modern memplan：通过 `--plan-memory-impl=modern` 显式启用，当前实现Largest-First-Fit 与四道冲突闸门内存规划设计 + +PTOAS 当前已经具备 largest-first 风格的规划开关：`--plan-memory-order-by-size`。该选项会让 planner 在同一 AddressSpace 内优先处理更大的 buffer。当用户显式选择 `--plan-memory-impl=modern` 且未显式指定 `--plan-memory-order-by-size` 时，modern memplan 默认开启该排序；legacy memplan 仍保持默认关闭。 + +当前 `pto.alloc_tile` 的 lowering 路径保持 tile-native： + +```text +pto.alloc_tile(no addr) + -> PTOViewToMemref 透传，不转换成 memref.alloc + -> pto-plan-memory 收集为 local allocation root + -> modern/legacy memplan 按 level 校验并规划 local addr + -> 直接给 pto.alloc_tile 补常量 addr + -> 后续 pto.t* tile op 继续使用 !pto.tile_buf 形态 +``` + +用户显式写 `pto.alloc_tile addr` 时，level 语义仍由 memplan 校验：level1/level2 禁止显式 addr，level3 要求显式 local addr。也就是说，`pto.alloc_tile` 不再通过 `memref.alloc -> pto.pointer_cast -> pto.bind_tile` 这条中间链路表达地址；只有非 `alloc_tile` 的 memref/address root 仍可在 materialize 阶段使用 `pto.pointer_cast + pto.bind_tile` 表达规划后的地址与 tile metadata。 + +## 目标 + +本设计目标如下： + +- 复用 PTOAS 已有 `--plan-memory-order-by-size` largest-first 能力。 +- 将复用判定收敛为统一的 `canShare(a, b)` 谓词。 +- 引入统一的扁平冲突闸门模型，避免不安全复用。 +- 当前实现范围不包含 PyPTO 闸门 4；PTOAS 显式 multibuffer 已覆盖 ping-pong slot 分离语义。 +- 保持按 AddressSpace 独立规划。 +- 保持不回滚、不降级的 deterministic 策略。 +- 保持 `pto.alloc_tile(no addr)` tile-native 路径：memplan 直接补 `addr`。 +- 保持 legacy memplan 默认行为不变。 + +## 非目标 + +本设计不包含以下内容： + +- 不修改 legacy memplan 的 StorageEntry / SPEC_LEVEL_1 / SPEC_LEVEL_2 逻辑。 +- 不在 legacy memplan 中实现四道闸门。 +- 不恢复旧版回滚式投机规划。 +- 不把 modern memplan 作为默认实现。 +- 不在本阶段实现跨函数、跨 module 的全局内存规划。 + +## 核心思想 + +将每个 plannable local allocation root 抽象成一个待装箱 item。这里的 item 对应当前 modern memplan 中的 `RootInfo`，不是所有 local SSA value： + +`pto.bind_tile`、`pto.slot_marker`、`memref.subview`、`memref.cast` 等 alias/view value 不会成为新的 item，而是通过 `valueToRoots` 归属到已有 root。四道闸门需要的额外约束数据不放进 `RootInfo`，而是由 `ConflictFacts` 这类 side table 提供。 + + +## 算法流程 + +### 1. Root 收集 + +modern memplan 继续收集以下 local root： + +- `pto.alloc_tile(no addr)`。 +- plannable local `memref.alloc`。 +- 后续如有其它明确的 local address root，可扩展到同一 root 表。 + +不参与 local memplan 的 root： + +- GM space。 +- `AddressSpace::Zero`。 +- 已经由 level3 显式指定地址的 local `pto.alloc_tile addr`。 +- 非静态 shape 或无法计算 element byte size 的 root。 + +### 2. LifetimeInterval + +每个 root 生成一个 `LifetimeInterval`： + +```text + Value root; + Operation *defOp; + AddressSpace space; + uint64_t slotBytes; + uint64_t totalBytes; + uint64_t alignmentBytes; + uint64_t slotCount; + unsigned allocIndex; + unsigned freeIndex; + unsigned stableOrder; + SmallVector offsets; +``` + +现代 planner 中已有线性化 walk 和 root alias 传播，本设计在此基础上补齐： + +- loop-aware lifetime extension。 +- branch / yield / iter_arg alias family 信息。 +- per-root semantic metadata。 + +### 3. 按 AddressSpace 分桶 + +复用只允许发生在同一 AddressSpace 内： + +```text +Vec 只和 Vec 复用 +Mat 只和 Mat 复用 +Left/Right/Acc/Bias/Scaling 各自独立 +GM 不参与 local memplan +``` + +分桶是硬约束，在进入装箱前完成。 + +### 4. Largest-First-Fit 排序与装箱 + +PTOAS 已有 `--plan-memory-order-by-size` 对应的 largest-first 排序能力。打开该选项后，每个 AddressSpace 桶内排序： + +```text +sizeBytes 降序 +sizeBytes 相同则按 defIndex 升序 +defIndex 相同则按 stableOrder 升序 +``` + +这样先让大 buffer 更早参与规划，再把生命周期不冲突的小 buffer 放入可复用位置，避免 definition-order greedy 只能“小复用先出现的大”的单向问题。 + +将一个物理 local buffer 抽象成一个 bin： + +```text +ReuseGroup + representative root + members + slot size bytes + address space + chosen offset +``` + +因此，本设计不是重新新增一个独立的 largest-first 开关，而是要求四道闸门的 `canShare` 判定接入现有 `--plan-memory-order-by-size` 路径。 + +特点： + +- 不回滚。 +- 不降级。 +- 装进第一个可容纳 group 后立即停止。 +- 后续容量超限直接报错，不为了适配容量放松闸门。 + +## 四道禁止内存复用的闸门 + +四道闸门是扁平 AND 关系，任意一道闸门失败，两个 root 就不能进入同一个 `ReuseGroup`，也就不能复用同一个 base offset。 + +### 闸门 1：生命周期 + +**目的：** 禁止两个运行时可能同时存活的 local root 复用同一块物理地址。 + +基础规则： + +```text +如果 a 和 b 生命周期重叠，则不能复用。 +如果 a.lastUse == b.def 或 b.lastUse == a.def，则认为 touching，可继续检查其它闸门。 +``` + +PTOAS 现有 `lifetimesOverlap` 可扩展为： + +```text +overlap = !(a.lastUse <= b.def || b.lastUse <= a.def) +``` + +`<=` 表示 touching 不算生命周期冲突。 + +**适用场景 sample：普通直线代码中的临时 buffer 复用。** + +```text +%a = alloc vec[1024] +use(%a) // %a.lastUse + +%b = alloc vec[1024] // %b.def +use(%b) +``` + +如果 `%a.lastUse <= %b.def`，闸门 1 允许 `%a` 和 `%b` 进入后续闸门。只要其它闸门也通过，二者可以复用同一个 offset： + +```text +%a.offset = 0 +%b.offset = 0 +``` + +反例： + +```text +%a = alloc vec[1024] +%b = alloc vec[1024] +use(%a) +use(%b) +``` + +此时 `%a` 和 `%b` 的生命周期重叠，闸门 1 直接失败，必须分配不同 offset。 + +### 闸门 2：phi family + +**目的：** 处理互斥控制流分支，避免过度保守的 liveness extension 阻止本来安全的复用。 + +场景： + +```text +%r = scf.if %cond -> tile { + scf.yield %a +} else { + scf.yield %b +} +``` + +`%a` 和 `%b` 在程序序生命周期上可能被扩展到 `%r` 的最后使用点，看起来重叠；但运行时两个分支互斥，不会同时存在。因此同一 phi family 的 yield source 可以豁免闸门 1 的生命周期冲突。 + +需要收集： + +```text +phiFamilyIds[root] = set +``` + +规则： + +- 同 family 且无其它非 family 真重叠成员：允许共享。 +- 如果混入外部 live root 或非 family 真重叠 root：不豁免。 + +实现建议： + +```text +Gate2_LifetimeAndPhi(a, b): + if !lifetimeOverlap(a, b): + return true + + if samePhiFamily(a, b): + return true + + return false +``` + +因此闸门 2 在实现上可以嵌入闸门 1，也可以保留独立函数名但由闸门 1 调用。 + +**适用场景 sample：if/else 分支内 local root 互斥复用。** + +```text +%r = scf.if %cond -> tile { + %then_buf = alloc vec[1024] + produce(%then_buf) + scf.yield %then_buf +} else { + %else_buf = alloc vec[1024] + produce(%else_buf) + scf.yield %else_buf +} + +consume(%r) +``` + +如果简单地把 `%then_buf` 和 `%else_buf` 的生命周期都延伸到 `consume(%r)`，二者看起来重叠，闸门 1 会失败。但运行时 then/else 互斥，不会同时分配这两个 branch-local root。因此它们属于同一个 phi family 时，可以共享同一 offset： + +```text +%then_buf.offset = 0 +%else_buf.offset = 0 +``` + +反例： + +```text +%outer = alloc vec[1024] + +%r = scf.if %cond -> tile { + %then_buf = alloc vec[1024] + use(%outer) + scf.yield %then_buf +} else { + %else_buf = alloc vec[1024] + use(%outer) + scf.yield %else_buf +} +``` + +`%then_buf` 和 `%else_buf` 之间可以因为 phi family 互斥而复用；但它们不能借这个豁免和 `%outer` 这种外部 live root 错误复用。 + + +### 闸门 3：target-specific load/tpop hazard + +**目的：** 表达特定 target 上的局部硬件/后端 hazard。它不是通用生命周期问题，而是某些 load-derived buffer 与 consumes-tpop writer 在 touching 点复用会触发目标相关错误。 + +参考 PyPTO 的 Ascend910B split-AIV hazard。PTOAS 中设计为 target-gated 闸门： + +```text +如果目标架构/后端不存在该 hazard，则闸门恒通过。 +如果存在该 hazard，则禁止 load-derived buffer 与 consumes-split-tpop 的 writer 在 touching 点复用。 +``` + +当前 PTOAS modern memplan 的启用条件： +A5 上该闸门恒通过。A3 上也只有识别到 split tpop 派生值时才可能触发。 + +#### 设计原理 + +这道闸门保护的是一种普通静态生命周期分析看不到的 target-specific touching 复用风险。普通 memplan 只看 allocation root 的 def/use 区间：如果一个 load-derived buffer 的最后一次使用正好等于另一个 writer output 的写入点，生命周期上属于 touching，通常可以复用同一个 offset。但在特定 target 上，`tload/tprefetch` 产生的 load-derived buffer 与 consuming split-tpop 的 writer 可能在后端流水或指令调度中存在更细粒度的运行时 overlap；如果二者复用同一地址，writer 的写入可能破坏 load-derived input 在该 target 上仍然需要保持稳定的内容。 + +因此该闸门不把问题抽象成通用“生命周期重叠”，而是显式识别一类 target hazard： + +```text +load-derived input 的 last use +touching +consumes split-tpop 的 writer output 写入 +``` + +只有当目标架构确认存在该 hazard 时才禁止复用。当前 PTOAS 中 A3 开启该闸门，A5 恒通过，避免把 A3 的局部硬件/后端约束错误扩散成所有 target 的通用内存规划规则。 + +实现上使用 writer op index，而不是 allocation root 的 def index，原因是 hazard 发生在“某个 writer op 同时读取 load-derived input 和 split-tpop-derived value，并写出 DPS output”的这个操作点。allocation root 的 def 只表示 buffer 被声明出来，不一定等于真正发生写入和触发 target hazard 的位置；用 writer op index 可以把 touching 条件精确绑定到产生风险的 op 上。 + +相关事实拆开收集： + +```text +loadDerivedRoots: + 标记来自 tload/tprefetch 的 DPS dst root + +split-tpop-derived values: + 标记 split tpop 结果、split tpop tile operand 以及它们经过 alias/view op 派生出的 value + +tpopConsumerRoots: + 标记同时读取 load-derived root 和 split-tpop-derived value 的 DPS writer output root + +tpopConsumerWriteIndices: + 记录上述 writer op 的 op index，用于判断是否正好在 touching 点复用 +``` + +这种拆分可以避免过度保守：不是所有 load buffer 都禁止复用，也不是所有 tpop 相关 buffer 都禁止复用，只有“load-derived input 的最后使用点”碰到“consumes split-tpop writer 的写入点”时才失败。 + +当前 PTOAS modern memplan 对 DPS output root 使用 writer-def liveness：如果某个 DPS output 是 pure overwrite，即该 output 没有被 memory effects 标记为 Read，则它的 `allocIndex` 会从 `pto.alloc_tile` / `memref.alloc` 收缩到第一次有效 writer op。这样 load-derived input 的 `freeIndex` 与 writer output 的 `allocIndex` 可以形成 touching，是否允许复用继续由 target-specific load/tpop hazard 和 op semantic no-alias 闸门判断。若 output 需要旧值，例如 read-modify-write / accumulate 语义，则保持 allocation-start 的保守生命周期。 + +需要收集： + +```text +loadDerivedRoots +tpopConsumerRoots +tpopConsumerWriteIndices +targetHazardEnabled +``` + +判定： + +```text +input.root in loadDerivedRoots +&& writer.root in tpopConsumerRoots +&& input.lastUseIndex in tpopConsumerWriteIndices[writer.root] +``` + +双向检查： + +```text +hazard(a, b) || hazard(b, a) +``` + +**适用场景 sample：target-gated touching 复用禁止。** + +```text +%load_buf = alloc vec[1024] +load_or_tpop_producer(%load_buf) +last_use(%load_buf) + +%writer_dst = alloc vec[1024] +writer_consumes_tpop(...) outs(%writer_dst) +``` + +从普通生命周期看，`%load_buf.lastUse == %writer_dst.def`，属于 touching，可以复用。但如果 target 标记了： + +```text +%load_buf in loadDerivedRoots +%writer_dst in tpopConsumerRoots +tpopConsumerWriteIndices[%writer_dst] contains writer op index +targetHazardEnabled = true +``` + +则闸门 3 失败，二者不能复用。若目标没有该 hazard，闸门 3 恒通过，不影响复用。 + +当前实现的 fact 来源： + +```text +loadDerivedRoots: + pto.tload / pto.tprefetch 的 DPS dst root + +split tpop derived value: + A3 上 split != 0 的 pto.tpop_from_aic result + A3 上 split != 0 的 pto.tpop tile operand + 以及从这些 value 经过 bind_tile / slot_marker / cast / subview / select 等 alias/view op 派生出的 value + +tpopConsumerRoots: + 某个 DPS writer 同时读取 split tpop derived value 和 load-derived root 时， + 记录该 writer 的 DPS output root +``` + + +### 闸门 4：op semantic no-alias + +**目的：** 表达“从生命周期看可以复用，但从 op 语义看不能 alias”的约束。闸门 4 对应 PyPTO 的 inplace 机制：`not_inplace_safe()` 与 `forbid_output_alias(i)`，并覆盖 PTOAS legacy memplan 中已有的 scratch-output conflict。 + +在 modern memplan 中，闸门 4 不应再只叫笼统的 `semantic conflict`，而应建模为一张明确的 forbid-alias side table： + +```text +forbidAlias[root] = forbiddenRootSet +``` + +这里的 `root` 是 plannable local allocation root，而不是任意 SSA value。收集时需要先通过 `valueToRoots` 把 operand、view、alias value 归约到 root，再记录 root 与 root 之间的禁止复用关系。 + +#### 4.1 scratch-output conflict + +PTOAS legacy memplan 的 semantic conflict 主要就是 scratch-output conflict。modern memplan 应继续覆盖这类场景： + +```text +op implements PTO_DpsInitOpInterface +dpsInits = op outs(...) + +effects = MemoryEffectOpInterface::getEffects(op) +scratchOperands = + Write operand + && operand in op operands + && operand not in dpsInits + +for scratch in scratchOperands: + for dst in dpsInits: + forbidAlias[root(scratch)].insert(root(dst)) + forbidAlias[root(dst)].insert(root(scratch)) +``` + +**适用场景 sample：`tmp` scratch 不能和 output 复用。** + +```text +%src = alloc vec[1024] +%tmp = alloc vec[1024] +%dst = alloc vec[1024] + +pto.ttrans ins(%src, %tmp) outs(%dst) +``` + +`%tmp` 是 op 执行过程中的 scratch workspace，`%dst` 是最终 output。即使未来 liveness 把 `%dst` 看作在 op 上定义、从而让 `%tmp.lastUse == %dst.def` 成为 touching，二者也不能复用，否则 scratch 写入可能覆盖 output。闸门 3 记录： + +```text +forbidAlias[%tmp].insert(%dst) +forbidAlias[%dst].insert(%tmp) +``` + +#### 4.2 not_inplace_safe + +PyPTO 中 `not_inplace_safe()` 表示该 op 不能做 `src == dst` 的 inplace 执行。映射到 PTOAS 时，规则是： + +```text +if opPolicy.notInplaceSafe: + for operand in op operands excluding dpsInits: + for dst in dpsInits: + forbidAlias[root(operand)].insert(root(dst)) + forbidAlias[root(dst)].insert(root(operand)) +``` + +典型 op 包括： + +```text +pto.ttrans +pto.tgather +pto.tands / pto.tors / pto.txors +pto.tfillpad_expand +pto.tfmod / pto.tfmods +pto.trecip / pto.trsqrt +pto.trowmax / pto.trowmin / pto.trowsum / pto.trowprod +pto.trowargmax / pto.trowargmin +pto.tcolargmax / pto.tcolargmin +pto.tsort32 / pto.tmrgsort +``` + +其中 `pto.tands` / `pto.tors` / `pto.txors` 和 `pto.tfillpad_expand` 是 PTOAS 侧额外保守标记的 non-inplace-safe op。它们虽然不是 scratch-output conflict，但后端/ISA 语义没有明确承诺 input/output alias 安全，memplan 不应通过地址复用隐式把它们变成 inplace 执行。 + +**适用场景 sample：算法本身不支持 input/output alias。** + +```text +%x = alloc vec[1024] +%y = alloc vec[1024] + +pto.tfmod ins(%x, %rhs) outs(%y) +``` + +`tfmod` 的实现可能会在计算中间覆盖某个源值，但后续仍需要原始源值。因此 `%y` 不能复用 `%x` 的物理地址。即使生命周期或 touching 规则允许，也必须由闸门 3 禁止： + +```text +forbidAlias[%x].insert(%y) +``` + +#### 4.3 forbid_output_alias(i) + +PyPTO 中 `forbid_output_alias(i)` 表示 op 整体可以对某些 value operand 做 inplace，但 output 不能 alias 第 `i` 个特定 operand。PTOAS 中需要按 PTO IR 的 DPS operand 布局映射到具体 operand。 + +典型场景： + +```text +pto.tsel: + forbid mask + forbid tmp + +pto.trowexpand / pto.tcolexpand: + forbid broadcast source + +pto.trowexpand* / pto.tcolexpand*: + forbid row/column vector operand +``` + +**适用场景 sample：broadcast vector 不能被 output 覆盖。** + +```text +%row = alloc vec[16x1] +%dst = alloc vec[16x64] + +pto.trowexpand ins(%row) outs(%dst) +``` + +`%row` 会被重复读取并广播到 `%dst` 的多个位置。如果 `%dst` 复用 `%row` 的地址，写 output 的过程中可能覆盖后续仍要读取的 broadcast source。因此闸门 3 记录： + +```text +forbidAlias[%row].insert(%dst) +``` + +**适用场景 sample：select 的 mask/tmp 不能和 output alias。** + +```text +%mask = alloc vec[1024] +%tmp = alloc vec[1024] +%dst = alloc vec[1024] + +pto.tsel ins(%mask, %lhs, %rhs, %tmp) outs(%dst) +``` + +`%lhs` / `%rhs` 是否允许和 `%dst` inplace 取决于 op 语义；但 `%mask` 和 `%tmp` 是被 op 读取或作为 scratch 使用的特殊 operand，不能被 `%dst` 覆盖。闸门 3 记录： + +```text +forbidAlias[%mask].insert(%dst) +forbidAlias[%tmp].insert(%dst) +``` + +#### 4.4 闸门 4 判定 + +`canShare(a, b)` 中的闸门 4 是双向检查： + +```text +Gate4_OpSemanticNoAlias(a, b): + return !forbidAlias[a].contains(b) + && !forbidAlias[b].contains(a) +``` + +由于 PTOAS 中 `pto.bind_tile`、`pto.slot_marker`、`memref.subview` 等 view/alias value 不一定是 root，收集 forbid-alias 时必须先做 root 归约： + +```text +for value in op operands / dpsInits: + roots = valueToRoots[value] +``` + +如果一个 value 通过 alias closure 对应多个 root，则需要记录所有 root 组合。这样后续 `ReuseGroup` 只需要比较 root 与 root，不需要在装箱阶段重新理解每个 op 的 operand 语义。 + +## PTOAS 数据结构建议 + +### RootInfo + +```cpp +struct RootInfo { + Value root; + Operation *defOp = nullptr; + AddressSpace space = AddressSpace::Zero; + uint64_t slotBytes = 0; + uint64_t totalBytes = 0; + uint64_t alignmentBytes = 1; + uint64_t slotCount = 1; + unsigned allocIndex = 0; + unsigned freeIndex = 0; + unsigned stableOrder = 0; + SmallVector offsets; +}; +``` + +本设计保持当前 modern memplan 的 `RootInfo` 字段不变。`RootInfo` 只表达 local allocation root 的基础事实： + +```text +root identity +definition op +address space +slot size / total size / slot count +lifetime interval +stable order +planned offsets +``` + +四道闸门所需的额外事实不直接塞进 `RootInfo`，而是放在 `ConflictFacts` 这类 side table 中。这样可以避免 root 结构随着每个闸门膨胀，也便于后续按阶段启用或删除某个闸门。 + +### ReuseGroup + +```cpp +struct ReuseGroup { + Value representative; + SmallVector memberIndices; + AddressSpace space; + uint64_t slotSizeBytes; + uint64_t offsetBytes; +}; +``` + +### ConflictFacts + +```cpp +struct ConflictFacts { + DenseMap> forbidAlias; + DenseSet loadDerivedRoots; + DenseSet tpopConsumerRoots; + DenseMap> phiFamilyIds; + + // Reserved for future implicit pipeline lowering support. Not used by the + // current PTOAS design because explicit multibuffer owns ping-pong slot + // separation. + // DenseMap> pipelineMembership; + // DenseSet pipelineLoadRoots; +}; +``` + +### 说明：pipeline stage load conflict（预留，不纳入当前实现） + +- 当前不实现 PyPTO 闸门 4。 +- 不新增 pipeline stage load 复用负例。 +- 保留设计占位：若未来 PTOAS 引入隐式 pipeline lowering，并能稳定提供 `pipelineMembership[root] = (group, stage)`，再接入该闸门。 +- 显式 `pto.alloc_multi_tile count=N` 继续由 multibuffer slot 分配保证 ping-pong 正确性。 + +## 测试计划 + +### lit 测试 + +新增或扩展以下测试： + +- `plan_memory_order_by_size_*.pto` 继续作为 largest-first 覆盖。 +- `plan_memory_five_gates_lifetime_touching.pto` +- `plan_memory_five_gates_phi_family.pto` +- `plan_memory_five_gates_semantic_no_alias.pto` +- `plan_memory_five_gates_target_hazard.pto` +- 暂不新增 `plan_memory_five_gates_pipeline_load.pto`；闸门 4 当前为预留设计。 + +已有 `plan_memory_*.pto` 应继续保留 legacy + modern 双 RUN。 + +### 验证命令 + +```bash +cmake --build build --target ptoas -j8 + +PATH=/Users/fangrui/workspace/huawei/llvm21-workspace/llvm-project/llvm/build-assert/bin:$PATH \ + /Users/fangrui/workspace/huawei/llvm21-workspace/llvm-project/llvm/build-assert/bin/llvm-lit \ + -sv build/test/lit \ + --filter 'plan_memory' + +ctest --test-dir build --output-on-failure -L PTODSL +``` + +## 风险与注意事项 + +- `--plan-memory-order-by-size` 本身会改变 modern memplan 的 offset 分配顺序，测试应避免把 order-sensitive 预期错误地复用于默认路径。 +- 四道闸门是硬约束，不应为了容量不足而放松。 +- target hazard 需要先确认 PTOAS IR 中稳定的标记来源。 +- pipeline metadata 闸门当前不实现；如果未来启用，需要先定义稳定的 pipeline membership 来源。 +- phi family 豁免必须保守，不能让外部 live alias 借互斥分支错误复用。 +- legacy memplan 不应受该设计影响，默认行为保持不变。 diff --git a/docs/designs/ptoas-tile-native-mainline-op-migration.md b/docs/designs/ptoas-tile-native-mainline-op-migration.md new file mode 100644 index 0000000000..b66a7f5c68 --- /dev/null +++ b/docs/designs/ptoas-tile-native-mainline-op-migration.md @@ -0,0 +1,377 @@ +# PTOAS Tile-Native Mainline 按 Op 迁移设计 + +## 1. 背景 + +PTOAS 当前主 pipeline 在内存规划前后包含两个 ModuleOp pass： + +```text +PTOViewToMemref + -> implicit tmp materialization + -> legacy/modern memplan + -> InsertSync/GSS/BufidSync + -> PTOResolveBufferSelect + -> PTOMaterializeTileHandles + -> EmitC/VPTO +``` + +`PTOViewToMemref` 把函数 ABI、PTO view、部分 local tile 和计算 op 转成 +memref 形态；`PTOMaterializeTileHandles` 在规划和同步完成后重新构造 +`!pto.tile_buf` handle。普通 `pto.alloc_tile` 已经是例外：它保持 +tile-native，由 memplan 直接填写 `addr`。 + +本设计的目标是逐个 op 消除上述往返转换，最终删除： + +- `lib/PTO/Transforms/PTOViewToMemref.cpp` +- `lib/PTO/Transforms/PTOMaterializeTileHandles.cpp` +- `pto-view-to-memref` +- `pto-materialize-tile-handles` + +迁移过程中不增加用户可见的双 pipeline 开关。每个 op 在现有 pipeline 中 +先补齐 tile-native 分析和 lowering；全部 op 完成后一次性删除两个 pass。 + +## 2. 目标 IR + +```text +PTO tile/view IR + -> implicit tmp materialization（产生 pto.alloc_tile(no addr)） + -> memplan（直接填写 alloc_tile/alloc_multi_tile 地址） + -> sync（直接分析 PTO root、view、slot 和物理地址） + -> EmitC/VPTO（直接消费 PTO tile/view IR） +``` + +基本原则： + +1. local allocation root 使用 PTO allocation op 表达，不中转 `memref.alloc`。 +2. tile view 保持 `!pto.tile_buf`，view op 自身携带 shape/layout/offset 语义。 +3. GM tensor view 保持 PTO view op，直到 EmitC/VPTO backend lowering。 +4. memplan、InsertSync 和 backend 各只有一套算法主体，通过统一的 root/alias/address + 查询处理不同 PTO op。 +5. `pto.bind_tile`、`pto.declare_tile_memref`、`pto.slot_marker` 仅在完成对应 + tile-native 迁移后删除。 + +## 3. 每个 Op 的完成标准 + +每个 op 只有同时满足以下条件才算迁移完成： + +- verifier 和 parse/print 能表达目标 tile-native 语义； +- legacy 和 modern memplan 能识别其 root/alias/size/address； +- InsertSync、Graph Sync Solver 和 BufidSync 能恢复其物理内存访问； +- EmitC 和 VPTO 都能直接 lowering； +- level1/level2/level3 行为明确； +- A2/A3 和 A5 行为明确； +- 至少有一个正例和必要的 verifier/level 负例； +- 测试不再检查 `bind_tile`、`memref.subview` 等旧中间形态。 + +## 4. Allocation 与内部 Handle Op + +| Op | 当前行为 | Tile-native 目标 | Memplan | InsertSync | Backend 与测试 | +|---|---|---|---|---|---| +| `pto.alloc_tile` | `PTOViewToMemref` 已透传；memplan 填写 `addr` | 保持现状，作为单 slot local root | legacy/modern 均继续收集无地址 root；level3 校验显式地址 | 现有 InsertSync 已读取类型、大小和常量地址；GSS 需从 `alloc_tile addr` 构造 `PointerLikeInfo` | EmitC/VPTO 已有直接 lowering；保留 level1/2 自动地址和 level3 显式地址测试 | +| `pto.alloc_multi_tile` | 已保持 tile-native；memplan 写入内部 N-address 属性，level3 保留用户 base | op 本身作为 N-slot root，最终可由 backend 直接消费 | legacy/modern 已支持 `slotCount=N`、`slotBytes`、`totalBytes` 和每 slot offsets；禁止 sibling slot 互相 alias | InsertSync/GSS 已直接识别 multi root，不依赖 `slot_marker -> pointer_cast` | 当前由 `PTOResolveBufferSelect` 展开成带地址的 `alloc_tile`；已覆盖 constant/dynamic slot、loop/non-loop 测试 | +| `pto.multi_tile_get` | 已保持到 sync 和 `PTOResolveBufferSelect` | 最终继续保持到 backend | result 已 alias 到 multi root，并携带 slot expression | constant slot 只访问一个地址；dynamic slot 保守访问全部地址，并保留 dyn event-id 推导 | 当前 resolve pass 根据 slot 生成带单地址的 `alloc_tile`；不再生成 `slot_marker` | +| `pto.declare_tile` | 已保持 tile-native，不参与静态 local allocation | 保持现状，表示地址由 `tpop/tassign` 等运行时操作绑定 | legacy/modern 不收集为 allocation root | InsertSync 直接注册 symbolic root；GSS 按 declared SSA identity 建模 | EmitC 已直接声明 Tile；已覆盖 tpush/tpop 和 interleaved sync/GSS | +| `pto.declare_tile_memref` | 仅保留 legacy memref IR 兼容，不再由 `PTOViewToMemref` 生成 | 其余旧兼容用户清除后删除 | legacy 仅跳过，不分配 | InsertSync 暂保兼容入口 | 暂保 legacy EmitC pattern，最终随兼容路径删除 | +| `pto.bind_tile` | memref 与 tile metadata 的桥接和 alias anchor | local tile 不再需要；只允许迁移期兼容，最终删除或限制为外部 ABI bridge | 先统一 `getAliasSource()`，切换后删除 bind 分支 | 先统一 alias tracing，切换后删除 bind 分支 | 删除 materialize/EmitC bind lowering 测试 | +| `pto.pointer_cast` | 表达 memref root 的物理地址或 multi-address root | 仅保留真正的 raw-address ABI bridge；普通 local allocation 改由 allocation op 持有地址 | 不再作为 `alloc_tile` 的 materialization 结果 | 若保留，sync 继续读取其地址；multi-buffer 不再必须依赖它 | 后端保留 raw-address lowering，删除 alloc fallback 用法 | +| `pto.slot_marker` | `multi_tile_get` 在 memref 层的内部 slot 标签 | `multi_tile_get` 直接承担 slot 标签语义，最终删除 | 删除 alias 特判 | 将 slot narrowing 和 dyn event-id 逻辑迁移到 `multi_tile_get` | 删除 `PTOResolveBufferSelect` 对 slot_marker 的入口 | +| `pto.materialize_tile` | 用于显式恢复 tile handle 的兼容 op | 若无独立用户则随 materialize pass 一并删除 | 无 | 无 | 搜索并删除残余 pattern/test | + +## 5. Tile View 与 Metadata Op + +| Op | 当前行为 | Tile-native 目标 | Memplan / Alias | InsertSync | Backend 与测试 | +|---|---|---|---|---|---| +| `pto.subview` | 已在 memplan 和 sync 主线保持 tile-native，不再生成 `memref.subview + pto.bind_tile` | 同步完成前保留 `pto.subview` | result 与 source 同 root；按 layout/stride/offset 计算 byte range | InsertSync 已精确计算地址范围；GSS 通过通用 view alias 回溯 source | `PTOResolveBufferSelect` 在同步后解析为带偏移地址的 `alloc_tile`，供 EmitC/VPTO 共用；覆盖 row/col-major、boxed、dynamic offset 和 valid shape | +| `pto.treshape` | 已保持 tile-native，不再生成 memref view 或 `bind_tile` | 保持原 op | result 与 source 完全 alias，size 不变；legacy/modern memplan 已传播 alias | InsertSync/GSS 已直接传播 result-to-source alias | EmitC 已直接 lowering；覆盖 tile 参数、动态 valid shape 和静态 valid shape | +| `pto.bitcast` | 已保持 tile-native，不再生成 memref view 或 `bind_tile` | 保持原 op | result 与 source alias；legacy/modern memplan 已按 byte range 传播同一 root，并校验总容量一致 | InsertSync/GSS 已直接传播 alias，不按 element type 分裂 root | EmitC 已直接 lowering；覆盖 tile 参数和同容量 dtype 改变 | +| `pto.set_validshape` | 已保持 tile-native，输入已收紧为 `TileBufType` | 直接更新 tile handle metadata，不改变物理 root | 不产生新 root；物理 size 使用 allocation shape，不使用 valid shape | 对原 root 建模为 metadata Write | EmitC 已直接消费；覆盖 if 和动态 valid shape | +| `pto.get_validshape` | 已保持 tile-native，输入已收紧为 `TileBufType` | 直接读取 tile operand metadata | 不产生 root | 对原 root 建模为 metadata Read | EmitC 已直接 lowering；覆盖 tile 参数和动态值 | +| `pto.tile_buf_addr` | tile-native 输入已保持原 op；仅 legacy memref 输入仍走线性 memref 兼容路径 | 直接从 tile root/view 计算地址；返回 pointer-like PTO 类型 | 不产生新 root；legacy memplan 将 source 记录为 use | 包含该 op 的函数保持 tile ABI，地址结果保留 source provenance | EmitC 直接生成 `tile.data()`，VPTO pointer normalize 保持 typed pointer；覆盖 tile 参数和 alloc root | + +## 6. GM Tensor View 与地址 Op + +| Op | 当前行为 | Tile-native 目标 | Memplan / Sync | Backend 与测试 | +|---|---|---|---|---| +| `pto.make_tensor_view` | 已保持 PTO 形态穿过 `PTOViewToMemref`、memplan 和 sync | 最终保持到 backend | 不参与 local memplan；InsertSync 已把 result alias 到 pointer source | 当前在 `PTOResolveBufferSelect` 与完整 GM view 链一起转成 `memref.reinterpret_cast`，复用成熟 backend lowering | +| `pto.partition_view` | 已保持 PTO 形态穿过 memplan 和 sync | 最终保持到 backend | 不参与 local memplan；sync result alias source，并根据 offset/size 缩小 GM range | memref-backed source 在 `PTOResolveBufferSelect` 转成 `memref.subview`；`declare_global` 等运行时 source 保持 PTO op 走已有直接 EmitC lowering | +| `pto.get_tensor_view_dim` | 已保持 PTO 形态穿过 memplan 和 sync | 直接读取 PTO view shape | 不影响 memplan/sync | 当前在 `PTOResolveBufferSelect` 随所属 view 转成 `memref.dim` | +| `pto.get_tensor_view_stride` | 已保持 PTO 形态穿过 memplan 和 sync | 直接读取 PTO view stride | 不影响 memplan/sync | 当前在 `PTOResolveBufferSelect` 随所属 view 转成 strided memref metadata | +| `pto.inttoptr` | 结果改成 GM memref，并限制用途 | 保持 PTO pointer-like value | 不参与 local memplan；sync 将其视作 GM provenance | 保留 restricted-use verifier；EmitC/VPTO 直接 lowering | +| `pto.ptrtoint` | 折叠 `addptr` 链并生成 byte offset | 保持 PTO op，或迁移到独立 address-canonicalization pass | 不影响 local memplan；不能丢失 GM provenance | backend 生成整数地址；覆盖 ptr、addptr、view base | +| `pto.addptr` | 折叠进 tensor view、scalar load/store 或 pipe init | 保持到独立 address canonicalization/backend | sync 需要把 result alias 到 base，并记录 offset | EmitC/VPTO 直接 lowering；保留非法 escape 校验 | +| `pto.castptr` | 作为 pointer/memref 适配 op | 保持 PTO pointer cast | result alias source，地址空间变化必须校验 | 两后端直接 lowering | +| `pto.load_scalar` | `addptr` offset 被折叠进 op | op 直接接受 base+offset，或 backend 统一折叠 | InsertSync 记录 GM read；不参与 local memplan | 覆盖 inttoptr/addptr 和动态 offset | +| `pto.store_scalar` | `addptr` offset 被折叠进 op | op 直接接受 base+offset，或 backend 统一折叠 | InsertSync 记录 GM write | 覆盖跨 pipe flush/sync | +| `pto.initialize_l2g2l_pipe` | `gm_addr` 上的 addptr 被提前折叠 | 迁移到独立 address canonicalization 或 op verifier/lowering | sync 保留 GM base provenance | EmitC/VPTO 覆盖动态地址 | + +## 7. Control Flow 与函数 ABI + +| Op | 当前行为 | Tile-native 目标 | 需要改动 | +|---|---|---|---| +| `func.func` | tile 参数/结果可能改成 memref，入口插入 `bind_tile` | authored tile ABI 保持不变 | backend helper、外部声明和 kernel ABI 明确区分；memplan 仍按函数独立规划 | +| `func.call` | 依赖 ViewToMemref 的签名桥接，materialize 后恢复 helper tile ABI | caller/callee 直接使用一致的 tile/pointer ABI | call verifier、inline helper 和 InsertSync call effects 需要统一；跨函数不做 local allocation 合并 | +| `scf.if` | pass 重建 result type 和两个 `scf.yield` | tile result 原样穿过分支 | memplan 和 sync 合并两个分支的 root family；backend 支持 tile result | +| `scf.for` | pass 重建 iter_arg/result/yield 类型 | tile loop-carried value 原样保持 | memplan 扩展 loop-carried root 生命周期；sync 建立 iter_arg/yield alias 和 back-edge dependency | +| `scf.yield` | 随外层控制流重建 operand type | 直接 yield tile value | 不能把 yield 当新 root；传播到外层 result/下一轮 iter_arg | +| `pto.fusion_region` | 协调 region result 与 `pto.yield` 类型 | 保持 tile-native region ABI | FusionAnalysis、memplan、sync 和 backend 共享同一 root family | +| `pto.yield` | materialize 阶段恢复 tile operand | 直接 yield tile value | 与 `scf.yield` 相同，传播 alias/liveness | + +## 8. 特殊后端/同步 Op + +| Op | 当前依赖 | 删除两个 pass 后的工作 | +|---|---|---| +| `pto.vlds` | materialize pass 为 view operand 恢复 tile address | backend 和 sync 直接从 tile/view root 计算地址；保持 Read/Write effects | +| `pto.vsts` | 同上 | 直接处理 tile view 地址和 post-update 结果 | +| `pto.vsstb` | 同上 | 直接处理 tile view 地址、packed stride 和 post-update result | +| `pto.mgather` | ViewToMemref 会在无 tile root 的旧路径重建 op | 保持 tile operands；sync macro model 直接读取 index/src/dst/scratch 的 effects | +| `pto.mscatter` | 同上 | 保持 tile operands；直接计算 GM/local ranges 和 coalesce 语义 | +| `pto.tassign` | ViewToMemref 协调 result type，materialize 用它恢复地址 | 明确为 tile handle 地址重绑定；result 类型始终等于 tile operand 类型 | memplan 不把 result 当新 allocation；sync 将其 alias 到 tile root并更新地址 provenance；backend 直接 lowering | + +## 9. Compute Op 逐项处理 + +以下 op 在 `PTOViewToMemref` 的 Stage 3 中主要为了适配 memref operand 而被 +重新创建。tile-native 主路径中不应重建它们。每个 op 的共同要求是： + +- 保持 authored `!pto.tile_buf` operand/result； +- memplan 只通过 MemoryEffects、DPS output 和 alias side table 使用它，不改 op； +- InsertSync 直接读取 MemoryEffects；tmp/scratch 必须保持正确的 Read/Write； +- EmitC/VPTO 的现有 tile lowering 必须在没有 materialize pass 的情况下通过； +- 每个 op 至少保留一个 end-to-end lit 或 sample。 + +| Op | 额外检查 | +|---|---| +| `pto.tload` | GM view 到 local tile 的 MTE2 write；dst 是 writer output，地址来自 `alloc_tile`/subview | +| `pto.tstore` | local tile 到 GM view 的 MTE3 read；检查 reused local addr 与后续 writer 的同步 | +| `pto.ttrans` | optional tmp 的条件 materialization、A5 placeholder 和 scratch conflict | +| `pto.texp` | precision attr 原样保留 | +| `pto.tmul` | DPS inplace 规则和 input/output alias | +| `pto.tmuls` | scalar operand 顺序和 precision attr | +| `pto.tadd` | DPS inplace 规则 | +| `pto.taddc` | carry/result operand 顺序和多输出语义 | +| `pto.tadds` | scalar operand和 valid-shape 校验 | +| `pto.taddsc` | scalar/carry 组合和多输出语义 | +| `pto.tmatmul` | LEFT/RIGHT/ACC 地址空间、tile config 和 role 不再依赖 materialize 推断 | +| `pto.tmatmul_acc` | ACC init/output 的 Read+Write 和 inplace 语义 | +| `pto.tmatmul_bias` | bias root/address space 和 scratch/output conflict | +| `pto.tmatmul_mx` | MX scale tile role 和低精度 packed shape | +| `pto.tmatmul_mx_acc` | MX scale + ACC init/output | +| `pto.tmatmul_mx_bias` | MX scale + bias root | +| `pto.tgemv` | vector/matrix role 和 dst writer lifetime | +| `pto.tgemv_acc` | accumulator init/output effects | +| `pto.tgemv_bias` | bias tile role | +| `pto.tgemv_mx` | MX scale role | +| `pto.tgemv_mx_acc` | MX scale + accumulator effects | +| `pto.tgemv_mx_bias` | MX scale + bias effects | +| `pto.tmov` | identity removal、view metadata 和 src/dst alias | +| `pto.tmov_fp` | fp/pre-quant optional operands和地址空间 | +| `pto.tabs` | unary Read(src)/Write(dst) | +| `pto.tand` | binary input/output effects | +| `pto.tands` | scalar form operand 顺序 | +| `pto.tor` | binary input/output effects | +| `pto.tors` | scalar form operand 顺序 | +| `pto.tnot` | unary effects | +| `pto.tneg` | unary effects | +| `pto.tcmp` | mask/output类型和 compare mode | +| `pto.tcmps` | scalar compare operand 顺序 | +| `pto.tconcat` | 多 input root 的 liveness | +| `pto.tconcatidx` | index/output effects | +| `pto.tci` | implicit tmp、A2/A3 scratch、A5 unused placeholder | +| `pto.tcolexpand` | broadcast source/output shape | +| `pto.tcolexpandmul` | binary broadcast effects | +| `pto.tcolexpandmax` | binary broadcast effects | +| `pto.tcolexpandmin` | binary broadcast effects | +| `pto.tcolmax` | reduction tmp/output effects | +| `pto.tcolmin` | reduction tmp/output effects | +| `pto.tcolsum` | `isBinary` 条件 tmp 和 arch 行为 | +| `pto.tcvt` | 条件 tmp、rmode/satmode operand 顺序 | +| `pto.tdiv` | precision attr 和 inplace policy | +| `pto.tdivs` | scalar 顺序和 precision attr | +| `pto.texpands` | shape扩展和 scalar operand | +| `pto.textract` | tile role、offset 和可选 pre-quant | +| `pto.textract_fp` | fp tile role和地址空间 | +| `pto.tinsert` | materialize pass 当前对 tile config 有特殊推断；目标是由 result type 完整携带 | +| `pto.tinsert_fp` | fp/pre-quant tile role | +| `pto.tfillpad` | src/dst alias 和 A5 MAT/PIPE 选择 | +| `pto.tfillpad_inplace` | same-SSA inplace 和 MemoryEffects | +| `pto.tsetval` | tile writer 和 result type | +| `pto.tgetval` | tile reader和 scalar result | +| `pto.tgather` | optional tmp、compare/index form 和 sync macro model | +| `pto.tgatherb` | mask/index buffer effects | +| `pto.tlog` | precision attr | +| `pto.tlrelu` | unary input/output effects | +| `pto.tmax` | binary inplace policy | +| `pto.tmaxs` | scalar form | +| `pto.tmin` | binary inplace policy | +| `pto.tmins` | scalar form | +| `pto.tquant` | 删除 ViewToMemref 内部 tmp 补全；统一由 `PTOMaterializeImplicitTmp` 负责 | +| `pto.tmrgsort` | format1/format2、optional tmp 和多 source roots | +| `pto.tpartadd` | partition参数和 dst writer | +| `pto.tpartmul` | partition参数和 dst writer | +| `pto.tprint` | optional format tmp、tile read 和无输出语义 | + +以下 op 当前没有出现在 Stage 3 的逐 op 重建列表中，通常已经保持 tile-native； +但 `PTOMaterializeTileHandles` 会泛化处理所有 `pto.t*` 的 memref operand，因此 +删除 pass 前仍需逐项确认它们不依赖 memref metadata 回溯： + +| Op | 额外检查 | +|---|---| +| `pto.trowexpandadd` | optional tmp、A2/A3 scratch 和 shared-tmp PIPE_V dependency | +| `pto.trowexpandsub` | optional tmp 和 scratch conflict | +| `pto.trowexpandmul` | optional tmp 和 inplace dst | +| `pto.trowexpanddiv` | optional tmp、precision attr 和 PIPE_V barrier pruning | +| `pto.trowexpandmax` | optional tmp 和 reduction/broadcast mode | +| `pto.trowexpandmin` | optional tmp 和 reduction/broadcast mode | +| `pto.trowexpandexpdif` | tmp effects 和 dst writer | +| `pto.tcolexpandadd` | broadcast input/output alias | +| `pto.tcolexpandsub` | broadcast input/output alias | +| `pto.tcolexpanddiv` | precision attr 和 broadcast alias | +| `pto.tcolexpandexpdif` | tmp/output effects | +| `pto.trowmax` | A2/A3 tmp、A5 placeholder 和 output lifetime | +| `pto.trowmin` | A2/A3 tmp、A5 placeholder 和 output lifetime | +| `pto.trowsum` | A2/A3 tmp、A5 placeholder 和 output lifetime | +| `pto.trowprod` | A2/A3 tmp、A5 placeholder 和 output lifetime | +| `pto.tcolargmax` | tmp capacity、value/index outputs 和多结果 root | +| `pto.tcolargmin` | tmp capacity、value/index outputs 和多结果 root | +| `pto.trowargmax` | index-only/value+index 模式和条件 tmp | +| `pto.trowargmin` | index-only/value+index 模式和条件 tmp | +| `pto.tprelu` | ui8 tmp、tmp/dst no-alias 和 A5 unused tmp | +| `pto.trem` | 2-row tmp、precision attr 和 scratch effects | +| `pto.trems` | 1-row tmp、scalar operand 和 scratch effects | +| `pto.tsel` | ui32 mask tmp、mask/src/dst effects | +| `pto.tsels` | one-row tmp、scalar select 和 A5 unused tmp | +| `pto.tpow` | A2/A3 floating tmp、integer no-tmp 和 A5 no-tmp | +| `pto.tpows` | A2/A3 floating tmp、scalar exponent 和 A5 no-tmp | +| `pto.trsqrt` | API compatibility tmp 不应触发自动 allocation | +| `pto.tsort32` | 非 32 对齐尾部的条件 tmp | +| `pto.txor` | tmp dtype/shape 和 tmp/output conflict | +| `pto.txors` | scalar operand、tmp dtype/shape 和 conflict | +| `pto.tpartmax` | partition参数和 dst writer | +| `pto.tpartmin` | partition参数和 dst writer | + +其它没有在两个 pass 中出现、且 operand/result 始终是 tile-native 的 `pto.t*` +op 默认不需要迁移实现；仍需通过完整 lit 确认它们没有间接依赖 `bind_tile`、 +`pointer_cast` 或 memref metadata 回溯。 + +## 10. InsertSync 专项改造 + +### 10.1 普通 InsertSync + +现有 `PTOIRTranslator` 已直接支持： + +- `pto.alloc_tile` +- `pto.make_tensor_view` +- `pto.partition_view` +- `pto.subview` +- `pto.bind_tile` +- `memref.subview` + +删除两个 pass 前的支持状态： + +1. `pto.treshape` 和 `pto.bitcast` 的 alias 传播。 +2. `pto.declare_tile` runtime-bound root：已完成 EmitC、memplan、InsertSync/GSS 主路径迁移。 +3. `pto.alloc_multi_tile` 的 N-address root：已完成。 +4. `pto.multi_tile_get` 的 constant/dynamic slot narrowing：已完成。 +5. tile 类型 helper argument 和 `func.call` effects。 +6. `scf.yield/iter_arg`、`pto.yield/fusion result` 的 root family 传播检查。 +7. `tile_buf_addr` 结果到原 tile root 的 provenance。 + +`CanPrunePipeVBarrier()` 已支持 `TileBufType`，不需要因删除两个 pass重写； +但新增 view/multi-buffer alias 后必须确认 dependency pair 仍能区分普通 +output-input RAW 和 scratch WAW/WAR。 + +### 10.2 Graph Sync Solver + +GSS 原先主要从 `pto.pointer_cast` 读取物理地址。`getMemInfo()` 的支持状态： + +- `pto.alloc_tile addr` -> 单地址 `PointerLikeInfo`； +- `pto.alloc_multi_tile addrs` -> N 地址 `PointerLikeInfo`：已完成； +- `pto.multi_tile_get` -> constant slot 单地址或 dynamic slot 全地址：已完成； +- `pto.subview` -> base address加 byte offset并缩小访问范围； +- `treshape/bitcast/set_validshape` -> 传播原 root。 + +否则 memplan 把两个不同 SSA allocation 复用到同一物理地址后，GSS 可能漏掉 +cross-root hazard。 + +### 10.3 BufidSync 与 Macro Model + +检查所有 unwrap helper，去掉对 `bind_tile/memref.subview` 的必需依赖，直接穿透： + +- `pto.subview` +- `pto.treshape` +- `pto.bitcast` +- `pto.multi_tile_get` +- `pto.set_validshape` + +## 11. 推荐迁移顺序 + +按照风险从低到高逐 op 处理： + +1. `pto.treshape` +2. `pto.bitcast` +3. `pto.set_validshape/get_validshape` +4. `pto.subview` +5. `pto.tile_buf_addr` +6. `pto.declare_tile` +7. `pto.alloc_multi_tile` +8. `pto.multi_tile_get` +9. `pto.make_tensor_view` +10. `pto.partition_view` +11. `pto.get_tensor_view_dim/get_tensor_view_stride` +12. `pto.inttoptr/ptrtoint/addptr/castptr` +13. `pto.load_scalar/store_scalar/initialize_l2g2l_pipe` +14. `scf.if/scf.for/scf.yield` +15. `pto.fusion_region/pto.yield` +16. `func.func/func.call/helper ABI` +17. `vlds/vsts/vsstb` +18. 逐项回归第 9 节 compute op +19. 删除 `bind_tile/declare_tile_memref/slot_marker` 兼容路径 +20. 从 `ptoas.cpp` 删除两个 pass，随后删除源文件和 Passes.td 注册 + +`alloc_multi_tile/multi_tile_get` 迁移已完成：地址规划、slot identity 和动态 +event id 均不再依赖 `PTOViewToMemref`。剩余 op 仍按上述顺序逐项迁移。 + +## 12. 测试矩阵 + +每批 op 至少运行： + +```text +level1 + legacy memplan +level1 + modern memplan +level2 + legacy memplan +level2 + modern memplan +level3 explicit address + +A3 + InsertSync +A3 + Graph Sync Solver +A5 + BufidSync + +EmitC backend +VPTO backend +``` + +必须重点恢复/改写以下现有测试类别： + +- `materialize_tile_handles_*` +- `multi_tile_*` +- `subview_*` +- `treshape_*` +- `ptr_int_cast.pto` +- PTODSL subkernel helper ABI +- tile fusion control-flow result +- plan memory reused-address sync +- implicit tmp end-to-end + +旧测试中检查 `memref.alloc`、`memref.subview`、`pto.bind_tile`、 +`pto.pointer_cast` 或 `pto.slot_marker` 的部分，应改为检查 tile-native op、规划后 +地址和最终 backend 输出。 + +## 13. 最终删除条件 + +只有满足以下条件才能删除两个 pass： + +- 主 pipeline 中不再产生需要恢复为 tile 的 local memref； +- legacy/modern memplan 都通过完整测试； +- InsertSync/GSS/BufidSync 都能直接分析 tile-native multi-buffer 和 view； +- EmitC/VPTO 不再依赖 bind/memref metadata 回溯； +- helper、控制流、fusion 和 multi-buffer 测试全部通过； +- `rg "PTOViewToMemref|PTOMaterializeTileHandles"` 只剩历史文档； +- `rg "BindTileOp|DeclareTileMemRefOp|SlotMarkerOp"` 不再存在主路径依赖。 diff --git a/include/PTO/IR/PTOAttrs.td b/include/PTO/IR/PTOAttrs.td index b87e1486e6..da4a7237b5 100644 --- a/include/PTO/IR/PTOAttrs.td +++ b/include/PTO/IR/PTOAttrs.td @@ -600,19 +600,6 @@ def PTO_EventAttr : PTO_Attr<"Event", "event"> { }]; } -//===----------------------------------------------------------------------===// -// Plan Memory Mode -//===----------------------------------------------------------------------===// - -def PTO_LOCALMEMPLAN : I32EnumAttrCase<"LOCAL_MEM_PLAN", 0>; -def PTO_GLOBALWORKSPACEPLAN : I32EnumAttrCase<"GLOBAL_WORKSPACE_PLAN", 1>; - -def PTO_MemPlanModeEnum : PTO_I32Enum< - "MemPlanMode", "Mem Plan Mode", [ - PTO_LOCALMEMPLAN, - PTO_GLOBALWORKSPACEPLAN -]>; - //===----------------------------------------------------------------------===// // Pad Mode for LoadOp //===----------------------------------------------------------------------===// diff --git a/include/PTO/IR/PTOMultiBuffer.h b/include/PTO/IR/PTOMultiBuffer.h index 53c89f6c3c..1b99a15108 100644 --- a/include/PTO/IR/PTOMultiBuffer.h +++ b/include/PTO/IR/PTOMultiBuffer.h @@ -9,9 +9,10 @@ //===- PTOMultiBuffer.h - Shared constants for multi-buffer ----*- C++ -*-===// // // Shared constants for the multi-buffer expression scheme: -// - `kPtoMultiBufferAttrName` is the memref-level attribute name written by -// PTOViewToMemref when lowering an `alloc_multi_tile` op. PlanMemory and -// downstream passes read it to reserve N physical slots. +// - `kPtoMultiBufferAttrName` is the legacy memref-level slot-count +// attribute. +// - `kPtoMultiBufferAddrsAttrName` is the internal address list written +// directly on tile-native `pto.alloc_multi_tile` by PlanMemory. // - `kPtoMultiBufferMaxNum` is the upper bound on the slot count N. It is // kept in lock-step with the InsertSync `MAX_MULTI_BUFFER_NUM`. // @@ -29,6 +30,10 @@ namespace pto { inline constexpr llvm::StringLiteral kPtoMultiBufferAttrName = "pto.multi_buffer"; +/// Internal DenseI64ArrayAttr containing one byte address per physical slot. +inline constexpr llvm::StringLiteral kPtoMultiBufferAddrsAttrName = + "pto.multi_buffer_addrs"; + /// Upper bound for N; must stay consistent with `MAX_MULTI_BUFFER_NUM` in /// insert-sync. inline constexpr unsigned kPtoMultiBufferMaxNum = 16; diff --git a/include/PTO/IR/PTOOps.td b/include/PTO/IR/PTOOps.td index c722d5406a..60758484df 100644 --- a/include/PTO/IR/PTOOps.td +++ b/include/PTO/IR/PTOOps.td @@ -343,19 +343,18 @@ def AllocTileOp : PTO_Op<"alloc_tile", [AttrSizedOperandSegments]> { //===----------------------------------------------------------------------===// // // `pto.alloc_multi_tile` declares an N-slot logical tile buffer. The actual -// physical allocation of N slots is performed by PTOPlanMemory once the op -// has been lowered to `memref.alloc {pto.multi_buffer = N : i32}` by -// PTOViewToMemref. +// physical allocation of N slots is performed directly by PTOPlanMemory. +// The op remains tile-native until PTOResolveBufferSelect materializes an +// addressed `pto.alloc_tile` for each selected slot. // // `pto.multi_tile_get` is the *only* way to consume a `multi_tile_buf`: it // returns a regular `tile_buf` view onto the chosen slot. The slot index // may be a constant or any SSA `index` value; PTOAS does not synthesize // `iv mod N` automatically -- the user expression IS the slot selector. // -// `pto.slot_marker` is an internal op materialized by PTOViewToMemref to -// thread the slot SSA from `multi_tile_get` through the memref layer down -// to PlanMemory / sync analysis. It is not intended for direct frontend -// use. +// `pto.slot_marker` remains as an internal compatibility op for legacy +// memref-based multi-buffer IR. PTOViewToMemref does not create it for +// tile-native `pto.multi_tile_get`. def AllocMultiTileOp : PTO_Op<"alloc_multi_tile", [AttrSizedOperandSegments]> { let summary = "Allocate an N-slot multi-buffer tile"; @@ -415,9 +414,9 @@ def MultiTileGetOp : PTO_Op<"multi_tile_get", [ sync analysis treats this use as a slot-indexed dynamic access; ptoas does NOT rewrite the user expression into `iv mod N`. - This op is metadata-only (no data movement); the lowering merely - annotates the underlying memref view with the slot index for PlanMemory - / sync / EnableBufferSelect to consume. + This op is metadata-only (no data movement). PlanMemory preserves it, + sync analysis consumes its slot index directly, and + PTOResolveBufferSelect replaces it with an addressed `pto.alloc_tile`. }]; let arguments = (ins @@ -560,7 +559,7 @@ def SetValidShapeOp : PTO_Op<"set_validshape", [ }]; let arguments = (ins - TileBufOrMemRef:$source, + TileBufType:$source, Index:$valid_row, Index:$valid_col ); @@ -584,7 +583,7 @@ def GetValidShapeOp : PTO_Op<"get_validshape", [ }]; let arguments = (ins - TileBufOrMemRef:$source + TileBufType:$source ); let results = (outs @@ -1556,11 +1555,10 @@ def SlotMarkerOp : PTO_Op<"slot_marker", [ ]> { let summary = "Tag a memref view as referring to one slot of a multi_tile_buf"; let description = [{ - Internal op materialized by `PTOViewToMemref` while lowering - `pto.multi_tile_get`. It carries the slot SSA index through the memref - layer so that PlanMemory, sync analysis (InsertSync / GraphSyncSolver), - and the buffer-select lowering pass can identify which physical slot - this memref reference touches. + Internal compatibility op for legacy memref-based multi-buffer IR. It + carries a slot SSA index so sync analysis (InsertSync / GraphSyncSolver) + and the buffer-select lowering pass can identify which physical slot a + memref reference touches. The op is metadata-only (no data movement, no extra address arithmetic); its result memref aliases the source memref byte-for-byte. Frontends do @@ -2634,12 +2632,10 @@ def LocalArraySetOp : PTO_Op<"local_array_set"> { } def DeclareTileMemRefOp : PTO_Op<"declare_tile_memref"> { - let summary = "Internal memref placeholder for a tile whose address is assigned later"; + let summary = "Legacy memref placeholder for a runtime-bound tile"; let description = [{ - Internal lowering op used by PTOViewToMemref. This op does not allocate - storage; it only provides a memref-typed SSA handle so later passes can - attach tile metadata through pto.bind_tile before the address is filled by - pipe operations such as pto.tpop. + Compatibility op for legacy memref-based IR. New PTOViewToMemref pipelines + preserve `pto.declare_tile` directly instead of producing this op. }]; let results = (outs AnyMemRef:$result); diff --git a/include/PTO/IR/VPTOOps.td b/include/PTO/IR/VPTOOps.td index 757034284e..603a18dc51 100644 --- a/include/PTO/IR/VPTOOps.td +++ b/include/PTO/IR/VPTOOps.td @@ -220,11 +220,10 @@ def TensorViewAddrOp : PTO_Op<"tensor_view_addr", [Pure]> { typed PTO pointer, depending on the requested destination type. In authoring-form IR this op preserves the descriptor-style surface; - during view-to-memref lowering it collapses to the underlying memref value - or to a memref-derived pointer. + tile-native inputs remain unchanged through memory planning and sync. - This op may also accept a memref operand after earlier view lowering, in - which case it behaves as an identity marker and is removed by lowering. + This op may also accept a legacy memref operand, in which case it behaves + as an identity marker and is removed by compatibility lowering. }]; let arguments = (ins AnyTypeOf<[TensorViewType, PartitionTensorViewType, AnyMemRef], @@ -247,9 +246,9 @@ def TileBufAddrOp : PTO_Op<"tile_buf_addr", [Pure]> { shape and address space. This op is emitted by TileLang DSL templates and resolved by the - FoldTileBufIntrinsics pass after inlining. Hand-written `.pto` may also - use it directly on the memref result of `pto.bind_tile` / lowered - `pto.alloc_tile`. + FoldTileBufIntrinsics pass after inlining. Tile-native uses remain in PTO + IR through memory planning and sync. Hand-written legacy `.pto` may also + use it directly on the memref result of `pto.bind_tile`. }]; let arguments = (ins AnyTypeOf<[TileBufType, AnyMemRef], diff --git a/include/PTO/Transforms/InsertSync/PTOIRTranslator.h b/include/PTO/Transforms/InsertSync/PTOIRTranslator.h index 71b3f1b10e..0012f79f29 100644 --- a/include/PTO/Transforms/InsertSync/PTOIRTranslator.h +++ b/include/PTO/Transforms/InsertSync/PTOIRTranslator.h @@ -70,6 +70,8 @@ class PTOIRTranslator { // --- 内存/Alias 分析 --- void UpdateKernelArgMemInfo(); LogicalResult UpdateAllocTileOpMemInfo(pto::AllocTileOp op); + LogicalResult UpdateAllocMultiTileOpMemInfo(pto::AllocMultiTileOp op); + LogicalResult UpdateDeclareTileOpMemInfo(pto::DeclareTileOp op); LogicalResult UpdateDeclareGlobalOpMemInfo(pto::DeclareGlobalOp op); LogicalResult UpdateDeclareTileMemRefOpMemInfo(pto::DeclareTileMemRefOp op); LogicalResult UpdatePointerCastOpMemInfo(pto::PointerCastOp op); @@ -81,6 +83,9 @@ class PTOIRTranslator { void UpdateMemrefSubViewAliasBufferInfo(memref::SubViewOp op); void UpdateTileSubViewAliasBufferInfo(pto::SubViewOp op); void UpdateSlotMarkerAliasBufferInfo(pto::SlotMarkerOp op); + void UpdateMultiTileGetAliasBufferInfo(pto::MultiTileGetOp op); + void UpdateSlotSelectedAliasBufferInfo(Value result, Value source, + Value slot); // --- 控制流处理 (SCF) --- void UpdateForOpInfo(scf::ForOp forOp); diff --git a/include/PTO/Transforms/Passes.h b/include/PTO/Transforms/Passes.h index 3555735a06..d88b918c95 100644 --- a/include/PTO/Transforms/Passes.h +++ b/include/PTO/Transforms/Passes.h @@ -67,10 +67,10 @@ std::unique_ptr createConvertToPTOOpPass(); /// PTO Ops. std::unique_ptr createInferPTOMemScopePass(); -/// Create a pass to plan memory. std::unique_ptr -createPlanMemoryPass(const PlanMemoryOptions &planMemoryOption = {}); - +createPlanMemoryPass(const PlanMemoryOptions &options = {}); +std::unique_ptr +createPlanMemoryModernPass(const PlanMemoryOptions &options); std::unique_ptr createPTORemoveRedundantBarrierPass(); std::unique_ptr createPTOViewToMemrefPass(); std::unique_ptr createPTOValidateIntToPtrUsesPass(); diff --git a/include/PTO/Transforms/Passes.td b/include/PTO/Transforms/Passes.td index 00b57ee697..210f9d8160 100644 --- a/include/PTO/Transforms/Passes.td +++ b/include/PTO/Transforms/Passes.td @@ -193,37 +193,26 @@ def PTORematerializeFixpipeVectorQuant } def PlanMemory : Pass<"pto-plan-memory", "ModuleOp"> { - let summary = "Plan memory for PTO Ops"; + let summary = "Plan memory for PTO Ops"; let constructor = "mlir::pto::createPlanMemoryPass()"; - let dependentDialects = ["pto::PTODialect", ]; + let options = [ - Option<"memMode", "mem-plan-mode", "pto::MemPlanMode", - "pto::MemPlanMode::LOCAL_MEM_PLAN", - "plan mem mode (default is LOCAL_MEM_PLAN)", - [{::llvm::cl::values( - clEnumValN(pto::MemPlanMode::LOCAL_MEM_PLAN, "local-mem-plan", - "plan mem mode is for memref.alloc"), - clEnumValN( - pto::MemPlanMode::GLOBAL_WORKSPACE_PLAN, - "global-work-space-plan", - "plan mem mode is for memref_ext.alloc_workspace"))}]>, - Option<"enableGlobalReuse", "enable-global-workspace-reuse", "bool", - /*default=*/"false", - "Enable global workspace reuse ,default : false">, - Option<"enablePrintMemoryAllocatedSize", "enable-print-memory-allocated-size", "bool", - /*default=*/"false", - "Enable print memory allocated size, default : false">, - Option<"restrictInplaceAsISA", "restrict-inplace-as-isa", "bool", - /*default=*/"false", - "restrict memory inplace as isa, default : false">, - Option<"orderBySize", "order-by-size", "bool", - /*default=*/"false", - "Process buffers largest-first (first-fit-decreasing order) during " - "local memory planning instead of the default DMA-first order. " - "Decreasing-size order packs heterogeneous-size buffers tighter " - "(matches XLA/TVM/SOMAS). default : false">, + Option<"memMode", "mem-mode", "std::string", "\"local\"", + "Planning mode. Supported value today: local">, + Option<"orderBySize", "order-by-size", "bool", "false", + "Plan larger buffers first inside one AddressSpace before applying " + "SPEC_LEVEL_0 first-fit reuse"> + ]; + + let dependentDialects = [ + "mlir::pto::PTODialect", + "mlir::memref::MemRefDialect", + "mlir::arith::ArithDialect", + "mlir::func::FuncDialect", + "mlir::scf::SCFDialect" ]; } + def PTOLoweringSyncToPipe : Pass<"pto-lowering-sync-to-pipe", "func::FuncOp"> { let summary = "Lower high-level sync ops to low-level pipe ops"; let description = [{ @@ -752,25 +741,20 @@ def PTOMaterializeTileHandles : Pass<"pto-materialize-tile-handles", "ModuleOp"> } def PTOResolveBufferSelect : Pass<"pto-resolve-buffer-select", "ModuleOp"> { - let summary = "Lower pto.slot_marker to single-slot pointer_cast (multi-buffer)"; + let summary = "Resolve multi-buffer slot selections to addressed tile handles"; let description = [{ - Consumes `pto.slot_marker %src[%k]` ops written by PTOViewToMemref to - thread multi-buffer slot selection through the memref layer. - - The op is replaced by a fresh single-address `pto.pointer_cast` that - refers to the chosen physical slot: + Primarily consumes tile-native `pto.multi_tile_get %src[%k]` operations. + The source `pto.alloc_multi_tile` carries either planner-assigned physical + slot addresses or a level3 base address. Each selection is replaced by a + `pto.alloc_tile` carrying the selected address; constant slots select one + address directly, while dynamic slots use an N-way `arith.select` chain. - - constant slot `k`: a single `pto.pointer_cast(addrK)` is emitted at - the use site, using slot k from the underlying multi-address - pointer_cast created by PTOPlanMemory / AllocToPointerCast. - - dynamic slot `%k`: per-slot single-address `pto.pointer_cast`s are - created and an N-way `arith.select` chain picks one according to the - user-supplied SSA. This pass does NOT rewrite `%k` into `iv mod N`; - the frontend expression is the slot selector. + The pass also retains compatibility with legacy memref-based + `pto.slot_marker` IR, where the selection is materialized as a + single-address `pto.pointer_cast`. - The original multi-address `pto.pointer_cast` is kept in IR as the - "alloc anchor" so future sync passes can recognize the multi-buffer - geometry. + This pass does not rewrite `%k` into `iv mod N`; the frontend expression + remains the slot selector. }]; let constructor = "mlir::pto::createPTOResolveBufferSelectPass()"; diff --git a/lib/PTO/IR/PTO.cpp b/lib/PTO/IR/PTO.cpp index 1b66634049..30000d78e4 100644 --- a/lib/PTO/IR/PTO.cpp +++ b/lib/PTO/IR/PTO.cpp @@ -2408,13 +2408,19 @@ LogicalResult mlir::pto::MakeTensorViewOp::verify() { if (!tvTy) return emitOpError("result must be pto.tensor_view<...>"); - auto pty = dyn_cast(getPtr().getType()); - if (!pty) - return emitOpError("ptr operand must be !pto.ptr<...>"); + Type ptrElemTy; + if (auto pty = dyn_cast(getPtr().getType())) + ptrElemTy = pty.getElementType(); + else if (auto memrefTy = dyn_cast(getPtr().getType())) + ptrElemTy = memrefTy.getElementType(); + else + return emitOpError( + "ptr operand must be !pto.ptr<...> or a memref-backed pointer"); - if (pty.getElementType() != tvTy.getElementType()) - return emitOpError() << "ptr element type must match tensor_view element type, but got ptr=" - << pty.getElementType() << " view=" << tvTy.getElementType(); + if (ptrElemTy != tvTy.getElementType()) + return emitOpError() << "ptr element type must match tensor_view element " + "type, but got ptr=" + << ptrElemTy << " view=" << tvTy.getElementType(); int64_t rank = tvTy.getRank(); @@ -3036,6 +3042,43 @@ LogicalResult AllocMultiTileOp::verify() { << kPtoMultiBufferMaxNum << "] (got " << count << ")"; } + if (Attribute rawAddrs = (*this)->getAttr(kPtoMultiBufferAddrsAttrName)) { + auto addrs = dyn_cast(rawAddrs); + if (!addrs) + return emitOpError() << "expects internal '" + << kPtoMultiBufferAddrsAttrName + << "' to be a dense i64 array"; + if (getAddr()) + return emitOpError() << "cannot carry both base 'addr' and internal '" + << kPtoMultiBufferAddrsAttrName << "'"; + if (addrs.size() != count) + return emitOpError() << "expects " << count << " planned slot addresses, got " + << addrs.size(); + + uint64_t elemBytes = getPTOStorageElemByteSize(slotTy.getElementType()); + uint64_t slotBytes = elemBytes; + for (int64_t dim : slotTy.getShape()) { + if (dim == ShapedType::kDynamic) + return emitOpError( + "planned multi-buffer addresses require a static slot shape"); + slotBytes *= static_cast(dim); + } + for (auto [lhsIdx, lhs] : llvm::enumerate(addrs.asArrayRef())) { + if (lhs < 0) + return emitOpError("planned slot addresses must be non-negative"); + uint64_t lhsBegin = static_cast(lhs); + uint64_t lhsEnd = lhsBegin + slotBytes; + for (size_t rhsIdx = lhsIdx + 1; + rhsIdx < static_cast(addrs.size()); ++rhsIdx) { + uint64_t rhsBegin = static_cast(addrs[rhsIdx]); + uint64_t rhsEnd = rhsBegin + slotBytes; + if (std::max(lhsBegin, rhsBegin) < std::min(lhsEnd, rhsEnd)) + return emitOpError() << "planned slots " << lhsIdx << " and " + << rhsIdx << " overlap"; + } + } + } + return success(); } @@ -6343,7 +6386,18 @@ llvm::LogicalResult mlir::pto::TCvtOp::verify() { if (failed(verifyTileBufCommon(*this, srcTy, "src", /*allowLowPrecision=*/true)) || failed(verifyTileBufCommon(*this, dstTy, "dst", /*allowLowPrecision=*/true))) return failure(); - if (failed(verifyTileBufSameLogicalExtent(*this, srcTy, dstTy, "src", "dst", + auto isResolvedSubview = [](Value value) { + auto alloc = value.getDefiningOp(); + auto semantics = + alloc ? alloc->getAttrOfType("pto.view_semantics") + : StringAttr(); + return semantics && semantics.getValue() == "subview"; + }; + // A resolved subview keeps its parent's physical shape so the generated + // Tile retains the parent stride. Its valid shape is the logical tcvt + // extent, so comparing physical shapes would reject a valid sliced tile. + if (!isResolvedSubview(getSrc()) && !isResolvedSubview(getDst()) && + failed(verifyTileBufSameLogicalExtent(*this, srcTy, dstTy, "src", "dst", /*compareValidShape=*/false))) return failure(); if (failed(verifyTileBufSameLogicalExtent(*this, srcTy, dstTy, "src", "dst", @@ -11112,9 +11166,6 @@ static std::optional getElemBytes(Type elemTy) { return mlir::isa(ty); } -static constexpr llvm::StringLiteral kLoweredSetValidShapeAttrName = - "__pto.lowered_set_validshape"; - static bool isLocallyBoundTileSource(Value value) { if (!value || isa(value)) return false; @@ -11154,32 +11205,22 @@ static std::optional getConstIndexLike(Value v) { mlir::LogicalResult mlir::pto::SetValidShapeOp::verify() { SmallVector shape; - if (auto srcTy = llvm::dyn_cast(getSource().getType())) { - if (srcTy.getRank() != 2) - return emitOpError("expects rank-2 tile_buf source"); + auto srcTy = getSource().getType(); + if (srcTy.getRank() != 2) + return emitOpError("expects rank-2 tile_buf source"); - ArrayRef validShape = srcTy.getValidShape(); - if (validShape.size() != 2) - return emitOpError("expects source validShape to be rank-2"); - if (!srcTy.hasDynamicValid()) - return emitOpError("expects source tile_buf to have dynamic validShape (?, ?)"); + ArrayRef validShape = srcTy.getValidShape(); + if (validShape.size() != 2) + return emitOpError("expects source validShape to be rank-2"); + if (!srcTy.hasDynamicValid()) + return emitOpError("expects source tile_buf to have dynamic validShape (?, ?)"); - shape.assign(srcTy.getShape().begin(), srcTy.getShape().end()); + shape.assign(srcTy.getShape().begin(), srcTy.getShape().end()); - if (!isLocallyBoundTileSource(getSource())) - return emitOpError( - "requires a locally bound tile source; function arguments/results " - "are unsupported"); - } else if (auto srcTy = llvm::dyn_cast(getSource().getType())) { - if (!(*this)->hasAttr(kLoweredSetValidShapeAttrName)) - return emitOpError( - "expects tile_buf source; memref source is only valid for the internal lowered form"); - if (srcTy.getRank() != 2) - return emitOpError("expects rank-2 memref source after tile lowering"); - shape.assign(srcTy.getShape().begin(), srcTy.getShape().end()); - } else { - return emitOpError("expects tile_buf source (or lowered memref source)"); - } + if (!isLocallyBoundTileSource(getSource())) + return emitOpError( + "requires a locally bound tile source; function arguments/results " + "are unsupported"); auto checkDim = [&](Value operand, unsigned dimIdx, StringRef dimName) -> LogicalResult { @@ -11206,19 +11247,12 @@ mlir::LogicalResult mlir::pto::SetValidShapeOp::verify() { } mlir::LogicalResult mlir::pto::GetValidShapeOp::verify() { - if (auto srcTy = llvm::dyn_cast(getSource().getType())) { - if (srcTy.getRank() != 2) - return emitOpError("expects rank-2 tile_buf source"); - if (srcTy.getValidShape().size() != 2) - return emitOpError("expects source validShape to be rank-2"); - return success(); - } - if (auto srcTy = llvm::dyn_cast(getSource().getType())) { - if (srcTy.getRank() != 2) - return emitOpError("expects rank-2 memref source after tile lowering"); - return success(); - } - return emitOpError("expects tile_buf source (or lowered memref source)"); + auto srcTy = getSource().getType(); + if (srcTy.getRank() != 2) + return emitOpError("expects rank-2 tile_buf source"); + if (srcTy.getValidShape().size() != 2) + return emitOpError("expects source validShape to be rank-2"); + return success(); } diff --git a/lib/PTO/Transforms/AllocToPointerCast.cpp b/lib/PTO/Transforms/AllocToPointerCast.cpp index f560458e5f..925993e8ec 100644 --- a/lib/PTO/Transforms/AllocToPointerCast.cpp +++ b/lib/PTO/Transforms/AllocToPointerCast.cpp @@ -10,16 +10,9 @@ //===----------------------------------------------------------------------===// #include "AllocToPointerCast.h" +#include "PTO/IR/PTOMultiBuffer.h" #include "PTO/IR/PTOTypeUtils.h" -#include "PTO/Transforms/Passes.h" #include "mlir/Dialect/MemRef/IR/MemRef.h" -#include "mlir/Transforms/GreedyPatternRewriteDriver.h" - -namespace mlir { -#define GEN_PASS_DEF_ALLOCTOPOINTERCAST -#include "PTO/Transforms/Passes.h.inc" - -} // namespace mlir using namespace mlir; using namespace mlir::pto; @@ -30,62 +23,77 @@ namespace { constexpr uint64_t kDefaultAllocAlignmentBytes = 4096; constexpr size_t kDynamicValidShapeRank = 2; -static TileBufConfigAttr inferBindTileConfig(memref::AllocOp op) { +static uint64_t alignUpToDefault(uint64_t value) { + return ((value + kDefaultAllocAlignmentBytes - 1) / + kDefaultAllocAlignmentBytes) * + kDefaultAllocAlignmentBytes; +} + +static TileBufConfigAttr inferBindTileConfig(Value root, Operation *diagOp) { TileBufConfigAttr configAttr; - for (Operation *user : op.getResult().getUsers()) { + for (Operation *user : root.getUsers()) { auto bind = dyn_cast(user); - if (!bind || bind.getSource() != op.getResult()) + if (!bind || bind.getSource() != root) continue; if (!configAttr) { configAttr = bind.getConfigAttr(); continue; } if (configAttr != bind.getConfigAttr()) { - op.emitWarning("alloc has multiple bind_tile users with different configs; " - "using the first one"); + diagOp->emitWarning( + "alloc has multiple bind_tile users with different configs; " + "using the first one"); break; } } return configAttr; } -static SmallVector getAllocatedOffsets(memref::AllocOp op, - BaseMemRefType memRefType, - const DenseMap> &buffer2Offsets, - uint64_t &fallbackNextOffset) { - auto iter = buffer2Offsets.find(op.getResult()); +static SmallVector getAllocatedOffsets( + Value root, Operation *allocLikeOp, BaseMemRefType memRefType, + const DenseMap> &buffer2Offsets, + uint64_t &fallbackNextOffset) { + auto iter = buffer2Offsets.find(root); SmallVector offsets; if (iter != buffer2Offsets.end()) offsets = iter->second; - if (offsets.empty()) { - // Estimate buffer size (best-effort). Most PTO tile buffers are 32x32 and - // naturally align to 4096 bytes. - uint64_t bytes = kDefaultAllocAlignmentBytes; - if (auto memrefTy = dyn_cast(memRefType)) { - uint64_t elemBytes = getPTOStorageElemByteSize(memrefTy.getElementType()); - if (elemBytes != 0) { - uint64_t numel = 1; - bool allStatic = true; - for (int64_t d : memrefTy.getShape()) { - if (d == ShapedType::kDynamic) { - allStatic = false; - break; - } - numel *= static_cast(d); + if (!offsets.empty()) + return offsets; + + // Estimate buffer size (best-effort). Most PTO tile buffers are 32x32 and + // naturally align to 4096 bytes. + uint64_t bytes = kDefaultAllocAlignmentBytes; + if (auto memrefTy = dyn_cast(memRefType)) { + uint64_t elemBytes = getPTOStorageElemByteSize(memrefTy.getElementType()); + if (elemBytes != 0) { + uint64_t numel = 1; + bool allStatic = true; + for (int64_t d : memrefTy.getShape()) { + if (d == ShapedType::kDynamic) { + allStatic = false; + break; } - if (allStatic && numel != 0) - bytes = numel * elemBytes; + numel *= static_cast(d); } + if (allStatic && numel != 0) + bytes = numel * elemBytes; } - uint64_t stride = ((bytes + kDefaultAllocAlignmentBytes - 1) / - kDefaultAllocAlignmentBytes) * - kDefaultAllocAlignmentBytes; - uint64_t off = fallbackNextOffset; - fallbackNextOffset += - std::max(stride, kDefaultAllocAlignmentBytes); - offsets.push_back(off); } + + uint64_t slotCount = 1; + if (auto attr = + allocLikeOp->getAttrOfType(pto::kPtoMultiBufferAttrName)) { + slotCount = attr.getValue().getZExtValue(); + } + + uint64_t stride = alignUpToDefault(bytes); + uint64_t off = fallbackNextOffset; + for (uint64_t i = 0; i < slotCount; ++i) + offsets.push_back(off + i * stride); + + fallbackNextOffset += + std::max(stride * slotCount, kDefaultAllocAlignmentBytes); return offsets; } @@ -106,9 +114,10 @@ static std::pair getDynamicValidShapeValues(memref::AllocOp op) { LogicalResult MemrefAllocaOpToPointerCastOpPattern::matchAndRewrite( memref::AllocOp op, PatternRewriter &rewriter) const { const auto ¤tMemRefType = cast(op.getType()); - TileBufConfigAttr configAttr = inferBindTileConfig(op); - SmallVector offsets = getAllocatedOffsets( - op, currentMemRefType, buffer2Offsets, fallbackNextOffset); + TileBufConfigAttr configAttr = inferBindTileConfig(op.getResult(), op); + SmallVector offsets = + getAllocatedOffsets(op.getResult(), op, currentMemRefType, + buffer2Offsets, fallbackNextOffset); SmallVector addrs; addrs.reserve(offsets.size()); for (uint64_t offset : offsets) { diff --git a/lib/PTO/Transforms/CMakeLists.txt b/lib/PTO/Transforms/CMakeLists.txt index ccd135000a..92956a1a3d 100644 --- a/lib/PTO/Transforms/CMakeLists.txt +++ b/lib/PTO/Transforms/CMakeLists.txt @@ -55,6 +55,7 @@ add_mlir_dialect_library(PTOTransforms AllocToPointerCast.cpp InferPTOMemScope.cpp PTOPlanMemory.cpp + PTOPlanMemoryModern.cpp PTORemoveRedundantBarrier.cpp InferPTOLayout.cpp PTOA5NormalizeTMovPass.cpp diff --git a/lib/PTO/Transforms/GraphSyncSolver/MemInfo.cpp b/lib/PTO/Transforms/GraphSyncSolver/MemInfo.cpp index ba67ba3990..b0adfb765d 100644 --- a/lib/PTO/Transforms/GraphSyncSolver/MemInfo.cpp +++ b/lib/PTO/Transforms/GraphSyncSolver/MemInfo.cpp @@ -11,6 +11,7 @@ #include "PTO/Transforms/GraphSyncSolver/MemInfo.h" #include "PTO/IR/PTO.h" +#include "PTO/IR/PTOMultiBuffer.h" #include "PTO/IR/PTOTypeUtils.h" #include "../Utils.h" #include "mlir/Dialect/Arith/IR/Arith.h" @@ -26,6 +27,19 @@ using namespace pto::syncsolver; namespace mlir::pto::syncsolver { static std::optional getBufferBitSize(Value value) { + if (auto multiType = dyn_cast(value.getType())) { + auto slotType = multiType.getSlotType(); + auto bitWidth = getPTOStorageElemBitWidth(slotType.getElementType()); + if (bitWidth == 0) + return ShapedType::kDynamic; + int64_t numElements = 1; + for (int64_t dim : slotType.getShape()) { + if (dim == ShapedType::kDynamic) + return ShapedType::kDynamic; + numElements *= dim; + } + return numElements * bitWidth; + } auto shaped = dyn_cast(value.getType()); if (!shaped || !shaped.hasStaticShape()) { return ShapedType::kDynamic; @@ -72,6 +86,55 @@ PointerLikeInfo getPointerLikeInfo(pto::PointerCastOp pointerCastOp) { return pointerLikeInfo; } +static std::optional getConstantI64(Value value) { + IntegerAttr attr; + if (!value || !matchPattern(value, m_Constant(&attr))) + return std::nullopt; + return attr.getValue().getSExtValue(); +} + +static PointerLikeInfo getPointerLikeInfo(pto::AllocMultiTileOp alloc) { + PointerLikeInfo info(alloc); + info.allocateSize = getBufferBitSize(alloc.getResult()); + auto slotType = alloc.getResult().getType().getSlotType(); + if (auto space = dyn_cast_or_null( + slotType.getMemorySpace())) + info.addressSpace = space.getAddressSpace(); + + if (auto planned = alloc->getAttrOfType( + pto::kPtoMultiBufferAddrsAttrName)) { + for (int64_t address : planned.asArrayRef()) + info.addresses.push_back(address * pto::kBitsToByte); + } else if (auto base = getConstantI64(alloc.getAddr())) { + int64_t slotBits = info.allocateSize.value_or(ShapedType::kDynamic); + for (uint32_t slot = 0; slot < alloc.getResult().getType().getCount(); ++slot) + info.addresses.push_back(*base * pto::kBitsToByte + slot * slotBits); + } + if (auto loop = alloc->getParentOfType()) + info.parentLoop = loop; + return info; +} + +static MemInfo getMemInfoForMultiTileGet(pto::MultiTileGetOp get) { + auto alloc = get.getSource().getDefiningOp(); + if (!alloc) + return MemInfo(get.getResult(), isWorkSpaceFuncArgument(get.getResult())); + + PointerLikeInfo info = getPointerLikeInfo(alloc); + IntegerAttr slotAttr; + if (matchPattern(get.getSlot(), m_Constant(&slotAttr)) && + info.addresses.size() > 1) { + int64_t slot = slotAttr.getValue().getSExtValue(); + if (slot >= 0 && slot < static_cast(info.addresses.size())) { + int64_t address = info.addresses[static_cast(slot)]; + info.addresses.assign(1, address); + } + } + if (auto loop = get->getParentOfType()) + info.parentLoop = loop; + return MemInfo(get.getResult(), info); +} + // Walk back through metadata-only view ops (`pto.bind_tile`) to the // nearest `pto.pointer_cast`. Used to anchor slot_marker MemInfo on its // underlying multi-address alloc cast. @@ -143,6 +206,10 @@ MemInfo getMemInfo(Value val) { if (auto slotMarker = llvm::dyn_cast(defOp)) { return getMemInfoForSlotMarker(slotMarker); } + if (auto allocMulti = llvm::dyn_cast(defOp)) + return MemInfo(val, getPointerLikeInfo(allocMulti)); + if (auto multiGet = llvm::dyn_cast(defOp)) + return getMemInfoForMultiTileGet(multiGet); } return MemInfo(val, isWorkSpaceFuncArgument(val)); } diff --git a/lib/PTO/Transforms/GraphSyncSolver/SyncSolverIRTranslator.cpp b/lib/PTO/Transforms/GraphSyncSolver/SyncSolverIRTranslator.cpp index 38e8c602e0..9e9758e541 100644 --- a/lib/PTO/Transforms/GraphSyncSolver/SyncSolverIRTranslator.cpp +++ b/lib/PTO/Transforms/GraphSyncSolver/SyncSolverIRTranslator.cpp @@ -81,7 +81,7 @@ llvm::SmallVector IRTranslator::tracebackMemValsStep(Value val) { // `getOperationAliasInfo` path below would treat slot_marker as a // transparent view and let the trace fall through to the underlying // multi-address `pto.pointer_cast`, dropping the slot. - if (isa(defOp)) { + if (isa(defOp)) { return out; } @@ -131,8 +131,9 @@ llvm::SmallVector IRTranslator::tracebackMemVals(Value val) { // stop, `getOperationAliasInfo` would let the walk slip past slot_marker // and reach the underlying multi-address `pto.pointer_cast`, dropping // the slot index. - if (isa(defOp)) { + if (isa(defOp)) { leaves.insert(result); continue; } diff --git a/lib/PTO/Transforms/InsertSync/MemoryDependentAnalyzer.cpp b/lib/PTO/Transforms/InsertSync/MemoryDependentAnalyzer.cpp index 5d8a5e9789..2088f159db 100644 --- a/lib/PTO/Transforms/InsertSync/MemoryDependentAnalyzer.cpp +++ b/lib/PTO/Transforms/InsertSync/MemoryDependentAnalyzer.cpp @@ -15,6 +15,7 @@ #include "PTO/Transforms/InsertSync/InsertSyncDebug.h" #include "mlir/Interfaces/ViewLikeInterface.h" #include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/Arith/IR/Arith.h" #include "llvm/Support/Debug.h" #define DEBUG_TYPE "pto-inject-sync" @@ -104,7 +105,93 @@ static Value GetRealRoot(Value v) { } return v; } - + +// Extract the base UB address carried by `rootBuffer` when it is the i64 +// address SSA operand of a `pto.pointer_cast` (single-address path). In that +// model `rootBuffer` is an `arith.constant` i64 whose value is the absolute UB +// offset, and `BaseMemInfo::baseAddresses` holds {0} (relative deltas). For +// the multi-address path `rootBuffer` is the `PointerCastOp` result itself +// (not an i64 constant), so this returns 0 and `baseAddresses` already holds +// the absolute slot offsets — making `rootBase + baseAddresses[i]` resolve to +// the absolute address in both models. +static uint64_t getRootBaseAddress(Value rootBuffer) { + if (!rootBuffer) + return 0; + if (auto cst = rootBuffer.getDefiningOp()) { + if (auto attr = dyn_cast(cst.getValue())) + return attr.getValue().getZExtValue(); + } + return 0; +} + +static bool isIntegerConstant(Value value) { + if (!value) + return false; + auto cst = value.getDefiningOp(); + return cst && isa(cst.getValue()); +} + +static bool hasKnownLocalAbsoluteAddress(const BaseMemInfo *info) { + if (!info || info->baseAddresses.empty() || info->allocateSize == 0) + return false; + + Value root = info->rootBuffer; + if (!root) + return false; + + if (auto alloc = root.getDefiningOp()) + return isIntegerConstant(alloc.getAddr()); + + if (isIntegerConstant(root)) + return true; + + // Multi-address pointer_cast records absolute slot offsets in baseAddresses + // and uses the pointer_cast result as rootBuffer. + if (auto ptrCast = root.getDefiningOp()) + return ptrCast.getAddrs().size() > 1; + + return false; +} + +// Cross-root byte-range overlap check for local memory (UB/L1). +// +// `MemAlias` historically only checked byte-range overlap when the two +// `rootBuffer` SSA values were identical (same `alloc_tile`/`pointer_cast`). +// When PlanMemory reuses the same physical UB region for two *different* +// allocations, their `rootBuffer` values differ even though their byte ranges +// overlap — so `MemAlias` returned false and InsertSync silently dropped the +// cross-pipe hazard (issue #934: MTE3 tstore racing a V-pipe write into an +// overlapping-but-different-root buffer). +// +// This helper re-derives the absolute address as `getRootBaseAddress(root) + +// baseAddresses[i]` and performs the same `maxStart < minEnd` overlap test +// across every address pair, regardless of root identity. +static bool isLocalBufferOverlapCrossRoot(const BaseMemInfo *a, + const BaseMemInfo *b) { + // Cross-root checks are only meaningful after physical addresses have been + // materialized. For unknown-address roots keep the historical behavior: + // different roots are treated as independent. + if (!hasKnownLocalAbsoluteAddress(a) || !hasKnownLocalAbsoluteAddress(b)) + return false; + + uint64_t rootBaseA = getRootBaseAddress(a->rootBuffer); + uint64_t rootBaseB = getRootBaseAddress(b->rootBuffer); + + for (uint64_t addrA : a->baseAddresses) { + for (uint64_t addrB : b->baseAddresses) { + uint64_t aStart = rootBaseA + addrA; + uint64_t bStart = rootBaseB + addrB; + uint64_t aEnd = aStart + a->allocateSize; + uint64_t bEnd = bStart + b->allocateSize; + uint64_t maxStart = std::max(aStart, bStart); + uint64_t minEnd = std::min(aEnd, bEnd); + if (maxStart < minEnd) + return true; + } + } + return false; +} + bool MemoryDependentAnalyzer::DepBetween( const SmallVector &a, const SmallVector &b, @@ -196,8 +283,20 @@ bool MemoryDependentAnalyzer::MemAlias(const BaseMemInfo *a, if (isTraceEnabled()) llvm::errs() << " -> Mismatch. Real roots differ.\n"; } - - return false; + + // 2.3 Cross-root absolute-address overlap check. + // + // Roots genuinely differ (different alloc_tile / different pointer_cast + // address SSA). Historically MemAlias returned false here, but PlanMemory + // can reuse the same physical UB region for distinct allocations whose + // byte ranges overlap. Re-derive the absolute address and check overlap so + // a cross-pipe hazard (e.g. MTE3 tstore vs a V-pipe write into an + // overlapping region) is not silently dropped (issue #934). + bool crossRootOverlap = isLocalBufferOverlapCrossRoot(a, b); + if (isTraceEnabled()) + llvm::errs() << " -> Cross-root overlap check: " + << (crossRootOverlap ? "true" : "false") << "\n"; + return crossRootOverlap; } bool MemoryDependentAnalyzer::isGMBufferOverlap(const BaseMemInfo *a, diff --git a/lib/PTO/Transforms/InsertSync/PTOIRTranslator.cpp b/lib/PTO/Transforms/InsertSync/PTOIRTranslator.cpp index 7d19f50db0..dd8355fccc 100644 --- a/lib/PTO/Transforms/InsertSync/PTOIRTranslator.cpp +++ b/lib/PTO/Transforms/InsertSync/PTOIRTranslator.cpp @@ -12,6 +12,7 @@ // See LICENSE in the root of the software repository for the full text of the License. #include "PTO/Transforms/InsertSync/PTOIRTranslator.h" +#include "PTO/IR/PTOMultiBuffer.h" #include "PTO/IR/PTOTypeUtils.h" #include "PTO/Transforms/InsertSync/SyncMacroModel.h" #include "mlir/Dialect/Arith/IR/Arith.h" @@ -363,12 +364,21 @@ void PTOIRTranslator::RecursionIR(Region *region) { return WalkResult::interrupt(); } } + else if (auto allocMultiOp = dyn_cast(op)) { + if (failed(UpdateAllocMultiTileOpMemInfo(allocMultiOp))) { + return WalkResult::interrupt(); + } + } // 支持标准 memref.alloc else if (auto memAllocOp = dyn_cast(op)) { if (failed(UpdateMemrefAllocOpMemInfo(memAllocOp))) { return WalkResult::interrupt(); } } + else if (auto declareOp = dyn_cast(op)) { + if (failed(UpdateDeclareTileOpMemInfo(declareOp))) + return WalkResult::interrupt(); + } else if (auto declareOp = dyn_cast(op)) { if (failed(UpdateDeclareTileMemRefOpMemInfo(declareOp))) { return WalkResult::interrupt(); @@ -402,6 +412,15 @@ void PTOIRTranslator::RecursionIR(Region *region) { else if (auto slotMarker = dyn_cast(op)) { UpdateSlotMarkerAliasBufferInfo(slotMarker); } + else if (auto multiGet = dyn_cast(op)) { + UpdateMultiTileGetAliasBufferInfo(multiGet); + } + else if (auto reshape = dyn_cast(op)) { + UpdateAliasBufferInfo(reshape.getResult(), reshape.getSrc()); + } + else if (auto bitcast = dyn_cast(op)) { + UpdateAliasBufferInfo(bitcast.getResult(), bitcast.getSrc()); + } else if (auto castOp = dyn_cast(op)) { UpdateAliasBufferInfo(castOp.getResult(), castOp.getSource()); } @@ -517,6 +536,55 @@ LogicalResult PTOIRTranslator::UpdateAllocTileOpMemInfo(pto::AllocTileOp op) { return success(); } +LogicalResult +PTOIRTranslator::UpdateAllocMultiTileOpMemInfo(pto::AllocMultiTileOp op) { + Value result = op.getResult(); + auto multiType = op.getResult().getType(); + pto::TileBufType slotType = multiType.getSlotType(); + + uint64_t slotBytes = pto::getPTOStorageElemByteSize(slotType.getElementType()); + if (slotBytes == 0) + return failure(); + for (int64_t dim : slotType.getShape()) { + if (dim == ShapedType::kDynamic) + return op.emitError("requires a static slot shape for sync analysis"); + slotBytes *= static_cast(dim); + } + + pto::AddressSpace space = pto::AddressSpace::MAT; + if (auto attr = dyn_cast_or_null( + slotType.getMemorySpace())) + space = attr.getAddressSpace(); + + SmallVector addresses; + bool hasKnownAddresses = false; + if (auto planned = op->getAttrOfType( + pto::kPtoMultiBufferAddrsAttrName)) { + if (planned.size() != multiType.getCount()) + return op.emitError("planned address count does not match slot count"); + for (int64_t address : planned.asArrayRef()) + addresses.push_back(static_cast(address)); + hasKnownAddresses = true; + } else if (Value base = op.getAddr()) { + if (std::optional knownBase = getKnownPhysicalAddress(base)) { + for (uint32_t slot = 0; slot < multiType.getCount(); ++slot) + addresses.push_back(*knownBase + slot * slotBytes); + hasKnownAddresses = true; + } + } + + if (addresses.empty()) { + return op.emitError( + "requires planner-assigned slot addresses or a constant level3 base"); + } + + auto info = std::make_unique( + result, result, space, std::move(addresses), slotBytes, + hasKnownAddresses && isLocalAddressSpace(space)); + buffer2MemInfoMap_[result].emplace_back(info->clone()); + return success(); +} + LogicalResult PTOIRTranslator::UpdatePointerCastOpMemInfo(pto::PointerCastOp op) { Value res = op.getResult(); auto memRefType = dyn_cast(res.getType()); @@ -597,6 +665,36 @@ LogicalResult PTOIRTranslator::UpdatePointerCastOpMemInfo(pto::PointerCastOp op) return success(); } +LogicalResult +PTOIRTranslator::UpdateDeclareTileOpMemInfo(pto::DeclareTileOp op) { + Value result = op.getTile(); + auto tileType = dyn_cast(result.getType()); + if (!tileType) + return op.emitError("requires a tile_buf result for sync analysis"); + + uint64_t sizeInBytes = pto::getPTOStorageElemByteSize( + tileType.getElementType()); + if (sizeInBytes == 0) + return failure(); + for (int64_t dim : tileType.getShape()) { + if (dim == ShapedType::kDynamic) { + sizeInBytes = 0; + break; + } + sizeInBytes *= static_cast(dim); + } + + pto::AddressSpace space = pto::AddressSpace::MAT; + if (auto attr = dyn_cast_or_null( + tileType.getMemorySpace())) + space = attr.getAddressSpace(); + + auto info = std::make_unique( + result, result, space, SmallVector{0}, sizeInBytes); + buffer2MemInfoMap_[result].emplace_back(info->clone()); + return success(); +} + LogicalResult PTOIRTranslator::UpdateDeclareTileMemRefOpMemInfo(pto::DeclareTileMemRefOp op) { Value res = op.getResult(); @@ -986,14 +1084,24 @@ void PTOIRTranslator::UpdateConservativeAliasBufferInfo(Value result, } void PTOIRTranslator::UpdateSlotMarkerAliasBufferInfo(pto::SlotMarkerOp op) { - Value result = op.getResult(); - Value source = op.getSource(); + UpdateSlotSelectedAliasBufferInfo(op.getResult(), op.getSource(), + op.getSlot()); +} + +void PTOIRTranslator::UpdateMultiTileGetAliasBufferInfo( + pto::MultiTileGetOp op) { + UpdateSlotSelectedAliasBufferInfo(op.getResult(), op.getSource(), + op.getSlot()); +} + +void PTOIRTranslator::UpdateSlotSelectedAliasBufferInfo(Value result, + Value source, + Value slot) { if (!result || !source) return; if (!buffer2MemInfoMap_.contains(source)) return; - Value slot = op.getSlot(); IntegerAttr constAttr; bool isConstSlot = matchPattern(slot, m_Constant(&constAttr)); int64_t constSlotIdx = isConstSlot ? constAttr.getValue().getSExtValue() : -1; diff --git a/lib/PTO/Transforms/PTOMaterializeTileHandles.cpp b/lib/PTO/Transforms/PTOMaterializeTileHandles.cpp index 10cf9da2bd..2a4c4bf1d1 100644 --- a/lib/PTO/Transforms/PTOMaterializeTileHandles.cpp +++ b/lib/PTO/Transforms/PTOMaterializeTileHandles.cpp @@ -508,8 +508,50 @@ static Value computeSubviewAddress(memref::SubViewOp subview, return builder.create(loc, base, linearOffset); } +static Value computePTOSubViewAddress(pto::SubViewOp subview, + OpBuilder &builder, Location loc) { + Value base = computeExplicitAddress(subview.getSource(), builder, loc); + if (!base) + return Value(); + + auto sourceTy = dyn_cast(subview.getSource().getType()); + if (!sourceTy || sourceTy.getShape().size() < 2) + return Value(); + + unsigned elemBytes = getPTOStorageElemByteSize(sourceTy.getElementType()); + if (elemBytes == 0) + return Value(); + + int64_t rowStride = 0; + int64_t colStride = 0; + if (!getTilePointerStrides(sourceTy.getConfigAttr(), sourceTy.getElementType(), + sourceTy.getShape()[0], sourceTy.getShape()[1], + rowStride, colStride)) + return Value(); + + SmallVector strides = {rowStride, colStride}; + Value linearOffset; + for (auto [offset, stride] : + llvm::zip(subview.getOffsets(), ArrayRef(strides))) { + Value offsetI64 = ensureI64(offset, builder, loc); + if (!offsetI64) + return Value(); + linearOffset = + addI64(linearOffset, mulI64(offsetI64, stride, builder, loc), builder, + loc); + } + + if (!linearOffset) + return base; + linearOffset = mulI64(linearOffset, elemBytes, builder, loc); + return builder.create(loc, base, linearOffset); +} + static Value computeExplicitAddress(Value value, OpBuilder &builder, Location loc) { + if (auto alloc = value.getDefiningOp()) + return ensureI64(alloc.getAddr(), builder, loc); + if (auto bind = value.getDefiningOp()) return computeExplicitAddress(bind.getSource(), builder, loc); @@ -522,6 +564,9 @@ static Value computeExplicitAddress(Value value, OpBuilder &builder, if (auto subview = value.getDefiningOp()) return computeSubviewAddress(subview, builder, loc); + if (auto subview = value.getDefiningOp()) + return computePTOSubViewAddress(subview, builder, loc); + if (auto select = value.getDefiningOp()) { Value trueAddr = computeExplicitAddress(select.getTrueValue(), builder, loc); Value falseAddr = @@ -535,6 +580,11 @@ static Value computeExplicitAddress(Value value, OpBuilder &builder, if (auto cast = value.getDefiningOp()) return computeExplicitAddress(cast.getSource(), builder, loc); + if (auto cast = value.getDefiningOp()) { + if (cast.getNumOperands() == 1) + return computeExplicitAddress(cast.getOperand(0), builder, loc); + } + return Value(); } @@ -1414,6 +1464,20 @@ struct PTOMaterializeTileHandlesPass ModuleOp module = getOperation(); MLIRContext *ctx = module.getContext(); + bool hasUnresolvedMultiTile = false; + module.walk([&](Operation *op) { + if (!isa(op)) + return; + op->emitError( + "must be resolved by pto-resolve-buffer-select before " + "pto-materialize-tile-handles"); + hasUnresolvedMultiTile = true; + }); + if (hasUnresolvedMultiTile) { + signalPassFailure(); + return; + } + OpBuilder builder(ctx); DenseMap tileHandles; DenseSet mustMaterialize; diff --git a/lib/PTO/Transforms/PTOPlanMemory.cpp b/lib/PTO/Transforms/PTOPlanMemory.cpp index 9d672de706..cfd58e3481 100644 --- a/lib/PTO/Transforms/PTOPlanMemory.cpp +++ b/lib/PTO/Transforms/PTOPlanMemory.cpp @@ -13,6 +13,7 @@ #include "PTO/IR/PTOMultiBuffer.h" #include "PTO/IR/PTOTypeUtils.h" +#include "Utils.h" #include "mlir/Dialect/Arith/IR/Arith.h" #include "mlir/Dialect/GPU/IR/GPUDialect.h" @@ -93,6 +94,62 @@ static LocalMemSpec getLocalMemSpec(Operation *op, AddressSpace as) { } } +static bool isNameIn(StringRef name, ArrayRef names) { + return llvm::is_contained(names, name); +} + +static bool isIgnoredA5TmpOperandUse(OpOperand &use) { + Operation *owner = use.getOwner(); + unsigned operandNo = use.getOperandNumber(); + StringRef name = owner->getName().getStringRef(); + + if (auto dpsOp = dyn_cast(owner)) { + if (llvm::is_contained(dpsOp.getDpsInits(), use.get())) + return false; + } else if (auto dpsOp = dyn_cast(owner)) { + if (llvm::is_contained(dpsOp.getDpsInits(), use.get())) + return false; + } + + if (isNameIn(name, {"pto.trowargmax", "pto.trowargmin", "pto.trowmax", + "pto.trowmin", "pto.trowsum", "pto.trowprod"})) + return operandNo == 1; + + if (name == "pto.txors") + return operandNo == 2; + + if (isNameIn(name, {"pto.tprelu", "pto.txor", "pto.tsels", + "pto.trowexpand", "pto.tcolexpand", + "pto.trowexpandadd", "pto.trowexpanddiv", + "pto.trowexpandexpdif", "pto.trowexpandmax", + "pto.trowexpandmin", "pto.trowexpandmul", + "pto.trowexpandsub", "pto.tcolexpandadd", + "pto.tcolexpanddiv", "pto.tcolexpandexpdif", + "pto.tcolexpandmax", "pto.tcolexpandmin", + "pto.tcolexpandmul", "pto.tcolexpandsub"})) + return operandNo == 2; + + if (name == "pto.tsel") + return operandNo == 3; + + return false; +} + +static bool isA5IgnoredTmpAlloc(pto::AllocTileOp allocTile) { + if (getTargetArch(allocTile.getOperation()) != PTOArch::A5) + return false; + + Value value = allocTile.getResult(); + if (value.use_empty()) + return false; + + for (OpOperand &use : value.getUses()) { + if (!isIgnoredA5TmpOperandUse(use)) + return false; + } + return true; +} + static void collectStableValueOrder(Region ®ion, AsmState &asmState, DenseMap &stableValueKeys, @@ -407,10 +464,8 @@ void MemLivenessAnalysis::RecursionIR(Region *region, Liveness live) { if (mayAliasOp.has_value()) { auto aliasPair = mayAliasOp.value(); UpdateBufferAlias(aliasPair.first, aliasPair.second); - } else if (isa(op)) { - // Internal placeholder for a tile whose runtime address is assigned by - // pipe operations such as tpop. This op does not allocate local storage - // and should not participate in memory planning. + } else if (isa(op)) { + // Runtime-bound tile handles do not allocate static local storage. return WalkResult::advance(); } else if (auto bindOp = dyn_cast(op)) { // BindTile result is only an alias of the source buffer. Treat every use @@ -424,6 +479,46 @@ void MemLivenessAnalysis::RecursionIR(Region *region, Liveness live) { // op and is consumed later by PTOResolveBufferSelect / sync. UpdateBufferAlias(slotOp.getResult(), slotOp.getSource()); return WalkResult::advance(); + } else if (isLocalMemPlan() && dyn_cast(op)) { + auto allocTileOp = cast(op); + if (allocTileOp.getAddr()) { + return WalkResult::advance(); + } + if (isA5IgnoredTmpAlloc(allocTileOp)) + return WalkResult::advance(); + auto memorySpaceAttr = GetBufferSpaceAttr(allocTileOp.getResult()); + if (!isLocalBuffer(memorySpaceAttr)) { + allocTileOp.emitError("Alloc tile buffer not at local space"); + return WalkResult::interrupt(); + } + if (auto attr = allocTileOp->getAttrOfType( + mlir::pto::kPtoMultiBufferAttrName)) { + uint64_t n = attr.getValue().getZExtValue(); + if (n < mlir::pto::kPtoMultiBufferMinNum || + n > mlir::pto::kPtoMultiBufferMaxNum) { + allocTileOp.emitError() + << "pto.multi_buffer must be in [" + << mlir::pto::kPtoMultiBufferMinNum << ", " + << mlir::pto::kPtoMultiBufferMaxNum << "] (got " << n << ")"; + return WalkResult::interrupt(); + } + buffer2MultiNum[allocTileOp.getResult()] = static_cast(n); + } + UpdateOpBufferInfo(op, op->getResults()); + return WalkResult::advance(); + } else if (isLocalMemPlan() && dyn_cast(op)) { + auto allocMultiOp = cast(op); + if (allocMultiOp.getAddr()) + return WalkResult::advance(); + auto memorySpaceAttr = GetBufferSpaceAttr(allocMultiOp.getResult()); + if (!isLocalBuffer(memorySpaceAttr)) { + allocMultiOp.emitError("Alloc multi tile buffer not at local space"); + return WalkResult::interrupt(); + } + uint32_t count = allocMultiOp.getResult().getType().getCount(); + buffer2MultiNum[allocMultiOp.getResult()] = count; + UpdateOpBufferInfo(op, op->getResults()); + return WalkResult::advance(); } else if (isLocalMemPlan() && dyn_cast(op)) { auto allocOp = cast(op); if (failed(CheckLocalBufferAllocOp(op))) { @@ -453,6 +548,7 @@ void MemLivenessAnalysis::RecursionIR(Region *region, Liveness live) { OpKillHandle(curOpInfo, live, op->getBlock()); } else if (auto tprintOp = dyn_cast(op)) { // TPrintOp only reads from buffer, similar to LoadOp + UpdateOpGenInfo(curOpInfo, llvm::to_vector(op->getOperands())); OpKillHandle(curOpInfo, live, op->getBlock()); } else if (auto tgetvalOp = dyn_cast(op)) { (void)tgetvalOp; @@ -476,7 +572,7 @@ void MemLivenessAnalysis::RecursionIR(Region *region, Liveness live) { } else if (auto ptoDpsOp = dyn_cast(op)) { // PTO ops with destination (tile_buf, partition_view, etc.); no // tensor/memref-only verification. - SmallVector genBuffers = llvm::to_vector(ptoDpsOp.getDpsInits()); + SmallVector genBuffers = llvm::to_vector(op->getOperands()); auto scratchBuffers = getScratchBuffersFromEffects( op, ptoDpsOp.getDpsInits(), stableValueOrder); genBuffers.append(scratchBuffers.begin(), scratchBuffers.end()); @@ -499,6 +595,9 @@ void MemLivenessAnalysis::RecursionIR(Region *region, Liveness live) { UpdateBufferAlias(selectOp.getResult(), selectOp.getTrueValue(), true); UpdateBufferAlias(selectOp.getResult(), selectOp.getFalseValue(), true); OpKillHandle(curOpInfo, live, op->getBlock()); + } else if (auto tileBufAddrOp = dyn_cast(op)) { + UpdateOpGenInfo(curOpInfo, ValueRange{tileBufAddrOp.getSrc()}); + OpKillHandle(curOpInfo, live, op->getBlock()); } else if (auto callOp = dyn_cast(op)) { UpdateOpGenInfo(curOpInfo, llvm::to_vector(callOp->getOperands())); OpKillHandle(curOpInfo, live, op->getBlock()); @@ -914,14 +1013,26 @@ BufferInfo MemLivenessAnalysis::GetBufferInfo(Operation *op, Value operand, bufferInfo.operation = op; bufferInfo.bufferScope = bufferScope; // get buffer size, now for static shape - Value traceValue = tracebackMemRef(operand); - auto memRefType = cast(traceValue.getType()); - bufferInfo.bufferType = memRefType.getElementType(); + ArrayRef shape; + Type elementType; + if (auto tileType = dyn_cast(operand.getType())) { + shape = tileType.getShape(); + elementType = tileType.getElementType(); + } else if (auto multiType = dyn_cast(operand.getType())) { + shape = multiType.getSlotType().getShape(); + elementType = multiType.getSlotType().getElementType(); + } else { + Value traceValue = tracebackMemRef(operand); + auto memRefType = cast(traceValue.getType()); + shape = memRefType.getShape(); + elementType = memRefType.getElementType(); + } + bufferInfo.bufferType = elementType; std::optional totalStaticSize = - getStaticTotalSize(memRefType.getShape()); + getStaticTotalSize(shape); if (!totalStaticSize.has_value()) llvm::report_fatal_error("failed to obtain buffer static shape size"); - unsigned elemBytes = getPTOStorageElemByteSize(memRefType.getElementType()); + unsigned elemBytes = getPTOStorageElemByteSize(elementType); if (elemBytes == 0) llvm::report_fatal_error("failed to obtain buffer element byte size"); bufferInfo.constBits = @@ -2423,8 +2534,105 @@ PlanRecord MemPlan::RollbackOutline(PlanRecHis &history, } namespace { + +class LegacyAllocTileOpAddPlannedAddressPattern + : public OpRewritePattern { +public: + explicit LegacyAllocTileOpAddPlannedAddressPattern( + MLIRContext *context, + DenseMap> buffer2Offsets) + : OpRewritePattern(context), + buffer2Offsets(std::move(buffer2Offsets)) {} + + LogicalResult matchAndRewrite(pto::AllocTileOp op, + PatternRewriter &rewriter) const override { + if (op.getAddr()) + return failure(); + + auto tileType = dyn_cast(op.getResult().getType()); + if (!tileType) + return failure(); + + auto it = buffer2Offsets.find(op.getResult()); + if (it == buffer2Offsets.end() || it->second.empty()) + return failure(); + + if (it->second.size() != 1) { + return rewriter.notifyMatchFailure( + op, "single alloc_tile root expects exactly one planned address"); + } + + Value addr = rewriter.create(op.getLoc(), + it->second.front(), 64); + auto planned = rewriter.create( + op.getLoc(), tileType, addr, + op.getValidRow() ? op.getValidRow() : Value(), + op.getValidCol() ? op.getValidCol() : Value()); + for (NamedAttribute attr : op->getAttrs()) { + if (attr.getName().getValue() == "operandSegmentSizes") + continue; + planned->setAttr(attr.getName(), attr.getValue()); + } + + rewriter.replaceOp(op, planned.getResult()); + return success(); + } + +private: + DenseMap> buffer2Offsets; +}; + +class LegacyAllocMultiTileOpAddPlannedAddressesPattern + : public OpRewritePattern { +public: + explicit LegacyAllocMultiTileOpAddPlannedAddressesPattern( + MLIRContext *context, + DenseMap> buffer2Offsets) + : OpRewritePattern(context), + buffer2Offsets(std::move(buffer2Offsets)) {} + + LogicalResult matchAndRewrite(pto::AllocMultiTileOp op, + PatternRewriter &rewriter) const override { + if (op.getAddr() || op->hasAttr(pto::kPtoMultiBufferAddrsAttrName)) + return failure(); + auto it = buffer2Offsets.find(op.getResult()); + if (it == buffer2Offsets.end() || it->second.empty()) + return failure(); + if (it->second.size() != op.getResult().getType().getCount()) { + return rewriter.notifyMatchFailure( + op, "planned address count does not match multi_tile_buf count"); + } + + SmallVector addrs; + addrs.reserve(it->second.size()); + for (uint64_t offset : it->second) + addrs.push_back(static_cast(offset)); + rewriter.modifyOpInPlace(op, [&] { + op->setAttr(pto::kPtoMultiBufferAddrsAttrName, + rewriter.getDenseI64ArrayAttr(addrs)); + }); + return success(); + } + +private: + DenseMap> buffer2Offsets; +}; + +static FailureOr parseLegacyMemPlanMode(func::FuncOp func, + llvm::StringRef memMode) { + if (memMode.equals_insensitive("local") || + memMode.equals_insensitive("local-mem-plan")) + return MemPlanMode::LOCAL_MEM_PLAN; + if (memMode.equals_insensitive("global-work-space-plan")) + return MemPlanMode::GLOBAL_WORKSPACE_PLAN; + func.emitError("unsupported mem-mode '") + << memMode << "'; only 'local' is supported by the PTOAS pipeline"; + return failure(); +} + struct PlanMemoryPass : public mlir::pto::impl::PlanMemoryBase { public: + PlanMemoryPass() = default; explicit PlanMemoryPass(const mlir::pto::PlanMemoryOptions &planMemoryOption) : PlanMemoryBase(planMemoryOption) {} @@ -2432,11 +2640,15 @@ struct PlanMemoryPass : public mlir::pto::impl::PlanMemoryBase { private: void populateBufferAddressToAllocOp( - RewritePatternSet &patterns, + RewritePatternSet &patterns, MemPlanMode mode, DenseMap> buffer2Offsets) { - if (this->memMode == MemPlanMode::LOCAL_MEM_PLAN) { + if (mode == MemPlanMode::LOCAL_MEM_PLAN) { patterns.add(patterns.getContext(), buffer2Offsets); + patterns.add( + patterns.getContext(), buffer2Offsets); + patterns.add( + patterns.getContext(), buffer2Offsets); } } }; @@ -2445,12 +2657,17 @@ struct PlanMemoryPass : public mlir::pto::impl::PlanMemoryBase { void PlanMemoryPass::runOnOperation() { ModuleOp moduleOp = getOperation(); for (auto funcOp : moduleOp.getOps()) { + auto parsedMode = parseLegacyMemPlanMode(funcOp, this->memMode); + if (failed(parsedMode)) + return signalPassFailure(); + MemPlanMode mode = *parsedMode; + ReserveBufferPlans reservePlans; - if (this->memMode == MemPlanMode::LOCAL_MEM_PLAN && + if (mode == MemPlanMode::LOCAL_MEM_PLAN && failed(analyzeReserveBufferPlans(funcOp, reservePlans))) { return signalPassFailure(); } - if (this->memMode == MemPlanMode::LOCAL_MEM_PLAN) { + if (mode == MemPlanMode::LOCAL_MEM_PLAN) { for (ReserveBufferPlan &reservePlan : reservePlans) { if (reservePlan.mode != ReserveBufferMode::Manual) continue; @@ -2461,12 +2678,14 @@ void PlanMemoryPass::runOnOperation() { } } - MemLivenessAnalysis memLiveness(funcOp, this->memMode); + MemLivenessAnalysis memLiveness(funcOp, mode); memLiveness.build(); - MemPlan memPlan(this->memMode, this->enableGlobalReuse, - this->enablePrintMemoryAllocatedSize, - this->restrictInplaceAsISA, this->orderBySize); + constexpr bool enableGlobalReuse = false; + constexpr bool enablePrintMemoryAllocatedSize = false; + constexpr bool restrictInplaceAsISA = false; + MemPlan memPlan(mode, enableGlobalReuse, enablePrintMemoryAllocatedSize, + restrictInplaceAsISA, this->orderBySize); if (failed(memPlan.InitMemSpecsFromModule(funcOp))) { return signalPassFailure(); } @@ -2485,21 +2704,36 @@ void PlanMemoryPass::runOnOperation() { // Keep reserve_buffer allocation outside the core MemPlan algorithm: // normal local buffers are planned first, then reserve_buffer claims one // aligned hole in its target address space. - if (this->memMode == MemPlanMode::LOCAL_MEM_PLAN && - failed(assignAutoReserveBufferBases(reservePlans, memLiveness.bufferInfos, - memPlan.GetBuffer2Offsets()))) { + if (mode == MemPlanMode::LOCAL_MEM_PLAN && + failed(assignAutoReserveBufferBases( + reservePlans, memLiveness.bufferInfos, memPlan.GetBuffer2Offsets()))) { return signalPassFailure(); } RewritePatternSet patterns(&getContext()); - populateBufferAddressToAllocOp(patterns, memPlan.GetBuffer2Offsets()); + populateBufferAddressToAllocOp(patterns, mode, memPlan.GetBuffer2Offsets()); if (failed(applyPatternsGreedily(funcOp, std::move(patterns)))) { return signalPassFailure(); } + + bool hasUnplannedAllocTile = false; + funcOp.walk([&](pto::AllocTileOp op) { + if (op.getAddr()) + return; + if (op->use_empty()) + return; + if (isA5IgnoredTmpAlloc(op)) + return; + op.emitError( + "PTOPlanMemory failed to assign an address to pto.alloc_tile"); + hasUnplannedAllocTile = true; + }); + if (hasUnplannedAllocTile) + return signalPassFailure(); } } std::unique_ptr -mlir::pto::createPlanMemoryPass(const PlanMemoryOptions &planMemoryOption) { - return std::make_unique(planMemoryOption); +mlir::pto::createPlanMemoryPass(const PlanMemoryOptions &options) { + return std::make_unique(options); } diff --git a/lib/PTO/Transforms/PTOPlanMemory.h b/lib/PTO/Transforms/PTOPlanMemory.h index 81f362be25..d6e2a37381 100644 --- a/lib/PTO/Transforms/PTOPlanMemory.h +++ b/lib/PTO/Transforms/PTOPlanMemory.h @@ -43,6 +43,11 @@ enum BufferStatus { UNDEFFINED = 0, DEFFINED, GENED, KILLED }; /// Pair of inplace Value. using ValuePair = std::pair; +enum class MemPlanMode { + LOCAL_MEM_PLAN, + GLOBAL_WORKSPACE_PLAN, +}; + /// Result status after plan memory. enum PlanStatus { PLAN_SUCCESS = 0, diff --git a/lib/PTO/Transforms/PTOPlanMemoryModern.cpp b/lib/PTO/Transforms/PTOPlanMemoryModern.cpp new file mode 100644 index 0000000000..2fa4d9882d --- /dev/null +++ b/lib/PTO/Transforms/PTOPlanMemoryModern.cpp @@ -0,0 +1,1233 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===- PTOPlanMemoryModern.cpp - modern static local memory planner -------===// + +#include "AllocToPointerCast.h" +#include "PTO/IR/PTOMultiBuffer.h" +#include "PTO/IR/PTOTypeUtils.h" +#include "PTO/Transforms/Passes.h" +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/Interfaces/SideEffectInterfaces.h" +#include "mlir/Transforms/GreedyPatternRewriteDriver.h" +#include "llvm/ADT/DenseSet.h" +#include "llvm/ADT/MapVector.h" +#include "llvm/ADT/STLExtras.h" + +using namespace mlir; +using namespace mlir::pto; + +namespace mlir::pto { +LogicalResult runModernPlanMemory(func::FuncOp func, llvm::StringRef memMode, + bool orderBySize); +} // namespace mlir::pto + +namespace { + +struct MemSpec { + uint64_t capacityBytes = 0; + uint64_t alignmentBytes = 1; +}; + +struct RootInfo { + Value root; + Operation *defOp = nullptr; + AddressSpace space = AddressSpace::Zero; + uint64_t slotBytes = 0; + uint64_t totalBytes = 0; + uint64_t alignmentBytes = 1; + uint64_t slotCount = 1; + unsigned allocIndex = 0; + unsigned freeIndex = 0; + unsigned stableOrder = 0; + bool hasWriter = false; + bool hasUseBeforeFirstWrite = false; + SmallVector offsets; +}; + +using RootList = SmallVector; + +struct ConflictFacts { + DenseMap forbidAlias; + DenseSet loadDerivedRoots; + DenseSet tpopConsumerRoots; + DenseMap> tpopConsumerWriteIndices; + DenseMap> phiFamilyIds; + bool targetHazardEnabled = false; +}; + +struct InplacePolicy { + bool notInplaceSafe = false; + SmallVector forbidOutputAliasOperands; +}; + +struct ReuseGroup { + AddressSpace space = AddressSpace::Zero; + uint64_t sizeBytes = 0; + uint64_t alignmentBytes = 1; + uint64_t offsetBytes = 0; + SmallVector members; +}; + +static uint64_t alignUp(uint64_t value, uint64_t align) { + if (align == 0) + return value; + return ((value + align - 1) / align) * align; +} + +static std::optional getBufferAddressSpace(Type type) { + if (auto tileType = dyn_cast(type)) { + if (auto attr = + dyn_cast_or_null(tileType.getMemorySpace())) + return attr.getAddressSpace(); + return std::nullopt; + } + + if (auto multiType = dyn_cast(type)) + return getBufferAddressSpace(multiType.getSlotType()); + + if (auto memRefType = dyn_cast(type)) { + if (auto attr = + dyn_cast_or_null(memRefType.getMemorySpace())) + return attr.getAddressSpace(); + } + return std::nullopt; +} + +static bool isPlannableLocalSpace(std::optional space) { + return space && *space != AddressSpace::GM && *space != AddressSpace::Zero; +} + +static bool isNameIn(StringRef name, ArrayRef names) { + return llvm::is_contained(names, name); +} + +static bool isIgnoredA5TmpOperandUse(OpOperand &use) { + Operation *owner = use.getOwner(); + unsigned operandNo = use.getOperandNumber(); + StringRef name = owner->getName().getStringRef(); + + if (auto dpsOp = dyn_cast(owner)) { + if (llvm::is_contained(dpsOp.getDpsInits(), use.get())) + return false; + } else if (auto dpsOp = dyn_cast(owner)) { + if (llvm::is_contained(dpsOp.getDpsInits(), use.get())) + return false; + } + + if (isNameIn(name, {"pto.trowargmax", "pto.trowargmin", "pto.trowmax", + "pto.trowmin", "pto.trowsum", "pto.trowprod"})) + return operandNo == 1; + + if (name == "pto.txors") + return operandNo == 2; + + if (isNameIn(name, {"pto.tprelu", "pto.txor", "pto.tsels", + "pto.trowexpand", "pto.tcolexpand", + "pto.trowexpandadd", "pto.trowexpanddiv", + "pto.trowexpandexpdif", "pto.trowexpandmax", + "pto.trowexpandmin", "pto.trowexpandmul", + "pto.trowexpandsub", "pto.tcolexpandadd", + "pto.tcolexpanddiv", "pto.tcolexpandexpdif", + "pto.tcolexpandmax", "pto.tcolexpandmin", + "pto.tcolexpandmul", "pto.tcolexpandsub"})) + return operandNo == 2; + + if (name == "pto.tsel") + return operandNo == 3; + + return false; +} + +static bool isA5IgnoredTmpAlloc(pto::AllocTileOp allocTile) { + if (getTargetArch(allocTile.getOperation()) != PTOArch::A5) + return false; + + Value value = allocTile.getResult(); + if (value.use_empty()) + return false; + + for (OpOperand &use : value.getUses()) { + if (!isIgnoredA5TmpOperandUse(use)) + return false; + } + return true; +} + +static MemSpec getMemSpec(PTOArch arch, AddressSpace space) { + switch (space) { + case AddressSpace::VEC: + return {arch == PTOArch::A5 ? 253952ull : 196608ull, 256}; + case AddressSpace::MAT: + return {524288ull, 256}; + case AddressSpace::LEFT: + case AddressSpace::RIGHT: + return {65536ull, 4096}; + case AddressSpace::ACC: + return {arch == PTOArch::A5 ? 262144ull : 131072ull, 4096}; + case AddressSpace::BIAS: + case AddressSpace::SCALING: + return {524288ull, 256}; + case AddressSpace::GM: + case AddressSpace::Zero: + break; + } + return {}; +} + +static FailureOr computeStaticBufferBytes(Value value) { + ArrayRef shape; + Type elementType; + + if (auto tileType = dyn_cast(value.getType())) { + shape = tileType.getShape(); + elementType = tileType.getElementType(); + } else if (auto multiType = dyn_cast(value.getType())) { + shape = multiType.getSlotType().getShape(); + elementType = multiType.getSlotType().getElementType(); + } else if (auto memRefType = dyn_cast(value.getType())) { + shape = memRefType.getShape(); + elementType = memRefType.getElementType(); + } else { + return failure(); + } + + uint64_t elemBytes = getPTOStorageElemByteSize(elementType); + if (elemBytes == 0) + return failure(); + + uint64_t numel = 1; + for (int64_t dim : shape) { + if (dim == ShapedType::kDynamic) + return failure(); + numel *= static_cast(dim); + } + return numel * elemBytes; +} + +static void appendUniqueRoot(RootList &roots, Value root) { + if (llvm::is_contained(roots, root)) + return; + roots.push_back(root); +} + +static RootList unionRoots(const RootList &lhs, const RootList &rhs) { + RootList result = lhs; + for (Value root : rhs) + appendUniqueRoot(result, root); + return result; +} + +static bool isOneOf(StringRef name, ArrayRef names) { + return llvm::is_contained(names, name); +} + +static InplacePolicy getInplacePolicy(Operation *op) { + StringRef name = op->getName().getStringRef(); + InplacePolicy policy; + + policy.notInplaceSafe = isOneOf( + name, { + "pto.tands", "pto.tfillpad_expand", "pto.tfmod", + "pto.tfmods", "pto.tgather", "pto.tmrgsort", + "pto.tors", "pto.trecip", "pto.trsqrt", + "pto.tsort32", "pto.ttrans", "pto.trowargmax", + "pto.trowargmin", "pto.trowmax", "pto.trowmin", + "pto.trowprod", "pto.trowsum", "pto.tcolargmax", + "pto.tcolargmin", "pto.txors", + }); + + if (name == "pto.tsel") { + policy.forbidOutputAliasOperands.push_back(0); // mask + policy.forbidOutputAliasOperands.push_back(3); // tmp + } + + if (isOneOf(name, { + "pto.trowexpand", + "pto.tcolexpand", + })) { + policy.forbidOutputAliasOperands.push_back(0); + } + + if (isOneOf(name, { + "pto.trowexpandadd", + "pto.trowexpanddiv", + "pto.trowexpandexpdif", + "pto.trowexpandmax", + "pto.trowexpandmin", + "pto.trowexpandmul", + "pto.trowexpandsub", + "pto.tcolexpandadd", + "pto.tcolexpanddiv", + "pto.tcolexpandexpdif", + "pto.tcolexpandmax", + "pto.tcolexpandmin", + "pto.tcolexpandmul", + "pto.tcolexpandsub", + })) { + policy.forbidOutputAliasOperands.push_back(1); + } + + return policy; +} + +static SmallVector getWrittenNonDpsOperands(Operation *op, + ValueRange dpsInits) { + SmallVector scratchOperands; + auto memEffect = dyn_cast(op); + if (!memEffect) + return scratchOperands; + + SmallVector, 8> effects; + memEffect.getEffects(effects); + for (const auto &effect : effects) { + if (!isa(effect.getEffect())) + continue; + Value value = effect.getValue(); + if (!value) + continue; + if (!llvm::is_contained(op->getOperands(), value)) + continue; + if (llvm::is_contained(dpsInits, value)) + continue; + if (!llvm::is_contained(scratchOperands, value)) + scratchOperands.push_back(value); + } + return scratchOperands; +} + +static bool hasReadEffectOnValue(Operation *op, Value value) { + auto memEffect = dyn_cast(op); + if (!memEffect) + return false; + + SmallVector, 8> effects; + memEffect.getEffects(effects); + for (const auto &effect : effects) { + if (!isa(effect.getEffect())) + continue; + if (effect.getValue() == value) + return true; + } + return false; +} + +struct PlannerAnalysis { + func::FuncOp func; + DenseMap valueToRoots; + DenseSet splitTpopDerivedValues; + SmallVector linearOps; + DenseMap opToIndex; + SmallVector roots; + DenseMap rootIndexByValue; + ConflictFacts facts; + unsigned nextPhiFamilyId = 0; + bool failed = false; + + explicit PlannerAnalysis(func::FuncOp func) : func(func) { + facts.targetHazardEnabled = + getTargetArch(func.getOperation()) == PTOArch::A3; + } + + RootList getRoots(Value value) const { + auto it = valueToRoots.find(value); + if (it == valueToRoots.end()) + return {}; + return it->second; + } + + void setRoots(Value value, const RootList &roots) { + if (roots.empty()) + return; + valueToRoots[value] = roots; + } + + void addRoot(Value value, Operation *defOp) { + if (rootIndexByValue.count(value)) + return; + + auto space = getBufferAddressSpace(value.getType()); + if (!isPlannableLocalSpace(space)) + return; + + auto bytesOr = computeStaticBufferBytes(value); + if (mlir::failed(bytesOr)) { + defOp->emitError("requires a static local buffer shape and known element " + "byte size for memory planning"); + failed = true; + return; + } + + uint64_t slotCount = 1; + if (auto multiType = dyn_cast(value.getType())) + slotCount = multiType.getCount(); + else if (auto attr = + defOp->getAttrOfType(pto::kPtoMultiBufferAttrName)) + slotCount = attr.getValue().getZExtValue(); + MemSpec spec = getMemSpec(getTargetArch(func), *space); + uint64_t slotBytes = alignUp(*bytesOr, spec.alignmentBytes); + + RootInfo info; + info.root = value; + info.defOp = defOp; + info.space = *space; + info.slotBytes = slotBytes; + info.totalBytes = slotBytes * slotCount; + info.alignmentBytes = spec.alignmentBytes; + info.slotCount = slotCount; + info.stableOrder = roots.size(); + roots.push_back(info); + rootIndexByValue[value] = roots.size() - 1; + valueToRoots[value] = RootList{value}; + } + + void markUse(Value value, unsigned index) { + for (Value root : getRoots(value)) { + auto found = rootIndexByValue.find(root); + if (found == rootIndexByValue.end()) + continue; + RootInfo &info = roots[found->second]; + if (!info.hasWriter) + info.hasUseBeforeFirstWrite = true; + if (index < info.allocIndex) + info.allocIndex = index; + info.freeIndex = std::max(info.freeIndex, index); + } + } + + void markWrite(Value value, unsigned index, bool pureOverwrite) { + for (Value root : getRoots(value)) { + auto found = rootIndexByValue.find(root); + if (found == rootIndexByValue.end()) + continue; + RootInfo &info = roots[found->second]; + if (!info.hasWriter) { + if (pureOverwrite && !info.hasUseBeforeFirstWrite) + info.allocIndex = index; + info.hasWriter = true; + } + info.freeIndex = std::max(info.freeIndex, index); + } + } + + void addForbidAlias(Value a, Value b) { + if (a == b) + return; + if (!rootIndexByValue.count(a) || !rootIndexByValue.count(b)) + return; + appendUniqueRoot(facts.forbidAlias[a], b); + appendUniqueRoot(facts.forbidAlias[b], a); + } + + bool hasForbidAlias(Value a, Value b) const { + if (a == b) + return false; + auto it = facts.forbidAlias.find(a); + if (it == facts.forbidAlias.end()) + return false; + return llvm::is_contained(it->second, b); + } + + void addForbidAliasBetweenRoots(const RootList &lhsRoots, + const RootList &rhsRoots) { + for (Value lhs : lhsRoots) + for (Value rhs : rhsRoots) + addForbidAlias(lhs, rhs); + } + + void markRoots(DenseSet &set, const RootList &roots) { + for (Value root : roots) + set.insert(root); + } + + bool rootsContain(const DenseSet &set, const RootList &roots) const { + return llvm::any_of(roots, [&](Value root) { return set.contains(root); }); + } + + bool isSplitTpopDerived(Value value) const { + return splitTpopDerivedValues.contains(value); + } + + bool operandsContainSplitTpopDerived(Operation *op) const { + return llvm::any_of(op->getOperands(), [&](Value operand) { + return isSplitTpopDerived(operand); + }); + } + + bool operandsContainLoadDerivedRoot(Operation *op) const { + return llvm::any_of(op->getOperands(), [&](Value operand) { + return rootsContain(facts.loadDerivedRoots, getRoots(operand)); + }); + } + + void propagateSplitTpopDerived(Value result, ValueRange sources) { + if (!result) + return; + if (llvm::any_of(sources, [&](Value source) { + return isSplitTpopDerived(source); + })) + splitTpopDerivedValues.insert(result); + } + + void propagateSplitTpopDerivedFromOperands(Operation *op) { + if (!operandsContainSplitTpopDerived(op)) + return; + for (Value result : op->getResults()) + splitTpopDerivedValues.insert(result); + } + + bool isRootDefinedInRegion(Value root, Region ®ion) const { + auto it = rootIndexByValue.find(root); + if (it == rootIndexByValue.end()) + return false; + + Operation *defOp = roots[it->second].defOp; + for (Operation *cur = defOp; cur; cur = cur->getParentOp()) { + if (cur->getParentRegion() == ®ion) + return true; + } + return false; + } + + void addPhiFamily(Value root, unsigned familyId) { + if (!rootIndexByValue.count(root)) + return; + SmallVector &families = facts.phiFamilyIds[root]; + if (!llvm::is_contained(families, familyId)) + families.push_back(familyId); + } + + void recordIfPhiFamilies(scf::IfOp ifOp) { + if (ifOp.getNumResults() == 0) + return; + + auto thenYield = cast(ifOp.thenBlock()->getTerminator()); + auto elseYield = cast(ifOp.elseBlock()->getTerminator()); + for (auto [thenVal, elseVal] : + llvm::zip(thenYield.getResults(), elseYield.getResults())) { + RootList roots = unionRoots(getRoots(thenVal), getRoots(elseVal)); + SmallVector branchLocalRoots; + for (Value root : roots) { + if (isRootDefinedInRegion(root, ifOp.getThenRegion()) || + isRootDefinedInRegion(root, ifOp.getElseRegion())) + branchLocalRoots.push_back(root); + } + if (branchLocalRoots.size() < 2) + continue; + + unsigned familyId = nextPhiFamilyId++; + for (Value root : branchLocalRoots) + addPhiFamily(root, familyId); + } + } + + void recordDpsScratchConflicts(Operation *op, ValueRange dpsInits) { + RootList outputRoots; + for (Value init : dpsInits) + outputRoots = unionRoots(outputRoots, getRoots(init)); + if (outputRoots.empty()) + return; + + for (Value scratch : getWrittenNonDpsOperands(op, dpsInits)) + addForbidAliasBetweenRoots(getRoots(scratch), outputRoots); + } + + void recordInplacePolicyConflicts(Operation *op, ValueRange dpsInits) { + RootList outputRoots; + for (Value init : dpsInits) + outputRoots = unionRoots(outputRoots, getRoots(init)); + if (outputRoots.empty()) + return; + + InplacePolicy policy = getInplacePolicy(op); + if (policy.notInplaceSafe) { + for (Value operand : op->getOperands()) { + if (llvm::is_contained(dpsInits, operand)) + continue; + addForbidAliasBetweenRoots(getRoots(operand), outputRoots); + } + } + + for (unsigned operandIndex : policy.forbidOutputAliasOperands) { + if (operandIndex >= op->getNumOperands()) + continue; + Value operand = op->getOperand(operandIndex); + if (llvm::is_contained(dpsInits, operand)) + continue; + addForbidAliasBetweenRoots(getRoots(operand), outputRoots); + } + } + + void recordDpsInplaceConflicts(Operation *op) { + auto dpsOp = dyn_cast(op); + if (!dpsOp) + return; + + ValueRange dpsInits = dpsOp.getDpsInits(); + recordDpsScratchConflicts(op, dpsInits); + recordInplacePolicyConflicts(op, dpsInits); + } + + void recordLoadDerivedRoots(Operation *op, ValueRange dpsInits) { + if (!isa(op)) + return; + + for (Value init : dpsInits) + markRoots(facts.loadDerivedRoots, getRoots(init)); + } + + void recordTpopConsumerRoots(Operation *op, ValueRange dpsInits, + unsigned opIndex) { + if (!facts.targetHazardEnabled) + return; + if (!operandsContainSplitTpopDerived(op) || !operandsContainLoadDerivedRoot(op)) + return; + + RootList outputRoots; + for (Value init : dpsInits) + outputRoots = unionRoots(outputRoots, getRoots(init)); + markRoots(facts.tpopConsumerRoots, outputRoots); + for (Value root : outputRoots) { + SmallVector &indices = facts.tpopConsumerWriteIndices[root]; + if (!llvm::is_contained(indices, opIndex)) + indices.push_back(opIndex); + } + } + + void recordDpsTargetHazardFacts(Operation *op, unsigned opIndex) { + auto dpsOp = dyn_cast(op); + if (!dpsOp) + return; + + ValueRange dpsInits = dpsOp.getDpsInits(); + recordLoadDerivedRoots(op, dpsInits); + recordTpopConsumerRoots(op, dpsInits, opIndex); + } + + void recordSplitTpopDerivedValue(Operation *op) { + if (!facts.targetHazardEnabled) + return; + + if (auto pop = dyn_cast(op)) { + if (pop.getSplit() != 0) + splitTpopDerivedValues.insert(pop.getTile()); + return; + } + + if (auto pop = dyn_cast(op)) { + if (pop.getSplit() != 0) + splitTpopDerivedValues.insert(pop.getTile()); + return; + } + + propagateSplitTpopDerivedFromOperands(op); + } + + void seedForIterArgAliases(scf::ForOp forOp) { + if (forOp.getRegion().empty()) + return; + Block &body = forOp.getRegion().front(); + for (auto [iterArg, initArg] : + llvm::zip(body.getArguments().drop_front(1), forOp.getInitArgs())) { + setRoots(iterArg, getRoots(initArg)); + } + } + + void walkRegion(Region ®ion) { + for (Block &block : region) { + for (Operation &opRef : block) { + Operation *op = &opRef; + unsigned index = linearOps.size(); + linearOps.push_back(op); + opToIndex[op] = index; + + if (auto allocTile = dyn_cast(op)) { + if (!allocTile.getAddr()) { + if (isA5IgnoredTmpAlloc(allocTile)) + continue; + addRoot(allocTile.getResult(), op); + if (failed) + return; + auto found = rootIndexByValue.find(allocTile.getResult()); + if (found != rootIndexByValue.end()) { + roots[found->second].allocIndex = index; + roots[found->second].freeIndex = index; + } + } + } else if (auto allocMulti = dyn_cast(op)) { + if (!allocMulti.getAddr()) { + addRoot(allocMulti.getResult(), op); + if (failed) + return; + auto found = rootIndexByValue.find(allocMulti.getResult()); + if (found != rootIndexByValue.end()) { + roots[found->second].allocIndex = index; + roots[found->second].freeIndex = index; + } + } + } else if (auto alloc = dyn_cast(op)) { + addRoot(alloc.getResult(), op); + if (failed) + return; + auto found = rootIndexByValue.find(alloc.getResult()); + if (found != rootIndexByValue.end()) { + roots[found->second].allocIndex = index; + roots[found->second].freeIndex = index; + } + } + + if (auto bind = dyn_cast(op)) { + setRoots(bind.getResult(), getRoots(bind.getSource())); + propagateSplitTpopDerived(bind.getResult(), ValueRange{bind.getSource()}); + } else if (auto slotMarker = dyn_cast(op)) { + setRoots(slotMarker.getResult(), getRoots(slotMarker.getSource())); + propagateSplitTpopDerived(slotMarker.getResult(), + ValueRange{slotMarker.getSource()}); + } else if (auto multiGet = dyn_cast(op)) { + setRoots(multiGet.getResult(), getRoots(multiGet.getSource())); + propagateSplitTpopDerived(multiGet.getResult(), + ValueRange{multiGet.getSource()}); + } else if (auto select = dyn_cast(op)) { + setRoots(select.getResult(), + unionRoots(getRoots(select.getTrueValue()), + getRoots(select.getFalseValue()))); + propagateSplitTpopDerived(select.getResult(), + ValueRange{select.getTrueValue(), + select.getFalseValue()}); + } else if (auto castOp = dyn_cast(op)) { + setRoots(castOp.getResult(), getRoots(castOp.getSource())); + propagateSplitTpopDerived(castOp.getResult(), + ValueRange{castOp.getSource()}); + } else if (auto subview = dyn_cast(op)) { + setRoots(subview.getResult(), getRoots(subview.getSource())); + propagateSplitTpopDerived(subview.getResult(), + ValueRange{subview.getSource()}); + } else if (auto subview = dyn_cast(op)) { + setRoots(subview.getResult(), getRoots(subview.getSource())); + propagateSplitTpopDerived(subview.getResult(), + ValueRange{subview.getSource()}); + } else if (auto bitcast = dyn_cast(op)) { + setRoots(bitcast.getResult(), getRoots(bitcast.getSrc())); + propagateSplitTpopDerived(bitcast.getResult(), + ValueRange{bitcast.getSrc()}); + } else if (auto reshape = dyn_cast(op)) { + setRoots(reshape.getResult(), getRoots(reshape.getSrc())); + propagateSplitTpopDerived(reshape.getResult(), + ValueRange{reshape.getSrc()}); + } else if (auto reinterpret = dyn_cast(op)) { + setRoots(reinterpret.getResult(), getRoots(reinterpret.getSource())); + propagateSplitTpopDerived(reinterpret.getResult(), + ValueRange{reinterpret.getSource()}); + } else if (auto forOp = dyn_cast(op)) { + seedForIterArgAliases(forOp); + } + + auto dpsOp = dyn_cast(op); + ValueRange dpsInits = dpsOp ? dpsOp.getDpsInits() : ValueRange(); + for (Value operand : op->getOperands()) { + if (llvm::is_contained(dpsInits, operand)) + continue; + markUse(operand, index); + } + for (Value init : dpsInits) { + bool readsOldValue = hasReadEffectOnValue(op, init); + if (readsOldValue) + markUse(init, index); + markWrite(init, index, /*pureOverwrite=*/!readsOldValue); + } + + for (Region &nested : op->getRegions()) { + walkRegion(nested); + if (failed) + return; + } + + if (auto ifOp = dyn_cast(op)) { + if (ifOp.getNumResults() != 0) { + auto thenYield = + cast(ifOp.thenBlock()->getTerminator()); + auto elseYield = + cast(ifOp.elseBlock()->getTerminator()); + for (auto [result, thenVal, elseVal] : + llvm::zip(ifOp.getResults(), thenYield.getResults(), + elseYield.getResults())) { + setRoots(result, + unionRoots(getRoots(thenVal), getRoots(elseVal))); + } + recordIfPhiFamilies(ifOp); + } + } else if (auto forOp = dyn_cast(op)) { + auto yieldOp = cast(forOp.getBody()->getTerminator()); + for (auto [result, initArg, yielded] : + llvm::zip(forOp.getResults(), forOp.getInitArgs(), + yieldOp.getResults())) { + setRoots(result, unionRoots(getRoots(initArg), getRoots(yielded))); + } + } + + recordSplitTpopDerivedValue(op); + recordDpsTargetHazardFacts(op, index); + recordDpsInplaceConflicts(op); + } + } + } +}; + +static bool lifetimesStrictlyOverlap(const RootInfo &lhs, const RootInfo &rhs) { + return !(lhs.freeIndex <= rhs.allocIndex || rhs.freeIndex <= lhs.allocIndex); +} + +static bool sharePhiFamily(Value lhs, Value rhs, const ConflictFacts &facts) { + auto lhsIt = facts.phiFamilyIds.find(lhs); + auto rhsIt = facts.phiFamilyIds.find(rhs); + if (lhsIt == facts.phiFamilyIds.end() || rhsIt == facts.phiFamilyIds.end()) + return false; + + for (unsigned lhsFamily : lhsIt->second) { + if (llvm::is_contained(rhsIt->second, lhsFamily)) + return true; + } + return false; +} + +static bool gateLifetimeAndPhi(const RootInfo &lhs, const RootInfo &rhs, + const ConflictFacts &facts) { + if (!lifetimesStrictlyOverlap(lhs, rhs)) + return true; + + // Gate 5 is intentionally embedded in Gate 1: branch-local roots yielded by + // the same scf.if result are mutually exclusive at runtime, so their + // post-phi liveness extension does not by itself forbid reuse. + return sharePhiFamily(lhs.root, rhs.root, facts); +} + +static bool hasTargetHazard(const RootInfo &input, const RootInfo &writer, + const ConflictFacts &facts) { + if (!facts.targetHazardEnabled) + return false; + + if (!facts.loadDerivedRoots.contains(input.root) || + !facts.tpopConsumerRoots.contains(writer.root)) + return false; + + auto found = facts.tpopConsumerWriteIndices.find(writer.root); + if (found == facts.tpopConsumerWriteIndices.end()) + return false; + + return llvm::is_contained(found->second, input.freeIndex); +} + +static bool gateTargetHazard(const RootInfo &lhs, const RootInfo &rhs, + const ConflictFacts &facts) { + return !hasTargetHazard(lhs, rhs, facts) && !hasTargetHazard(rhs, lhs, facts); +} + +static bool gateSemanticNoAlias(const RootInfo &lhs, const RootInfo &rhs, + const PlannerAnalysis &analysis) { + return !analysis.hasForbidAlias(lhs.root, rhs.root); +} + +static bool canShare(const RootInfo &lhs, const RootInfo &rhs, + const PlannerAnalysis &analysis) { + const ConflictFacts &facts = analysis.facts; + return gateLifetimeAndPhi(lhs, rhs, facts) && + gateTargetHazard(lhs, rhs, facts) && + gateSemanticNoAlias(lhs, rhs, analysis); +} + +static bool canJoinReuseGroup(const RootInfo &info, const ReuseGroup &group, + const PlannerAnalysis &analysis) { + if (info.space != group.space) + return false; + for (const RootInfo *member : group.members) { + if (!canShare(info, *member, analysis)) + return false; + } + return true; +} + +static SmallVector buildSlotOffsets(uint64_t base, uint64_t slotBytes, + uint64_t slotCount) { + SmallVector offsets; + offsets.reserve(slotCount); + for (uint64_t slot = 0; slot < slotCount; ++slot) + offsets.push_back(base + slot * slotBytes); + return offsets; +} + +static FailureOr planReserveBufferBase( + pto::ReserveBufferOp reserveOp, const MemSpec &spec, + SmallVectorImpl> &occupied) { + uint64_t sizeBytes = + alignUp(static_cast(reserveOp.getSize()), spec.alignmentBytes); + llvm::sort(occupied, [](const auto &lhs, const auto &rhs) { + return lhs.first < rhs.first; + }); + + SmallVector> merged; + for (const auto &interval : occupied) { + if (merged.empty() || interval.first > merged.back().second) { + merged.push_back(interval); + continue; + } + merged.back().second = std::max(merged.back().second, interval.second); + } + + uint64_t cursor = 0; + for (const auto &interval : merged) { + cursor = alignUp(cursor, spec.alignmentBytes); + if (cursor + sizeBytes <= interval.first) + return cursor; + cursor = std::max(cursor, interval.second); + } + cursor = alignUp(cursor, spec.alignmentBytes); + if (cursor + sizeBytes > spec.capacityBytes) + return failure(); + occupied.push_back({cursor, cursor + sizeBytes}); + return cursor; +} + +static LogicalResult +validateManualReserveBufferBase(pto::ReserveBufferOp reserveOp, + const MemSpec &spec) { + auto baseAttr = reserveOp.getBaseAttr(); + if (!baseAttr) + return reserveOp.emitError("expects 'base' when 'auto' is false"); + + int64_t signedBase = baseAttr.getInt(); + if (signedBase < 0) + return reserveOp.emitError( + "expects 'base' to be non-negative when present"); + + uint64_t base = static_cast(signedBase); + if (base % spec.alignmentBytes != 0) { + return reserveOp.emitError("expects 'base' to be aligned to ") + << spec.alignmentBytes << " bytes for " + << stringifyEnum(reserveOp.getLocation().getAddressSpace()); + } + + uint64_t size = static_cast(reserveOp.getSize()); + if (base > spec.capacityBytes || size > spec.capacityBytes - base) { + return reserveOp.emitError("reserved range exceeds ") + << stringifyEnum(reserveOp.getLocation().getAddressSpace()) + << " capacity: base " << base << " + size " << size << " > " + << spec.capacityBytes << " bytes"; + } + + return success(); +} + +static LogicalResult +validateReserveBufferForPlanMemory(pto::ReserveBufferOp reserveOp, + PTOArch arch) { + MemSpec spec = getMemSpec(arch, reserveOp.getLocation().getAddressSpace()); + + if (reserveOp.getAutoAlloc()) { + if (reserveOp.getBaseAttr()) { + return reserveOp.emitError( + "unexpected pre-populated 'base' on auto reserve_buffer before " + "pto-plan-memory; omit 'base' and let pto-plan-memory assign it"); + } + return success(); + } + + if (mlir::failed(validateManualReserveBufferBase(reserveOp, spec))) + return failure(); + + return reserveOp.emitError( + "pto.reserve_buffer with explicit 'base' (auto = false) is not " + "supported in PlanMemory; use --pto-level=level3 or set auto = true"); +} + +class AllocTileOpAddPlannedAddressPattern + : public OpRewritePattern { +public: + explicit AllocTileOpAddPlannedAddressPattern( + MLIRContext *context, + DenseMap> buffer2Offsets) + : OpRewritePattern(context), + buffer2Offsets(std::move(buffer2Offsets)) {} + + LogicalResult matchAndRewrite(pto::AllocTileOp op, + PatternRewriter &rewriter) const override { + if (op.getAddr()) + return failure(); + + auto tileType = dyn_cast(op.getResult().getType()); + if (!tileType) + return failure(); + + auto it = buffer2Offsets.find(op.getResult()); + if (it == buffer2Offsets.end() || it->second.empty()) + return failure(); + + if (it->second.size() != 1) { + return rewriter.notifyMatchFailure( + op, "single alloc_tile root expects exactly one planned address"); + } + + Value addr = rewriter.create(op.getLoc(), + it->second.front(), 64); + auto planned = rewriter.create( + op.getLoc(), tileType, addr, + op.getValidRow() ? op.getValidRow() : Value(), + op.getValidCol() ? op.getValidCol() : Value()); + for (NamedAttribute attr : op->getAttrs()) { + if (attr.getName().getValue() == "operandSegmentSizes") + continue; + planned->setAttr(attr.getName(), attr.getValue()); + } + + rewriter.replaceOp(op, planned.getResult()); + return success(); + } + +private: + DenseMap> buffer2Offsets; +}; + +class AllocMultiTileOpAddPlannedAddressesPattern + : public OpRewritePattern { +public: + explicit AllocMultiTileOpAddPlannedAddressesPattern( + MLIRContext *context, + DenseMap> buffer2Offsets) + : OpRewritePattern(context), + buffer2Offsets(std::move(buffer2Offsets)) {} + + LogicalResult matchAndRewrite(pto::AllocMultiTileOp op, + PatternRewriter &rewriter) const override { + if (op.getAddr() || op->hasAttr(pto::kPtoMultiBufferAddrsAttrName)) + return failure(); + + auto it = buffer2Offsets.find(op.getResult()); + if (it == buffer2Offsets.end() || it->second.empty()) + return failure(); + if (it->second.size() != op.getResult().getType().getCount()) { + return rewriter.notifyMatchFailure( + op, "planned address count does not match multi_tile_buf count"); + } + + SmallVector addrs; + addrs.reserve(it->second.size()); + for (uint64_t offset : it->second) + addrs.push_back(static_cast(offset)); + rewriter.modifyOpInPlace(op, [&] { + op->setAttr(pto::kPtoMultiBufferAddrsAttrName, + rewriter.getDenseI64ArrayAttr(addrs)); + }); + return success(); + } + +private: + DenseMap> buffer2Offsets; +}; + +static LogicalResult materializePlannedOffsets( + func::FuncOp func, DenseMap> buffer2Offsets) { + RewritePatternSet patterns(func.getContext()); + patterns.add(patterns.getContext(), + buffer2Offsets); + patterns.add(patterns.getContext(), + buffer2Offsets); + patterns.add( + patterns.getContext(), buffer2Offsets); + if (mlir::failed(applyPatternsGreedily(func, std::move(patterns)))) + return failure(); + return success(); +} + +} // namespace + +LogicalResult mlir::pto::runModernPlanMemory(func::FuncOp func, + llvm::StringRef memMode, + bool orderBySize) { + if (!memMode.equals_insensitive("local")) { + func.emitError("unsupported mem-mode '") + << memMode << "'; only 'local' is currently implemented"; + return failure(); + } + + PlannerAnalysis analysis(func); + analysis.walkRegion(func.getBody()); + if (analysis.failed) { + return failure(); + } + + DenseMap> buffer2Offsets; + llvm::MapVector> rootsBySpace; + for (RootInfo &info : analysis.roots) + rootsBySpace[info.space].push_back(&info); + + for (auto &entry : rootsBySpace) { + AddressSpace space = entry.first; + SmallVector &roots = entry.second; + MemSpec spec = getMemSpec(getTargetArch(func), space); + + llvm::stable_sort(roots, [&](const RootInfo *lhs, const RootInfo *rhs) { + if (orderBySize && lhs->totalBytes != rhs->totalBytes) + return lhs->totalBytes > rhs->totalBytes; + if (lhs->allocIndex != rhs->allocIndex) + return lhs->allocIndex < rhs->allocIndex; + return lhs->stableOrder < rhs->stableOrder; + }); + + SmallVector groups; + uint64_t scopeRequiredBytes = 0; + for (RootInfo *info : roots) { + ReuseGroup *chosen = nullptr; + for (ReuseGroup &group : groups) { + if (canJoinReuseGroup(*info, group, analysis)) { + chosen = &group; + break; + } + } + + if (!chosen) { + ReuseGroup group; + group.space = space; + group.alignmentBytes = info->alignmentBytes; + groups.push_back(std::move(group)); + chosen = &groups.back(); + } + + chosen->members.push_back(info); + chosen->sizeBytes = std::max(chosen->sizeBytes, info->totalBytes); + chosen->alignmentBytes = + std::max(chosen->alignmentBytes, info->alignmentBytes); + } + + uint64_t cursor = 0; + for (ReuseGroup &group : groups) { + cursor = alignUp(cursor, group.alignmentBytes); + group.offsetBytes = cursor; + if (group.offsetBytes + group.sizeBytes > spec.capacityBytes) { + scopeRequiredBytes = group.offsetBytes + group.sizeBytes; + break; + } + + for (RootInfo *member : group.members) { + member->offsets = buildSlotOffsets(group.offsetBytes, member->slotBytes, + member->slotCount); + } + scopeRequiredBytes = + std::max(scopeRequiredBytes, group.offsetBytes + group.sizeBytes); + cursor = group.offsetBytes + group.sizeBytes; + } + + if (scopeRequiredBytes > spec.capacityBytes) { + func.emitError() << stringifyEnum(space) << " overflow, requires " + << (scopeRequiredBytes * 8) << " bits while " + << (spec.capacityBytes * 8) << " bits available"; + return failure(); + } + + for (RootInfo *info : roots) + buffer2Offsets[info->root] = info->offsets; + } + + DenseMap>> + occupiedBySpace; + for (const RootInfo &info : analysis.roots) { + if (info.offsets.empty()) + continue; + occupiedBySpace[info.space].push_back( + {info.offsets.front(), info.offsets.front() + info.totalBytes}); + } + + MLIRContext *ctx = func.getContext(); + PTOArch arch = getTargetArch(func); + func.walk([&](pto::ReserveBufferOp reserveOp) -> WalkResult { + if (mlir::failed(validateReserveBufferForPlanMemory(reserveOp, arch))) { + analysis.failed = true; + return WalkResult::interrupt(); + } + + AddressSpace space = reserveOp.getLocation().getAddressSpace(); + MemSpec spec = getMemSpec(arch, space); + auto baseOr = + planReserveBufferBase(reserveOp, spec, occupiedBySpace[space]); + if (mlir::failed(baseOr)) { + reserveOp.emitError("failed to reserve a local memory hole for " + "reserve_buffer"); + analysis.failed = true; + return WalkResult::interrupt(); + } + + reserveOp->setAttr("base", IntegerAttr::get(IntegerType::get(ctx, 32), + static_cast(*baseOr))); + return WalkResult::advance(); + }); + + if (analysis.failed || + mlir::failed(materializePlannedOffsets(func, buffer2Offsets))) { + return failure(); + } + + bool hasUnplannedAllocTile = false; + func.walk([&](pto::AllocTileOp op) { + if (op.getAddr()) + return; + if (op->use_empty()) + return; + if (isA5IgnoredTmpAlloc(op)) + return; + op.emitError("PTOPlanMemory failed to assign an address to pto.alloc_tile"); + hasUnplannedAllocTile = true; + }); + if (hasUnplannedAllocTile) + return failure(); + return success(); +} + +namespace { +struct PlanMemoryModernPass + : public PassWrapper> { + MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(PlanMemoryModernPass) + + PlanMemoryModernPass() = default; + explicit PlanMemoryModernPass(const PlanMemoryOptions &options) + : memMode(options.memMode), orderBySize(options.orderBySize) {} + + StringRef getArgument() const final { return "pto-plan-memory"; } + StringRef getDescription() const final { + return "Plan local memory using the modern PTO planner"; + } + StringRef getName() const final { return "PlanMemoryModern"; } + + void getDependentDialects(DialectRegistry ®istry) const override { + registry.insert(); + registry.insert(); + registry.insert(); + registry.insert(); + registry.insert(); + } + + void runOnOperation() override { + ModuleOp moduleOp = getOperation(); + for (func::FuncOp funcOp : moduleOp.getOps()) { + if (failed( + mlir::pto::runModernPlanMemory(funcOp, memMode, orderBySize))) { + signalPassFailure(); + return; + } + } + } + +private: + std::string memMode = "local"; + bool orderBySize = false; +}; +} // namespace + +std::unique_ptr +mlir::pto::createPlanMemoryModernPass(const PlanMemoryOptions &options) { + return std::make_unique(options); +} diff --git a/lib/PTO/Transforms/PTOResolveBufferSelect.cpp b/lib/PTO/Transforms/PTOResolveBufferSelect.cpp index 858593e6d3..e6ef8ae590 100644 --- a/lib/PTO/Transforms/PTOResolveBufferSelect.cpp +++ b/lib/PTO/Transforms/PTOResolveBufferSelect.cpp @@ -30,11 +30,13 @@ #include "PTO/IR/PTO.h" #include "PTO/IR/PTOMultiBuffer.h" +#include "PTO/IR/PTOTypeUtils.h" #include "PTO/Transforms/Passes.h" #include "mlir/Dialect/Arith/IR/Arith.h" #include "mlir/Dialect/Func/IR/FuncOps.h" #include "mlir/Dialect/MemRef/IR/MemRef.h" +#include "mlir/Dialect/SCF/IR/SCF.h" #include "mlir/IR/BuiltinOps.h" #include "mlir/IR/IRMapping.h" #include "mlir/IR/Matchers.h" @@ -54,6 +56,414 @@ using namespace mlir; namespace { +static Value ensureI64(Value value, IRRewriter &rewriter, Location loc) { + if (!value) + return {}; + if (value.getType().isInteger(64)) + return value; + if (value.getType().isIndex()) + return rewriter.create(loc, rewriter.getI64Type(), value); + if (isa(value.getType())) + return rewriter.create(loc, rewriter.getI64Type(), value); + return {}; +} + +static bool getTilePointerStrides(pto::TileBufType type, int64_t &rowStride, + int64_t &colStride) { + auto shape = type.getShape(); + if (shape.size() != 2 || llvm::is_contained(shape, ShapedType::kDynamic)) + return false; + + auto config = type.getConfigAttr(); + int32_t bl = static_cast(config.getBLayout().getValue()); + int32_t sl = static_cast(config.getSLayout().getValue()); + if (sl == 0) { + bool rowPlusOne = + type.getCompactModeI32() == + static_cast(pto::CompactMode::RowPlusOne); + rowStride = bl == 1 ? 1 : shape[1] + (rowPlusOne ? 1 : 0); + colStride = bl == 1 ? shape[0] + (rowPlusOne ? 1 : 0) : 1; + return true; + } + + unsigned elemBytes = pto::getPTOStorageElemByteSize(type.getElementType()); + if (elemBytes == 0) + return false; + int64_t innerRows = 1; + int64_t innerCols = 1; + int32_t fractal = config.getSFractalSize().getInt(); + if (fractal == 1024) { + innerRows = 16; + innerCols = 16; + } else if (fractal == 32) { + innerRows = 16; + innerCols = 2; + } else if (fractal == 512 && sl == 1) { + innerRows = 16; + innerCols = 32 / elemBytes; + } else if (fractal == 512 && sl == 2) { + innerRows = 32 / elemBytes; + innerCols = 16; + } else { + return false; + } + + if (bl == 1) { + if (sl != 1) + return false; + rowStride = innerCols; + colStride = shape[0]; + } else { + rowStride = shape[1]; + colStride = innerRows; + } + return true; +} + +static Value computeTileAddress(Value value, IRRewriter &rewriter, + Location loc) { + if (auto alloc = value.getDefiningOp()) + return ensureI64(alloc.getAddr(), rewriter, loc); + if (auto subview = value.getDefiningOp()) { + Value base = computeTileAddress(subview.getSource(), rewriter, loc); + auto sourceType = subview.getSource().getType(); + int64_t rowStride = 0; + int64_t colStride = 0; + if (!base || !getTilePointerStrides(sourceType, rowStride, colStride) || + subview.getOffsets().size() != 2) + return {}; + Value row = ensureI64(subview.getOffsets()[0], rewriter, loc); + Value col = ensureI64(subview.getOffsets()[1], rewriter, loc); + if (!row || !col) + return {}; + Value rowScale = rewriter.create(loc, rowStride, 64); + Value colScale = rewriter.create(loc, colStride, 64); + row = rewriter.create(loc, row, rowScale); + col = rewriter.create(loc, col, colScale); + Value elements = rewriter.create(loc, row, col); + int64_t elemBytes = static_cast( + pto::getPTOStorageElemByteSize(sourceType.getElementType())); + if (elemBytes == 0) + return {}; + Value byteScale = rewriter.create(loc, elemBytes, 64); + Value bytes = rewriter.create(loc, elements, byteScale); + return rewriter.create(loc, base, bytes); + } + return {}; +} + +static pto::TileBufType getSubviewPhysicalType(pto::SubViewOp op) { + pto::TileBufType sourceType = op.getSource().getType(); + pto::TileBufType resultType = op.getResult().getType(); + return pto::TileBufType::get( + op.getContext(), sourceType.getShape(), resultType.getElementType(), + resultType.getMemorySpace(), resultType.getValidShape(), + resultType.getConfigAttr()); +} + +static Value getSubviewValidOperand(pto::SubViewOp op, + pto::TileBufType physicalType, + unsigned dim, IRRewriter &rewriter) { + Value operand = dim == 0 ? op.getValidRow() : op.getValidCol(); + ArrayRef validShape = physicalType.getValidShape(); + if (validShape.size() <= dim || validShape[dim] >= 0) + return {}; + if (operand) + return operand; + ArrayRef shape = physicalType.getShape(); + if (shape.size() > dim && shape[dim] != ShapedType::kDynamic) + return rewriter.create(op.getLoc(), shape[dim]); + return {}; +} + +static LogicalResult resolveTileNativeSubviews(ModuleOp module, + MLIRContext *ctx) { + SmallVector subviews; + module.walk([&](pto::SubViewOp op) { subviews.push_back(op); }); + for (pto::SubViewOp op : subviews) { + IRRewriter rewriter(ctx); + rewriter.setInsertionPoint(op); + Value addr = computeTileAddress(op.getResult(), rewriter, op.getLoc()); + // A tile function argument is a symbolic runtime-bound handle. Keep its + // subview tile-native; only planned local roots can be normalized to an + // addressed alloc_tile here. + if (!addr) + continue; + pto::TileBufType physicalType = getSubviewPhysicalType(op); + auto alloc = rewriter.create( + op.getLoc(), physicalType, addr, + getSubviewValidOperand(op, physicalType, 0, rewriter), + getSubviewValidOperand(op, physicalType, 1, rewriter)); + alloc->setAttr("pto.view_semantics", rewriter.getStringAttr("subview")); + rewriter.replaceOp(op, alloc.getResult()); + } + return success(); +} + +static LogicalResult reconcileSCFViewResultTypes(ModuleOp module) { + SmallVector ifOps; + module.walk([&](scf::IfOp op) { ifOps.push_back(op); }); + for (scf::IfOp op : llvm::reverse(ifOps)) { + if (op.getNumResults() == 0) + continue; + auto thenYield = dyn_cast(op.thenBlock()->getTerminator()); + auto elseYield = dyn_cast(op.elseBlock()->getTerminator()); + if (!thenYield || !elseYield || + thenYield.getNumOperands() != op.getNumResults() || + elseYield.getNumOperands() != op.getNumResults()) + return op.emitError("cannot reconcile scf.if view result types"); + for (unsigned i = 0; i < op.getNumResults(); ++i) { + Type thenType = thenYield.getOperand(i).getType(); + if (thenType != elseYield.getOperand(i).getType()) + return op.emitError("scf.if branches yield different view types"); + op.getResult(i).setType(thenType); + } + } + return success(); +} + +static LogicalResult resolveGMTensorViews(ModuleOp module, MLIRContext *ctx) { + SmallVector makeViews; + module.walk([&](pto::MakeTensorViewOp op) { makeViews.push_back(op); }); + DenseMap loweredViews; + for (pto::MakeTensorViewOp op : makeViews) { + IRRewriter rewriter(ctx); + rewriter.setInsertionPoint(op); + Value base = op.getPtr(); + OpFoldResult offset = rewriter.getIndexAttr(0); + Value totalOffset; + while (auto add = base.getDefiningOp()) { + Value term = add.getOffset(); + totalOffset = + totalOffset + ? rewriter.create(op.getLoc(), totalOffset, term) + : term; + base = add.getPtr(); + } + if (totalOffset) + offset = totalOffset; + + auto baseType = dyn_cast(base.getType()); + if (!baseType) + continue; + int64_t rank = static_cast(op.getShape().size()); + SmallVector dynamicShape(rank, ShapedType::kDynamic); + SmallVector dynamicStrides(rank, ShapedType::kDynamic); + auto layout = + StridedLayoutAttr::get(ctx, ShapedType::kDynamic, dynamicStrides); + auto resultType = MemRefType::get(dynamicShape, baseType.getElementType(), + layout, baseType.getMemorySpace()); + SmallVector sizes(op.getShape().begin(), op.getShape().end()); + SmallVector strides(op.getStrides().begin(), + op.getStrides().end()); + auto view = rewriter.create( + op.getLoc(), resultType, base, offset, sizes, strides); + if (totalOffset) + view->setAttr("pto.addptr_trace", rewriter.getUnitAttr()); + if (auto layoutAttr = op.getLayoutAttr()) + view->setAttr("layout", layoutAttr); + loweredViews[op.getResult()] = view.getResult(); + } + + SmallVector dimOps; + module.walk([&](pto::GetTensorViewDimOp op) { dimOps.push_back(op); }); + for (pto::GetTensorViewDimOp op : dimOps) { + Value view = op.getTensorView(); + if (Value lowered = loweredViews.lookup(view)) + view = lowered; + if (!isa(view.getType())) + continue; + IRRewriter rewriter(ctx); + rewriter.setInsertionPoint(op); + rewriter.replaceOpWithNewOp(op, view, op.getDimIndex()); + } + + SmallVector partitions; + module.walk([&](pto::PartitionViewOp op) { partitions.push_back(op); }); + for (pto::PartitionViewOp op : partitions) { + Value source = op.getSource(); + if (Value lowered = loweredViews.lookup(source)) + source = lowered; + auto sourceType = dyn_cast(source.getType()); + if (!sourceType) + continue; + IRRewriter rewriter(ctx); + rewriter.setInsertionPoint(op); + SmallVector staticSizes; + SmallVector sizes; + for (Value size : op.getSizes()) { + IntegerAttr attr; + if (matchPattern(size, m_Constant(&attr))) { + int64_t value = attr.getValue().getSExtValue(); + staticSizes.push_back(value); + sizes.push_back(rewriter.getIndexAttr(value)); + } else { + staticSizes.push_back(ShapedType::kDynamic); + sizes.push_back(size); + } + } + SmallVector offsets(op.getOffsets().begin(), + op.getOffsets().end()); + SmallVector strides(sourceType.getRank(), + rewriter.getIndexAttr(1)); + SmallVector dynamicStrides(sourceType.getRank(), + ShapedType::kDynamic); + auto layout = + StridedLayoutAttr::get(ctx, ShapedType::kDynamic, dynamicStrides); + auto resultType = MemRefType::get(staticSizes, sourceType.getElementType(), + layout, sourceType.getMemorySpace()); + auto subview = rewriter.create( + op.getLoc(), resultType, source, offsets, sizes, strides); + if (Operation *def = source.getDefiningOp()) + if (auto attr = def->getAttrOfType("layout")) + subview->setAttr("layout", attr); + rewriter.replaceOp(op, subview.getResult()); + } + + SmallVector strideOps; + module.walk([&](pto::GetTensorViewStrideOp op) { strideOps.push_back(op); }); + for (pto::GetTensorViewStrideOp op : strideOps) { + Value view = op.getTensorView(); + if (Value lowered = loweredViews.lookup(view)) + view = lowered; + auto type = dyn_cast(view.getType()); + if (!type) + continue; + IntegerAttr dimAttr; + if (!matchPattern(op.getDimIndex(), m_Constant(&dimAttr))) + return op.emitError( + "get_tensor_view_stride expects a constant dim index"); + int64_t dim = dimAttr.getValue().getSExtValue(); + if (dim < 0 || dim >= type.getRank()) + return op.emitError("get_tensor_view_stride dim index is out of bounds"); + IRRewriter rewriter(ctx); + rewriter.setInsertionPoint(op); + SmallVector staticStrides; + int64_t staticOffset = ShapedType::kDynamic; + if (succeeded(pto::getPTOMemRefStridesAndOffset(type, staticStrides, + staticOffset)) && + staticStrides[dim] != ShapedType::kDynamic) { + rewriter.replaceOpWithNewOp(op, + staticStrides[dim]); + continue; + } + auto metadata = + rewriter.create(op.getLoc(), view); + rewriter.replaceOp(op, metadata.getStrides()[dim]); + } + + // Replace roots only after all strongly typed PTO view users have been + // removed. This avoids constructing an intermediate partition_view whose + // source has already changed from tensor_view to memref. + for (pto::MakeTensorViewOp op : makeViews) { + Value lowered = loweredViews.lookup(op.getResult()); + if (!lowered) + continue; + IRRewriter rewriter(ctx); + rewriter.replaceOp(op, lowered); + } + return reconcileSCFViewResultTypes(module); +} + +static FailureOr getStaticSlotBytes(pto::TileBufType slotType) { + uint64_t elemBytes = pto::getPTOStorageElemByteSize(slotType.getElementType()); + if (elemBytes == 0) + return failure(); + uint64_t bytes = elemBytes; + for (int64_t dim : slotType.getShape()) { + if (dim == ShapedType::kDynamic) + return failure(); + bytes *= static_cast(dim); + } + return bytes; +} + +static LogicalResult getMultiTileAddresses(pto::AllocMultiTileOp alloc, + IRRewriter &rewriter, + SmallVectorImpl &addrs) { + uint32_t count = alloc.getResult().getType().getCount(); + if (auto planned = alloc->getAttrOfType( + pto::kPtoMultiBufferAddrsAttrName)) { + if (planned.size() != count) + return alloc.emitError("planned address count does not match slot count"); + for (int64_t address : planned.asArrayRef()) + addrs.push_back(rewriter.create( + alloc.getLoc(), address, 64)); + return success(); + } + + Value base = alloc.getAddr(); + if (!base) + return alloc.emitError( + "has neither a level3 base address nor planner-assigned slot addresses"); + auto slotBytes = getStaticSlotBytes(alloc.getResult().getType().getSlotType()); + if (failed(slotBytes)) + return alloc.emitError( + "requires a static slot shape and known element byte size"); + + addrs.push_back(base); + for (uint32_t slot = 1; slot < count; ++slot) { + Value offset = rewriter.create( + alloc.getLoc(), static_cast(slot * *slotBytes), 64); + addrs.push_back( + rewriter.create(alloc.getLoc(), base, offset)); + } + return success(); +} + +static LogicalResult resolveTileNativeMultiGets(ModuleOp module, + MLIRContext *ctx) { + SmallVector gets; + module.walk([&](pto::MultiTileGetOp op) { gets.push_back(op); }); + + for (pto::MultiTileGetOp op : gets) { + auto alloc = op.getSource().getDefiningOp(); + if (!alloc) + return op.emitError( + "currently requires a direct pto.alloc_multi_tile source"); + + IRRewriter rewriter(ctx); + rewriter.setInsertionPoint(op); + SmallVector addrs; + if (failed(getMultiTileAddresses(alloc, rewriter, addrs))) + return failure(); + + Value selectedAddr; + IntegerAttr constSlotAttr; + if (matchPattern(op.getSlot(), m_Constant(&constSlotAttr))) { + int64_t slot = constSlotAttr.getValue().getSExtValue(); + if (slot < 0 || slot >= static_cast(addrs.size())) + return op.emitError("constant slot is outside planned address range"); + selectedAddr = addrs[static_cast(slot)]; + } else { + selectedAddr = addrs.front(); + for (uint32_t slot = 1; slot < addrs.size(); ++slot) { + Value slotValue = rewriter.create(op.getLoc(), slot); + Value matches = rewriter.create( + op.getLoc(), arith::CmpIPredicate::eq, op.getSlot(), slotValue); + selectedAddr = rewriter.create( + op.getLoc(), matches, addrs[slot], selectedAddr); + } + } + + auto slotHandle = rewriter.create( + op.getLoc(), op.getResult().getType(), selectedAddr, + alloc.getValidRow() ? alloc.getValidRow() : Value(), + alloc.getValidCol() ? alloc.getValidCol() : Value()); + rewriter.replaceOp(op, slotHandle.getResult()); + } + + SmallVector allocs; + module.walk([&](pto::AllocMultiTileOp op) { allocs.push_back(op); }); + for (pto::AllocMultiTileOp alloc : allocs) { + if (!alloc.getResult().use_empty()) + return alloc.emitError( + "has unsupported uses after resolving pto.multi_tile_get"); + alloc.erase(); + } + return success(); +} + /// Walk back through pure metadata ops (`pto.bind_tile`, `pto.slot_marker`) /// to find the root multi-address `pto.pointer_cast` that this view ties /// to. Returns nullptr if the chain does not terminate on a @@ -113,6 +523,19 @@ struct PTOResolveBufferSelectPass ModuleOp mod = getOperation(); MLIRContext *ctx = &getContext(); + if (failed(resolveTileNativeMultiGets(mod, ctx))) { + signalPassFailure(); + return; + } + if (failed(resolveTileNativeSubviews(mod, ctx))) { + signalPassFailure(); + return; + } + if (failed(resolveGMTensorViews(mod, ctx))) { + signalPassFailure(); + return; + } + SmallVector markers; mod.walk([&](pto::SlotMarkerOp op) { markers.push_back(op); }); if (markers.empty()) diff --git a/lib/PTO/Transforms/PTOToEmitC.cpp b/lib/PTO/Transforms/PTOToEmitC.cpp index ab57d08e24..5f369b9054 100644 --- a/lib/PTO/Transforms/PTOToEmitC.cpp +++ b/lib/PTO/Transforms/PTOToEmitC.cpp @@ -200,10 +200,6 @@ static Location getIndexedNameHintLoc(Location fallbackLoc, unsigned index) { fallbackLoc); } -[[maybe_unused]] static constexpr llvm::StringLiteral kLoweredSetValidShapeAttrName = - "__pto.lowered_set_validshape"; -[[maybe_unused]] static constexpr llvm::StringLiteral kLoweredSetValidShapeConfigAttrName = - "__pto.lowered_set_validshape_config"; static constexpr llvm::StringLiteral kForceDynamicValidShapeAttrName = "__pto.force_dynamic_valid_shape"; static constexpr llvm::StringLiteral kGlobalTensorStridesAttrName = @@ -1350,7 +1346,14 @@ static LogicalResult insertFixpipeConfigAliases(ModuleOp mop) { if (aliases.empty()) continue; - OpBuilder builder(&funcOp.front(), funcOp.front().begin()); + if (funcOp.empty()) { + funcOp.emitError("cannot insert fixpipe config aliases into an external " + "function"); + return failure(); + } + + OpBuilder builder(funcOp.getContext()); + builder.setInsertionPointToStart(&funcOp.front()); for (const auto &[pipeId, configTok] : aliases) { std::string line = "using " + buildFixpipeConfigAliasName(pipeId) + " = " + configTok + ";"; @@ -4641,6 +4644,104 @@ static FailureOr buildSyncAllWorkspaceTileValue( //===----------------------------------------------------------------------===// // pto.pointer_cast lowering //===----------------------------------------------------------------------=== + +struct CastPtrConversion : public OpConversionPattern { + using OpConversionPattern::OpConversionPattern; + + LogicalResult + matchAndRewrite(pto::CastPtrOp op, OpAdaptor adaptor, + ConversionPatternRewriter &rewriter) const override { + Type convertedResultType = + getTypeConverter()->convertType(op.getResult().getType()); + if (!convertedResultType) + return failure(); + + Value input = adaptor.getInput(); + Value peeledInput = peelUnrealized(input); + if (peeledInput.getType() == convertedResultType) { + rewriter.replaceOp(op, peeledInput); + return success(); + } + + if (auto resultPtrTy = dyn_cast(op.getResult().getType())) { + std::string elemTok = getEmitCScalarTypeToken(resultPtrTy.getElementType()); + std::optional as = + getAddressSpaceOrGM(resultPtrTy.getMemorySpace()); + if (!as) + return rewriter.notifyMatchFailure(op, "unsupported ptr address space"); + + if (isEmitCTileLikeType(peeledInput.getType())) { + Value ptr = rewriter + .create( + op.getLoc(), convertedResultType, "PTOAS__TILE_DATA", + ArrayAttr{}, ArrayAttr{}, ValueRange{peeledInput}) + .getResult(0); + rewriter.replaceOp(op, ptr); + return success(); + } + + Value ptr = materializeAddressAsPointer(rewriter, op.getLoc(), peeledInput, + *as, elemTok); + if (ptr.getType() != convertedResultType) + ptr = rewriter.create(op.getLoc(), convertedResultType, ptr) + .getResult(); + rewriter.replaceOp(op, ptr); + return success(); + } + + if (isa(op.getResult().getType()) && + emitc::isSupportedEmitCType(convertedResultType)) { + Value source = input; + if (!emitc::isSupportedEmitCType(source.getType())) { + if (auto inputPtrTy = dyn_cast(op.getInput().getType())) { + std::string elemTok = + getEmitCScalarTypeToken(inputPtrTy.getElementType()); + std::optional as = + getAddressSpaceOrGM(inputPtrTy.getMemorySpace()); + if (!as) + return rewriter.notifyMatchFailure(op, + "unsupported ptr address space"); + if (isEmitCTileLikeType(peeledInput.getType())) { + Type convertedInputType = + getTypeConverter()->convertType(op.getInput().getType()); + if (!convertedInputType) + return rewriter.notifyMatchFailure(op, + "failed to convert input ptr type"); + source = rewriter + .create( + op.getLoc(), convertedInputType, "PTOAS__TILE_DATA", + ArrayAttr{}, ArrayAttr{}, ValueRange{peeledInput}) + .getResult(0); + } else { + source = materializeAddressAsPointer(rewriter, op.getLoc(), + peeledInput, *as, elemTok); + } + } + } + if (!emitc::isSupportedEmitCType(source.getType())) + return rewriter.notifyMatchFailure(op, + "unsupported castptr integer source"); + auto templateArgs = rewriter.getArrayAttr( + {emitc::OpaqueAttr::get(rewriter.getContext(), + cast(convertedResultType) + .getValue())}); + auto cast = rewriter.create( + op.getLoc(), convertedResultType, "reinterpret_cast", ArrayAttr{}, + templateArgs, ValueRange{source}); + rewriter.replaceOp(op, cast.getResult(0)); + return success(); + } + + if (emitc::isSupportedEmitCType(input.getType()) && + emitc::isSupportedEmitCType(convertedResultType)) { + rewriter.replaceOpWithNewOp(op, convertedResultType, input); + return success(); + } + + return rewriter.notifyMatchFailure(op, "unsupported castptr conversion"); + } +}; + struct PointerCastConversion : public OpConversionPattern { static bool getIndexConst(Value v, int64_t &out) { if (auto cst = v.getDefiningOp()) { @@ -13063,7 +13164,8 @@ struct PTOAllocTileToEmitC static FailureOr createEmitCTileVariable(ConversionPatternRewriter &rewriter, Location loc, const TypeConverter *typeConverter, - pto::TileBufType tileTy) { + pto::TileBufType tileTy, + bool initializeDynamicValidToShape = false) { auto tileTypeString = getEmitCTileTypeString(tileTy); if (!tileTypeString) return failure(); @@ -13072,6 +13174,25 @@ createEmitCTileVariable(ConversionPatternRewriter &rewriter, Location loc, if (!convertedTy) convertedTy = emitc::OpaqueType::get(rewriter.getContext(), *tileTypeString); + if (initializeDynamicValidToShape && tileTy.hasDynamicValid()) { + auto shape = tileTy.getShape(); + if (shape.size() != 2 || llvm::is_contained(shape, ShapedType::kDynamic)) + return failure(); + Type i32Ty = emitc::OpaqueType::get(rewriter.getContext(), "int32_t"); + pto::BLayout blayout = getTileBufBLayoutValue(tileTy.getConfigAttr()); + SmallVector constructorArgs; + constructorArgs.push_back(makeEmitCIntConstant( + rewriter, loc, i32Ty, + renderTileTemplateDim(shape[0], tileTy.getElementType(), blayout, 0))); + constructorArgs.push_back(makeEmitCIntConstant( + rewriter, loc, i32Ty, + renderTileTemplateDim(shape[1], tileTy.getElementType(), blayout, 1))); + return rewriter + .create(loc, convertedTy, *tileTypeString, + ArrayAttr{}, ArrayAttr{}, constructorArgs) + .getResult(0); + } + Value tile = rewriter .create( loc, getEmitCVariableResultType(convertedTy), @@ -13080,6 +13201,27 @@ createEmitCTileVariable(ConversionPatternRewriter &rewriter, Location loc, return loadEmitCVariableIfNeeded(rewriter, loc, tile); } +struct PTODeclareTileToEmitC + : public OpConversionPattern { + using OpConversionPattern::OpConversionPattern; + + LogicalResult matchAndRewrite(pto::DeclareTileOp op, OpAdaptor adaptor, + ConversionPatternRewriter &rewriter) const override { + (void)adaptor; + auto tileType = dyn_cast(op.getTile().getType()); + if (!tileType) + return rewriter.notifyMatchFailure(op, "expected a tile_buf result"); + FailureOr tile = createEmitCTileVariable( + rewriter, op.getLoc(), getTypeConverter(), tileType, + /*initializeDynamicValidToShape=*/true); + if (failed(tile)) + return rewriter.notifyMatchFailure( + op, "only rank-2 declare_tile handles can be converted to EmitC"); + rewriter.replaceOp(op, *tile); + return success(); + } +}; + struct PTOTReshapeToEmitC : public OpConversionPattern { using OpConversionPattern::OpConversionPattern; @@ -13160,23 +13302,18 @@ struct PTOTileBufAddrToEmitC : public OpConversionPattern { LogicalResult matchAndRewrite(pto::TileBufAddrOp op, OpAdaptor adaptor, ConversionPatternRewriter &rewriter) const override { - auto ptrTy = dyn_cast(op.getResult().getType()); - if (!ptrTy) - return failure(); - Value src = peelUnrealized(adaptor.getSrc()); + Type dstTy = getTypeConverter()->convertType(op.getResult().getType()); + if (!dstTy) + return failure(); if (isEmitCTileLikeType(src.getType())) { rewriter.replaceOpWithNewOp( - op, - TypeRange{getTypeConverter()->convertType(op.getResult().getType())}, + op, TypeRange{dstTy}, "PTOAS__TILE_DATA", ArrayAttr{}, ArrayAttr{}, ValueRange{src}); return success(); } - Type dstTy = getTypeConverter()->convertType(op.getResult().getType()); - if (!dstTy) - return failure(); rewriter.replaceOpWithNewOp(op, dstTy, src); return success(); } @@ -14032,6 +14169,7 @@ static void populatePTOToEmitCPatterns(RewritePatternSet &patterns, PTOArch targetArch) { patterns.add(typeConverter, ctx); patterns.add(typeConverter, ctx); + patterns.add(typeConverter, ctx); patterns.add(typeConverter, ctx); patterns.add(typeConverter, ctx); patterns.add(typeConverter, ctx); @@ -14145,7 +14283,7 @@ static void populatePTOToEmitCPatterns(RewritePatternSet &patterns, patterns.add(typeConverter, ctx); patterns.add(typeConverter, ctx); patterns.add(typeConverter, ctx); - patterns.add(typeConverter, ctx); + patterns.add(typeConverter, ctx); patterns.add(op)) { + found = true; + return WalkResult::interrupt(); + } + return WalkResult::advance(); + }); + (void)result; + return found; +} + +static bool hasMigratedTileNativeOp(func::FuncOp func) { + bool found = false; + auto result = func.walk([&](Operation *op) { + if (isa(op)) { + found = true; + return WalkResult::interrupt(); + } + return WalkResult::advance(); + }); + (void)result; + return found; +} + constexpr size_t kTileRank2D = 2; -constexpr size_t kRowDimensionIndex = 0; -constexpr size_t kColumnDimensionIndex = 1; constexpr unsigned kShapeVectorInlineCapacity = 4; constexpr unsigned kOperationVectorInlineCapacity = 8; @@ -111,58 +178,6 @@ using SmallInlineVector = SmallVector; template using DefaultInlineVector = SmallVector; -// ============================================================================= -// Helper: Metadata Backtracking (核心机制) -// ============================================================================= -// 从一个 MemRef Value 向上回溯，找到它绑定的 TileBufConfig。 -// 这解决了 "Type Erasure" 问题：memref 类型本身不包含 config，但 SSA 定义链包含。 -static mlir::pto::TileBufConfigAttr lookupConfig(Value v) { - // 1. 最直接的情况：它就是 bind_tile 的结果 - if (auto bind = v.getDefiningOp()) { - return bind.getConfig(); - } - // PointerCastOp can also carry tile metadata (used when alloc_tile specifies - // an explicit address). - if (auto pc = v.getDefiningOp()) { - if (auto cfg = pc.getConfig()) - return *cfg; - return {}; - } - - // 2. 穿透 View 操作 (SubView, Cast 等) 向上查找 - if (auto subview = v.getDefiningOp()) { - return lookupConfig(subview.getSource()); - } - if (auto cast = v.getDefiningOp()) { - return lookupConfig(cast.getSource()); - } - if (auto cast = v.getDefiningOp()) { - return lookupConfig(cast.getSource()); - } - if (auto slot = v.getDefiningOp()) { - return lookupConfig(slot.getSource()); - } - - // 3. pto.fusion_region result 本身不携带 config；作为兜底情况，沿着 - // pto.yield 回溯到 region 内真正的 tile handle/memref 定义链继续查找。 - if (auto regionResult = dyn_cast(v)) { - if (auto fusionRegion = - dyn_cast(regionResult.getOwner())) { - auto yieldOp = dyn_cast( - fusionRegion.getBody().front().getTerminator()); - if (!yieldOp) - return {}; - unsigned resultIndex = regionResult.getResultNumber(); - if (resultIndex >= yieldOp.getNumOperands()) - return {}; - return lookupConfig(yieldOp.getOperand(resultIndex)); - } - } - - // 如果追溯到 BlockArgument (函数参数) 或其他无法穿透的 Op，则返回空 - return {}; -} - // ============================================================================= // Helper: Valid dims backtracking (v_row / v_col) // ============================================================================= @@ -236,27 +251,6 @@ static OpTy replaceOpWithClonedAttrs(IRRewriter &rewriter, Operation *op, return newOp; } -static Value resolveTileBufViewLikeSource(Value src) { - if (isa(src.getType())) - return src; - - if (auto regionResult = dyn_cast(src)) { - if (auto fusionRegion = - dyn_cast(regionResult.getOwner())) { - auto yieldOp = dyn_cast( - fusionRegion.getBody().front().getTerminator()); - if (!yieldOp) - return Value(); - unsigned resultIndex = regionResult.getResultNumber(); - if (resultIndex >= yieldOp.getNumOperands()) - return Value(); - return resolveTileBufViewLikeSource(yieldOp.getOperand(resultIndex)); - } - } - - return Value(); -} - static LogicalResult synthesizeMissingTQuantTmpOps(func::FuncOp func, MLIRContext *ctx) { if (mlir::pto::getTargetArch(func.getOperation()) == mlir::pto::PTOArch::A5) @@ -617,32 +611,6 @@ static SmallVector computeCompactStrides(ArrayRef shape) { return strides; } -static void materializeStaticValidDims(IRRewriter &rewriter, Location loc, - mlir::pto::TileBufType tbTy, Value &vRow, - Value &vCol) { - ArrayRef validShape = tbTy.getValidShape(); - if (tbTy.hasDynamicValid()) - return; - if (!validShape.empty() && validShape[kRowDimensionIndex] >= 0) - vRow = makeIndexConstant(rewriter, loc, validShape[kRowDimensionIndex]); - if (validShape.size() >= kTileRank2D && - validShape[kColumnDimensionIndex] >= 0) - vCol = makeIndexConstant(rewriter, loc, validShape[kColumnDimensionIndex]); -} - -static bool checkMultipleOf(Operation *op, int64_t value, int64_t divisor, - StringRef label) { - if (divisor <= 0) { - op->emitError("boxed layout requires positive divisor for ") << label; - return false; - } - if (value % divisor == 0) - return true; - op->emitError("boxed layout requires ") - << label << " multiple of " << divisor << ", got " << value; - return false; -} - // 确保 Value 是 Index 类型 static Value ensureIndex(IRRewriter &rewriter, Location loc, Value v, Operation *anchorOp) { @@ -692,31 +660,6 @@ static bool foldAddPtrChainIntoOffset(IRRewriter &rewriter, Location loc, return folded; } -static Value clampSubViewValidDim(IRRewriter &rewriter, Location loc, - Value explicitValid, int64_t size, - int64_t inferredValid, Operation *anchorOp) { - if (!explicitValid) { - // No explicit valid operand: take the valid extent the result type - // declares. For an ordinary subview this equals `size`; for an empty - // tail/no-op-replay tile the type carries 0, which must survive to - // bind_tile rather than being widened back to `size`. A dynamic declared - // extent is materialized via markForceDynamicValidShape, so fall back to - // `size` here. - int64_t fallback = inferredValid >= 0 ? inferredValid : size; - return rewriter.create(loc, fallback); - } - - Value sizeVal = rewriter.create(loc, size); - int64_t cst = 0; - if (getConstIndexValue(explicitValid, cst)) - return rewriter.create(loc, std::min(cst, size)); - - Value v = ensureIndex(rewriter, loc, explicitValid, anchorOp); - Value lt = rewriter.create(loc, arith::CmpIPredicate::slt, v, - sizeVal); - return rewriter.create(loc, lt, v, sizeVal); -} - [[maybe_unused]] static void dumpPretty(Operation *op, llvm::raw_ostream &os) { OpPrintingFlags flags; flags.useLocalScope(); @@ -892,25 +835,6 @@ static LogicalResult reconcileFusionRegionResultTypes(func::FuncOp func) { return success(); } -static LogicalResult markLoweredSetValidShapeOps(func::FuncOp func, - MLIRContext *ctx) { - WalkResult result = func.walk([&](mlir::pto::SetValidShapeOp op) { - if (isa(op.getSource().getType())) { - if (!lookupConfig(op.getSource())) { - op.emitError( - "set_validshape requires a locally bound tile source; function " - "arguments/results are unsupported"); - return WalkResult::interrupt(); - } - op->setAttr(kLoweredSetValidShapeAttrName, UnitAttr::get(ctx)); - return WalkResult::advance(); - } - op->removeAttr(kLoweredSetValidShapeAttrName); - return WalkResult::advance(); - }); - return result.wasInterrupted() ? failure() : success(); -} - static void markForceDynamicValidShape(Operation *op, bool force, MLIRContext *ctx) { if (force) { @@ -939,103 +863,6 @@ static void markForceDynamicValidShape(Operation *op, bool force, func.setFunctionType(FunctionType::get(ctx, newInputs, newResults)); } -[[maybe_unused]] static LogicalResult lowerAllocTileOps(func::FuncOp func, MLIRContext *ctx) { - DefaultInlineVector allocTiles; - func.walk([&](mlir::pto::AllocTileOp op) { allocTiles.push_back(op); }); - - for (auto op : allocTiles) { - IRRewriter rewriter(ctx); - rewriter.setInsertionPoint(op); - Location loc = op.getLoc(); - - auto tbTy = dyn_cast(op.getResult().getType()); - if (!tbTy) - continue; - - SmallInlineVector shape(tbTy.getShape().begin(), - tbTy.getShape().end()); - Type elemTy = tbTy.getElementType(); - SmallVector strides = buildTileMemRefStrides(tbTy); - - auto targetLayout = - StridedLayoutAttr::get(ctx, ShapedType::kDynamic, strides); - auto targetType = - MemRefType::get(shape, elemTy, targetLayout, tbTy.getMemorySpace()); - - Value vRow = op.getValidRow(); - Value vCol = op.getValidCol(); - materializeStaticValidDims(rewriter, loc, tbTy, vRow, vCol); - - auto configAttr = tbTy.getConfigAttr(); - if (!configAttr) - configAttr = pto::TileBufConfigAttr::getDefault(ctx); - - if (Value addr = op.getAddr()) { - auto pc = rewriter.create( - loc, targetType, ValueRange{addr}, vRow ? vRow : Value(), - vCol ? vCol : Value(), configAttr); - markForceDynamicValidShape(pc, tbTy.hasDynamicValid(), ctx); - auto bindOp = rewriter.create( - loc, targetType, pc.getResult(), vRow ? vRow : Value(), - vCol ? vCol : Value(), configAttr); - markForceDynamicValidShape(bindOp, tbTy.hasDynamicValid(), ctx); - rewriter.replaceOp(op, bindOp.getResult()); - continue; - } - - auto allocLayout = StridedLayoutAttr::get(ctx, 0, strides); - auto allocType = - MemRefType::get(shape, elemTy, allocLayout, tbTy.getMemorySpace()); - Value alloc = rewriter.create(loc, allocType); - auto bindOp = rewriter.create( - loc, targetType, alloc, vRow ? vRow : Value(), vCol ? vCol : Value(), - configAttr); - markForceDynamicValidShape(bindOp, tbTy.hasDynamicValid(), ctx); - rewriter.replaceOp(op, bindOp.getResult()); - } - return success(); -} - -[[maybe_unused]] static LogicalResult lowerDeclareTileOps(func::FuncOp func, MLIRContext *ctx) { - DefaultInlineVector declaredTiles; - func.walk([&](mlir::pto::DeclareTileOp op) { declaredTiles.push_back(op); }); - - for (auto op : declaredTiles) { - IRRewriter rewriter(ctx); - rewriter.setInsertionPoint(op); - Location loc = op.getLoc(); - - auto tbTy = dyn_cast(op.getTile().getType()); - if (!tbTy) { - op.emitError("declare_tile result must be tile_buf type"); - return failure(); - } - - auto targetType = dyn_cast(convertPTOTypeToMemRef(tbTy)); - if (!targetType) { - op.emitError("failed to convert declare_tile result to memref type"); - return failure(); - } - - auto configAttr = tbTy.getConfigAttr(); - if (!configAttr) - configAttr = pto::TileBufConfigAttr::getDefault(ctx); - - Value vRow; - Value vCol; - materializeStaticValidDims(rewriter, loc, tbTy, vRow, vCol); - - auto declaredMemRef = - rewriter.create(loc, targetType); - auto bindOp = rewriter.create( - loc, targetType, declaredMemRef.getResult(), vRow ? vRow : Value(), - vCol ? vCol : Value(), configAttr); - markForceDynamicValidShape(bindOp, tbTy.hasDynamicValid(), ctx); - rewriter.replaceOp(op, bindOp.getResult()); - } - return success(); -} - static Value castIndexToI64(IRRewriter &rewriter, Location loc, Value value) { Type i64Ty = rewriter.getI64Type(); if (value.getType() == i64Ty) @@ -1250,6 +1077,73 @@ static LogicalResult lowerPtrLikeTileBufAddrOps(func::FuncOp func, return success(); } +static LogicalResult bridgeMemRefOperandsToExternalPtrCallees(ModuleOp mod, + func::FuncOp func, + MLIRContext *ctx) { + SmallVector callOps; + func.walk([&](func::CallOp callOp) { callOps.push_back(callOp); }); + + for (auto callOp : callOps) { + auto callee = mod.lookupSymbol(callOp.getCallee()); + if (!callee || !callee.isExternal()) + continue; + + FunctionType calleeType = callee.getFunctionType(); + if (calleeType.getNumInputs() != callOp.getNumOperands()) { + callOp.emitError("callee signature does not match call operands"); + return failure(); + } + + IRRewriter rewriter(ctx); + rewriter.setInsertionPoint(callOp); + for (auto [index, expectedType] : + llvm::enumerate(calleeType.getInputs())) { + Value operand = callOp.getOperand(index); + if (!isa(expectedType) || + !isa(operand.getType())) + continue; + + auto castOp = rewriter.create( + callOp.getLoc(), expectedType, operand); + callOp->setOperand(index, castOp.getResult()); + } + } + return success(); +} + +static LogicalResult bridgeCallOperandsToConvertedCallees(ModuleOp mod, + MLIRContext *ctx) { + SmallVector callOps; + mod.walk([&](func::CallOp callOp) { callOps.push_back(callOp); }); + + for (func::CallOp callOp : callOps) { + auto callee = mod.lookupSymbol(callOp.getCallee()); + if (!callee || callee.isExternal()) + continue; + + FunctionType calleeType = callee.getFunctionType(); + if (calleeType.getNumInputs() != callOp.getNumOperands()) { + callOp.emitError("callee signature does not match call operands"); + return failure(); + } + + IRRewriter rewriter(ctx); + rewriter.setInsertionPoint(callOp); + for (auto [index, expectedType] : + llvm::enumerate(calleeType.getInputs())) { + Value operand = callOp.getOperand(index); + if (operand.getType() == expectedType) + continue; + + auto bridge = rewriter.create( + callOp.getLoc(), expectedType, operand); + callOp->setOperand(index, bridge.getResult(0)); + } + } + + return success(); +} + [[maybe_unused]] static LogicalResult lowerTensorViewDimOps(func::FuncOp func, MLIRContext *ctx) { DefaultInlineVector tvDims; func.walk([&](mlir::pto::GetTensorViewDimOp op) { tvDims.push_back(op); }); @@ -1386,212 +1280,6 @@ static LogicalResult lowerPartitionViewOps(func::FuncOp func, MLIRContext *ctx) return success(); } -static LogicalResult lowerSubViewOps(func::FuncOp func, MLIRContext *ctx) { - DefaultInlineVector subViews; - func.walk([&](mlir::pto::SubViewOp op) { subViews.push_back(op); }); - - for (auto op : subViews) { - IRRewriter rewriter(ctx); - rewriter.setInsertionPoint(op); - Location loc = op.getLoc(); - auto resultTileTy = - dyn_cast(op.getResult().getType()); - Value src = op->getOperand(0); - auto srcMrTy = dyn_cast(src.getType()); - if (!srcMrTy) { - op.emitError("pto.subview source must be lowered to memref first"); - return failure(); - } - - ArrayAttr sizeAttr = op.getSizes(); - SmallVector staticSizes; - SmallVector mixedSizes; - for (Attribute attr : sizeAttr) { - int64_t size = cast(attr).getInt(); - staticSizes.push_back(size); - mixedSizes.push_back(rewriter.getIndexAttr(size)); - } - - SmallVector mixedOffsets; - for (Value offset : op.getOffsets()) { - appendMixedIndex(rewriter, loc, offset, op, mixedOffsets); - } - - auto configAttr = lookupConfig(src); - if (!configAttr) - configAttr = pto::TileBufConfigAttr::getDefault(ctx); - - TileLayoutInfo layoutInfo; - if (!computeTileLayoutInfo(configAttr, srcMrTy.getElementType(), - srcMrTy.getShape(), layoutInfo)) { - op.emitError("unsupported tile layout for pto.subview"); - return failure(); - } - - if (layoutInfo.boxed) { - if (staticSizes.size() != kTileRank2D || - op.getOffsets().size() != kTileRank2D) { - op.emitError("boxed layout subview expects 2D sizes/offsets"); - return failure(); - } - if (!checkMultipleOf(op, staticSizes[0], layoutInfo.innerRows, "row size") || - !checkMultipleOf(op, staticSizes[1], layoutInfo.innerCols, "col size")) { - return failure(); - } - - int64_t off0 = 0; - int64_t off1 = 0; - bool off0Const = getConstIndexValue(op.getOffsets()[0], off0); - bool off1Const = getConstIndexValue(op.getOffsets()[1], off1); - if (off0Const && - !checkMultipleOf(op, off0, layoutInfo.innerRows, "row offset")) { - return failure(); - } - if (off1Const && - !checkMultipleOf(op, off1, layoutInfo.innerCols, "col offset")) { - return failure(); - } - - } - - SmallVector srcStrides; - int64_t srcOffset = ShapedType::kDynamic; - if (failed(mlir::pto::getPTOMemRefStridesAndOffset(srcMrTy, srcStrides, - srcOffset))) - srcStrides = computeCompactStrides(srcMrTy.getShape()); - - // Keep parent physical shape + strides for bound tile semantics. - auto resultLayout = - StridedLayoutAttr::get(ctx, ShapedType::kDynamic, srcStrides); - auto parentShape = srcMrTy.getShape(); - auto resultMemRefType = - MemRefType::get(parentShape, srcMrTy.getElementType(), resultLayout, - srcMrTy.getMemorySpace()); - - // Intermediate memref.subview keeps logical subview size. - auto subViewMemRefType = - MemRefType::get(staticSizes, srcMrTy.getElementType(), resultLayout, - srcMrTy.getMemorySpace()); - - SmallVector mixedStrides(staticSizes.size(), - rewriter.getIndexAttr(1)); - auto sv = rewriter.create(loc, subViewMemRefType, src, - mixedOffsets, mixedSizes, - mixedStrides); - - // When a valid operand is omitted, fall back to the extent the result type - // declares (which the verifier pins to either `sizes` or an empty 0 marker) - // rather than the physical subview size, so a no-op-replay v_row/v_col=0 - // survives lowering. - ArrayRef resultValid = - resultTileTy ? resultTileTy.getValidShape() : ArrayRef{}; - auto inferredValidDim = [&](unsigned d) -> int64_t { - return d < resultValid.size() ? resultValid[d] : ShapedType::kDynamic; - }; - - Value vRow; - Value vCol; - if (!staticSizes.empty()) - vRow = clampSubViewValidDim(rewriter, loc, op.getValidRow(), - staticSizes[0], inferredValidDim(0), op); - if (staticSizes.size() > 1) - vCol = clampSubViewValidDim(rewriter, loc, op.getValidCol(), - staticSizes[1], inferredValidDim(1), op); - - auto bindOp = rewriter.create( - loc, resultMemRefType, sv.getResult(), vRow ? vRow : Value(), - vCol ? vCol : Value(), configAttr); - markForceDynamicValidShape(bindOp, - resultTileTy && resultTileTy.hasDynamicValid(), - ctx); - bindOp->setAttr("pto.view_semantics", rewriter.getStringAttr("subview")); - rewriter.replaceOp(op, bindOp.getResult()); - } - return success(); -} - -static Value buildTileBufViewLikeValue(Operation *anchorOp, Value src, - mlir::pto::TileBufType tbTy, - StringRef viewSemantics, - MLIRContext *ctx) { - Location loc = anchorOp->getLoc(); - IRRewriter rewriter(ctx); - rewriter.setInsertionPoint(anchorOp); - - src = resolveTileBufViewLikeSource(src); - auto srcMrTy = dyn_cast_or_null(src.getType()); - if (!srcMrTy) { - anchorOp->emitError("tile_buf view op src must be lowered to memref first"); - return Value(); - } - - auto targetType = dyn_cast(convertPTOTypeToMemRef(tbTy)); - if (!targetType) { - anchorOp->emitError("failed to convert tile_buf type to memref type"); - return Value(); - } - for (int64_t dim : targetType.getShape()) { - if (dim == ShapedType::kDynamic) { - anchorOp->emitError("dynamic shapes are not supported for tile_buf view ops"); - return Value(); - } - } - - Value parentVRow; - Value parentVCol; - lookupValidDims(src, parentVRow, parentVCol); - Value vRow = parentVRow; - Value vCol = parentVCol; - materializeStaticValidDims(rewriter, loc, tbTy, vRow, vCol); - - auto configAttr = tbTy.getConfigAttr(); - if (!configAttr) - configAttr = pto::TileBufConfigAttr::getDefault(ctx); - - auto bindOp = rewriter.create( - loc, targetType, src, vRow ? vRow : Value(), vCol ? vCol : Value(), - configAttr); - markForceDynamicValidShape(bindOp, tbTy.hasDynamicValid(), ctx); - if (!viewSemantics.empty()) - bindOp->setAttr("pto.view_semantics", rewriter.getStringAttr(viewSemantics)); - return bindOp.getResult(); -} - -static LogicalResult lowerTileBufViewLikeOps(func::FuncOp func, MLIRContext *ctx) { - DefaultInlineVector reshapes; - func.walk([&](mlir::pto::TReshapeOp op) { reshapes.push_back(op); }); - for (auto op : reshapes) { - auto tbTy = dyn_cast(op.getResult().getType()); - if (!tbTy) { - op.emitError("treshape result must be tile_buf type"); - return failure(); - } - Value lowered = buildTileBufViewLikeValue(op, op->getOperand(0), tbTy, - "treshape", ctx); - if (!lowered) - return failure(); - IRRewriter rewriter(ctx); - rewriter.replaceOp(op, lowered); - } - - DefaultInlineVector bitcasts; - func.walk([&](mlir::pto::BitcastOp op) { bitcasts.push_back(op); }); - for (auto op : bitcasts) { - auto tbTy = dyn_cast(op.getResult().getType()); - if (!tbTy) { - op.emitError("bitcast result must be tile_buf type"); - return failure(); - } - Value lowered = buildTileBufViewLikeValue(op, op->getOperand(0), tbTy, - "bitcast", ctx); - if (!lowered) - return failure(); - IRRewriter rewriter(ctx); - rewriter.replaceOp(op, lowered); - } - return success(); -} - // ============================================================================= // The Pass Implementation // ============================================================================= @@ -1613,19 +1301,35 @@ struct PTOViewToMemrefPass return; } + // Keep external declarations in the authored ABI. PTOViewToMemref does + // not rewrite func.call operands for external/private helpers, so changing + // only the callee type can leave pointer-like call users mismatched. + if (func.isExternal()) + continue; + auto fnTy = func.getFunctionType(); + bool preserveTileABI = hasMigratedTileNativeOp(func); + bool tileNativeMainline = + preserveTileABI || hasTileNativeAllocationRoot(func); // ------------------------------------------------------------------ // Stage 0.10: Rewrite Function Signature // ------------------------------------------------------------------ SmallVector newInputs; - for (Type t : fnTy.getInputs()) newInputs.push_back(convertPTOTypeToMemRef(t)); + for (Type t : fnTy.getInputs()) { + newInputs.push_back(preserveTileABI && isa(t) + ? t + : convertPTOTypeToMemRef(t)); + } SmallVector newResults; - for (Type t : fnTy.getResults()) newResults.push_back(convertPTOTypeToMemRef(t)); + for (Type t : fnTy.getResults()) { + newResults.push_back(preserveTileABI && isa(t) + ? t + : convertPTOTypeToMemRef(t)); + } func.setFunctionType(FunctionType::get(ctx, newInputs, newResults)); - if (func.isExternal()) continue; Block &entry = func.front(); @@ -1636,6 +1340,11 @@ struct PTOViewToMemrefPass } } + if (failed(bridgeMemRefOperandsToExternalPtrCallees(mod, func, ctx))) { + signalPassFailure(); + return; + } + // ------------------------------------------------------------------ // Stage 0.20: lower pto.inttoptr result types to GM memrefs. // ------------------------------------------------------------------ @@ -1671,7 +1380,7 @@ struct PTOViewToMemrefPass for (unsigned i = 0; i < entry.getNumArguments(); ++i) { Type origTy = fnTy.getInputs()[i]; auto tbTy = dyn_cast(origTy); - if (!tbTy) + if (!tbTy || newInputs[i] == origTy) continue; auto configAttr = tbTy.getConfigAttr(); @@ -1696,378 +1405,25 @@ struct PTOViewToMemrefPass } // ------------------------------------------------------------------ - // Stage 0.5: lower pto.alloc_tile -> memref.alloc + pto.bind_tile + // Stage 0.5: keep pto.alloc_tile as a tile-native allocation root. + // + // Explicit-address alloc_tile already carries the physical address for + // level3. No-address alloc_tile is assigned an address by PTOPlanMemory. + // Neither case needs a pointer_cast + bind_tile detour here. // ------------------------------------------------------------------ - DefaultInlineVector allocTiles; - func.walk([&](mlir::pto::AllocTileOp op) { allocTiles.push_back(op); }); - - for (auto op : allocTiles) { - IRRewriter rewriter(ctx); - rewriter.setInsertionPoint(op); - Location loc = op.getLoc(); - - auto tbTy = dyn_cast(op.getResult().getType()); - if (!tbTy) continue; - - // 1. 获取 Shape 和 ElementType - SmallInlineVector shape(tbTy.getShape().begin(), tbTy.getShape().end()); - Type elemTy = tbTy.getElementType(); - - // 2. 计算 Strides (layout-aware when possible) - SmallVector strides; - TileLayoutInfo info; - if (computeTileLayoutInfo(tbTy.getConfigAttr(), elemTy, shape, info)) { - strides = {info.rowStride, info.colStride}; - } else { - strides.resize(shape.size()); - int64_t s = 1; - for (int i = (int)shape.size() - 1; i >= 0; --i) { - strides[i] = s; - if (shape[i] != ShapedType::kDynamic) s *= shape[i]; - } - } - - // 3. 构造 [BindTile 输出] 的动态类型 (Offset: ?) - // 这必须与 convertPTOTypeToMemRef 返回的类型一致，以便与 Subview 兼容 - auto targetLayout = - StridedLayoutAttr::get(ctx, ShapedType::kDynamic, strides); // offset = ? - auto targetType = - MemRefType::get(shape, elemTy, targetLayout, tbTy.getMemorySpace()); - - // 4. Preserve tile valid dims (v_row / v_col). - // - // `pto.alloc_tile` encodes the valid shape in the result TileBufType - // (e.g. acc tile may be rows=16 but v_row=1). The alloc op itself does - // not necessarily carry explicit operands for static valid dims, so we - // must materialize them from the type to keep them through - // tile_buf -> memref lowering. - // - // For dynamically valid tiles (validShape == [-1, -1]), preserve the - // runtime operands if present. - Value vRow = op.getValidRow(); - Value vCol = op.getValidCol(); - // TileBuf valid dims use a negative sentinel (e.g. '?' / -1), which is - // distinct from MLIR's ShapedType::kDynamic (INT64_MIN). Treat any - // negative value as dynamic here. - materializeStaticValidDims(rewriter, loc, tbTy, vRow, vCol); - - // 5. 获取 Config (保持不变) - auto configAttr = tbTy.getConfigAttr(); - if (!configAttr) configAttr = pto::TileBufConfigAttr::getDefault(ctx); - - // 6. If alloc_tile provides an explicit address, keep the original - // pointer_cast lowering intact and additionally rebind through - // pto.bind_tile. PointerCastOp continues to carry the tile metadata - // used by existing lowering paths, while BindTileOp provides the - // unified anchor EmitC uses to recover tile_buf information. - if (Value addr = op.getAddr()) { - auto pc = rewriter.create( - loc, targetType, ValueRange{addr}, vRow ? vRow : Value(), - vCol ? vCol : Value(), configAttr); - markForceDynamicValidShape(pc, tbTy.hasDynamicValid(), ctx); - auto bindOp = rewriter.create( - loc, targetType, pc.getResult(), vRow ? vRow : Value(), - vCol ? vCol : Value(), configAttr); - markForceDynamicValidShape(bindOp, tbTy.hasDynamicValid(), ctx); - rewriter.replaceOp(op, bindOp.getResult()); - continue; - } - - // 7. Otherwise allocate a concrete memref buffer and bind tile. - // memref.alloc 要求明确的 layout，不能是动态 offset。 - auto allocLayout = StridedLayoutAttr::get(ctx, 0, strides); // offset = 0 - auto allocType = MemRefType::get(shape, elemTy, allocLayout, tbTy.getMemorySpace()); - Value alloc = rewriter.create(loc, allocType); - - // BindTileOp 的 Builder 会自动处理空的 Value，将其视为静态维度 - auto bindOp = rewriter.create( - loc, targetType, alloc, vRow ? vRow : Value(), vCol ? vCol : Value(), - configAttr); - markForceDynamicValidShape(bindOp, tbTy.hasDynamicValid(), ctx); - - rewriter.replaceOp(op, bindOp.getResult()); - } // ------------------------------------------------------------------ - // Stage 0.6: lower pto.alloc_multi_tile / pto.multi_tile_get - // - // alloc_multi_tile produces a `multi_tile_buf`. We lower - // it into: - // %a = memref.alloc() {pto.multi_buffer = N : i32} : memref - // %m = pto.bind_tile %a, ... : memref -> memref - // and replace all uses of the multi_tile_buf SSA with %m. The N-way - // physical fan-out lives on the `pto.multi_buffer` attr and is - // materialized later by PTOPlanMemory. - // - // multi_tile_get consumes that memref and wraps it in - // %slot = pto.slot_marker %m[%k] : memref -> memref - // %t = pto.bind_tile %slot, ... : memref -> memref - // The slot_marker carries the user-supplied slot SSA forward so that - // PlanMemory / sync / EnableBufferSelect can identify which physical - // slot this use refers to. - // - // Ordering: alloc_multi_tile must be lowered before multi_tile_get so - // that the get's source SSA has already been rewired to a memref. + // Stage 0.6: keep pto.alloc_multi_tile / pto.multi_tile_get tile-native. + // PlanMemory writes the physical slot addresses on alloc_multi_tile; + // sync consumes the slot selector directly from multi_tile_get, and + // PTOResolveBufferSelect materializes the selected alloc_tile handle. // ------------------------------------------------------------------ - SmallVector allocMultiTiles; - func.walk( - [&](mlir::pto::AllocMultiTileOp op) { allocMultiTiles.push_back(op); }); - - for (auto op : allocMultiTiles) { - IRRewriter rewriter(ctx); - rewriter.setInsertionPoint(op); - Location loc = op.getLoc(); - - auto mtbTy = op.getResult().getType(); - if (!mtbTy) - continue; - auto tbTy = mtbTy.getSlotType(); - if (!tbTy) - continue; - - SmallVector shape(tbTy.getShape().begin(), - tbTy.getShape().end()); - Type elemTy = tbTy.getElementType(); - - // Stride / layout reuse the same logic as alloc_tile. - SmallVector strides; - TileLayoutInfo info; - if (computeTileLayoutInfo(tbTy.getConfigAttr(), elemTy, shape, info)) { - strides = {info.rowStride, info.colStride}; - } else { - strides.resize(shape.size()); - int64_t s = 1; - for (int i = (int)shape.size() - 1; i >= 0; --i) { - strides[i] = s; - if (shape[i] != ShapedType::kDynamic) - s *= shape[i]; - } - } - - auto targetLayout = - StridedLayoutAttr::get(ctx, ShapedType::kDynamic, strides); - auto targetType = - MemRefType::get(shape, elemTy, targetLayout, tbTy.getMemorySpace()); - - Value vRow = op.getValidRow(); - Value vCol = op.getValidCol(); - ArrayRef validShape = tbTy.getValidShape(); - if (!tbTy.hasDynamicValid()) { - if (validShape.size() >= 1 && validShape[0] >= 0) { - vRow = rewriter - .create( - loc, rewriter.getIndexType(), - rewriter.getIndexAttr(validShape[0])) - .getResult(); - } - if (validShape.size() >= 2 && validShape[1] >= 0) { - vCol = rewriter - .create( - loc, rewriter.getIndexType(), - rewriter.getIndexAttr(validShape[1])) - .getResult(); - } - } - - auto configAttr = tbTy.getConfigAttr(); - if (!configAttr) - configAttr = pto::TileBufConfigAttr::getDefault(ctx); - - // Level3 (caller-owned memory): an explicit base address is given and - // PlanMemory is skipped. Lay the N slots out contiguously as - // [addr, addr + slotBytes, ..., addr + (N-1)*slotBytes] and emit the - // multi-address `pto.pointer_cast` alloc anchor directly -- the same - // shape PlanMemory produces in the default pipeline (addrs are kept in - // slot order so PTOResolveBufferSelect can pick addrs[slot]). Sync and - // buffer-select then run unchanged. - if (Value addr = op.getAddr()) { - uint32_t n = mtbTy.getCount(); - int64_t elemBytes = getElemBytes(elemTy); - int64_t numElems = 1; - bool staticShape = true; - for (int64_t d : shape) { - if (d == ShapedType::kDynamic) { - staticShape = false; - break; - } - numElems *= d; - } - if (!staticShape || elemBytes <= 0) { - op.emitError("pto.alloc_multi_tile with explicit addr requires a " - "static slot shape and known element byte size"); - signalPassFailure(); - return; - } - int64_t slotBytes = numElems * elemBytes; - - SmallVector slotAddrs; - slotAddrs.reserve(n); - slotAddrs.push_back(addr); - for (uint32_t slot = 1; slot < n; ++slot) { - Value off = rewriter.create( - loc, rewriter.getI64Type(), - rewriter.getI64IntegerAttr(static_cast(slot) * - slotBytes)); - slotAddrs.push_back( - rewriter.create(loc, addr, off).getResult()); - } - - auto pc = rewriter.create( - loc, targetType, slotAddrs, vRow ? vRow : Value(), - vCol ? vCol : Value(), configAttr); - markForceDynamicValidShape(pc, tbTy.hasDynamicValid(), ctx); - - auto bindOp = rewriter.create( - loc, targetType, pc.getResult(), vRow ? vRow : Value(), - vCol ? vCol : Value(), configAttr); - markForceDynamicValidShape(bindOp, tbTy.hasDynamicValid(), ctx); - - rewriter.replaceOp(op, bindOp.getResult()); - continue; - } - - // memref.alloc with N-slot annotation. The actual N-way address - // expansion happens in PlanMemory. - auto allocLayout = StridedLayoutAttr::get(ctx, 0, strides); - auto allocType = - MemRefType::get(shape, elemTy, allocLayout, tbTy.getMemorySpace()); - auto allocOp = rewriter.create(loc, allocType); - auto i32Ty = IntegerType::get(ctx, 32); - allocOp->setAttr( - mlir::pto::kPtoMultiBufferAttrName, - IntegerAttr::get(i32Ty, static_cast(mtbTy.getCount()))); - - auto bindOp = rewriter.create( - loc, targetType, allocOp.getResult(), vRow ? vRow : Value(), - vCol ? vCol : Value(), configAttr); - markForceDynamicValidShape(bindOp, tbTy.hasDynamicValid(), ctx); - - rewriter.replaceOp(op, bindOp.getResult()); - } - - SmallVector multiTileGets; - func.walk( - [&](mlir::pto::MultiTileGetOp op) { multiTileGets.push_back(op); }); - - for (auto op : multiTileGets) { - IRRewriter rewriter(ctx); - rewriter.setInsertionPoint(op); - Location loc = op.getLoc(); - - // After alloc_multi_tile has been replaced, the source SSA value - // is a memref (no longer multi_tile_buf). We accept the raw Value. - Value srcMem = op->getOperand(0); - if (!llvm::isa(srcMem.getType())) { - op.emitError( - "multi_tile_get expects its source to have been lowered to " - "memref by alloc_multi_tile rewriting; got ") - << srcMem.getType(); - signalPassFailure(); - return; - } - auto srcMemTy = llvm::cast(srcMem.getType()); - - // Recover per-slot tile_buf info from the get op's result type - // (which still describes the per-slot tile shape, valid dims, - // config, etc.). - auto tbTy = llvm::dyn_cast(op.getResult().getType()); - if (!tbTy) { - op.emitError("multi_tile_get result must be `!pto.tile_buf<...>`"); - signalPassFailure(); - return; - } - - // The slot view aliases the source memref byte-for-byte. - auto slotMarker = rewriter.create( - loc, srcMemTy, srcMem, op.getSlot()); - - // Recover valid_row / valid_col from the per-slot tile type so - // downstream BindTile carries the same metadata as the alloc. - Value vRow; - Value vCol; - ArrayRef validShape = tbTy.getValidShape(); - if (!tbTy.hasDynamicValid()) { - if (validShape.size() >= 1 && validShape[0] >= 0) { - vRow = rewriter - .create( - loc, rewriter.getIndexType(), - rewriter.getIndexAttr(validShape[0])) - .getResult(); - } - if (validShape.size() >= 2 && validShape[1] >= 0) { - vCol = rewriter - .create( - loc, rewriter.getIndexType(), - rewriter.getIndexAttr(validShape[1])) - .getResult(); - } - } - - auto configAttr = tbTy.getConfigAttr(); - if (!configAttr) - configAttr = pto::TileBufConfigAttr::getDefault(ctx); - - // BindTile result type uses the same layout signature as in the - // alloc lowering, so subsequent subview / DMA ops see a consistent - // memref view. - SmallVector strides = buildTileMemRefStrides(tbTy); - auto targetLayout = - StridedLayoutAttr::get(ctx, ShapedType::kDynamic, strides); - auto targetType = - MemRefType::get(tbTy.getShape(), tbTy.getElementType(), targetLayout, - tbTy.getMemorySpace()); - - auto bindOp = rewriter.create( - loc, targetType, slotMarker.getResult(), vRow ? vRow : Value(), - vCol ? vCol : Value(), configAttr); - markForceDynamicValidShape(bindOp, tbTy.hasDynamicValid(), ctx); - - rewriter.replaceOp(op, bindOp.getResult()); - } // ------------------------------------------------------------------ - // Stage 0.75: lower pto.declare_tile -> pto.declare_tile_memref + - // pto.bind_tile + // Stage 0.75: keep pto.declare_tile tile-native. It is a symbolic handle + // whose address is assigned at runtime by tpop/tassign, not a static + // allocation root for PlanMemory. // ------------------------------------------------------------------ - DefaultInlineVector declaredTiles; - func.walk([&](mlir::pto::DeclareTileOp op) { declaredTiles.push_back(op); }); - - for (auto op : declaredTiles) { - IRRewriter rewriter(ctx); - rewriter.setInsertionPoint(op); - Location loc = op.getLoc(); - - auto tbTy = dyn_cast(op.getTile().getType()); - if (!tbTy) { - op.emitError("declare_tile result must be tile_buf type"); - signalPassFailure(); - return; - } - - auto targetType = dyn_cast(convertPTOTypeToMemRef(tbTy)); - if (!targetType) { - op.emitError("failed to convert declare_tile result to memref type"); - signalPassFailure(); - return; - } - - auto configAttr = tbTy.getConfigAttr(); - if (!configAttr) - configAttr = pto::TileBufConfigAttr::getDefault(ctx); - - Value vRow; - Value vCol; - materializeStaticValidDims(rewriter, loc, tbTy, vRow, vCol); - - auto declaredMemRef = - rewriter.create(loc, targetType); - auto bindOp = rewriter.create( - loc, targetType, declaredMemRef.getResult(), - vRow ? vRow : Value(), vCol ? vCol : Value(), configAttr); - markForceDynamicValidShape(bindOp, tbTy.hasDynamicValid(), ctx); - - rewriter.replaceOp(op, bindOp.getResult()); - } // ------------------------------------------------------------------ // Stage 0.8: normalize pto.tassign result type to match tile operand @@ -2101,8 +1457,13 @@ struct PTOViewToMemrefPass // ------------------------------------------------------------------ // Stage 1: Lower pto.make_tensor_view -> memref.reinterpret_cast // ------------------------------------------------------------------ + // GM view ops stay authored until PTOResolveBufferSelect when a + // function contains the migrated view family. Legacy functions keep + // the established memref lowering below. DefaultInlineVector makeViews; func.walk([&](mlir::pto::MakeTensorViewOp op) { makeViews.push_back(op); }); + if (preserveTileABI) + makeViews.clear(); for (auto op : makeViews) { IRRewriter rewriter(ctx); @@ -2195,30 +1556,21 @@ struct PTOViewToMemrefPass // ------------------------------------------------------------------ // Stage 1.3: Lower pto.partition_view -> memref.subview // ------------------------------------------------------------------ - if (failed(lowerPartitionViewOps(func, ctx))) { + if (!preserveTileABI && failed(lowerPartitionViewOps(func, ctx))) { signalPassFailure(); return; } // ------------------------------------------------------------------ - // Stage 1.35: Lower pto.subview -> memref.subview + pto.bind_tile + // Stage 1.35: pto.subview stays tile-native through memory and sync. // ------------------------------------------------------------------ - if (failed(lowerSubViewOps(func, ctx))) { - signalPassFailure(); - return; - } // ------------------------------------------------------------------ - // Stage 1.4: Lower tile_buf view-like ops (treshape/bitcast) + // Stage 1.4: treshape/bitcast stay tile-native. // ------------------------------------------------------------------ - if (failed(lowerTileBufViewLikeOps(func, ctx))) { - signalPassFailure(); - return; - } if (failed(reconcileFusionRegionResultTypes(func))) { signalPassFailure(); return; - } // ------------------------------------------------------------------ // Stage 1.5: Lower pto.get_tensor_view_stride -> strided memref metadata @@ -2263,6 +1615,7 @@ struct PTOViewToMemrefPass rewriter.create(loc, view); rewriter.replaceOp(op, metadata.getStrides()[dimIndex]); } + } // ------------------------------------------------------------------ // Stage 1.6: Fold pto.addptr chains into load/store_scalar. @@ -2406,6 +1759,7 @@ struct PTOViewToMemrefPass // Stage 3: Rewrite Compute Ops // [关键] 全面使用 op->getOperand(i) 避免 Typed Accessor Crash // ------------------------------------------------------------------ + if (!tileNativeMainline) { // --- TLoadOp [Src, Dst] --- DefaultInlineVector loads; @@ -4280,6 +3634,9 @@ struct PTOViewToMemrefPass auto srcTy = dyn_cast(src.getType()); auto tmpTy = tmp ? dyn_cast(tmp.getType()) : MemRefType(); if (!srcTy || (tmp && !tmpTy)) { + if (isa(src.getType()) && + (!tmp || isa(tmp.getType()))) + continue; op.emitError("ins/outs are not memref yet"); signalPassFailure(); return; @@ -4293,6 +3650,7 @@ struct PTOViewToMemrefPass dyn_cast_or_null( op.getProperties().printFormat)); } + } // ------------------------------------------------------------------ // Stage 4: Reconcile control-flow result types @@ -4305,13 +3663,11 @@ struct PTOViewToMemrefPass signalPassFailure(); return; } - // Mark memref-form set_validshape only after control-flow result-type - // reconciliation. Values such as scf.if results can stay tile_buf until - // this late stage. - if (failed(markLoweredSetValidShapeOps(func, ctx))) { - signalPassFailure(); - return; - } + } + + if (failed(bridgeCallOperandsToConvertedCallees(mod, ctx))) { + signalPassFailure(); + return; } // Debug Output diff --git a/lib/PTO/Transforms/SlotAffineAnalysis.cpp b/lib/PTO/Transforms/SlotAffineAnalysis.cpp index ab37697c6f..cf633b1795 100644 --- a/lib/PTO/Transforms/SlotAffineAnalysis.cpp +++ b/lib/PTO/Transforms/SlotAffineAnalysis.cpp @@ -30,6 +30,8 @@ Value findSlotMarkerExpr(Value v) { return {}; if (auto sm = dyn_cast(op)) return sm.getSlot(); + if (auto get = dyn_cast(op)) + return get.getSlot(); if (auto bind = dyn_cast(op)) { v = bind.getSource(); continue; diff --git a/lib/PTO/Transforms/Utils.cpp b/lib/PTO/Transforms/Utils.cpp index 44b6d1005f..4978986158 100644 --- a/lib/PTO/Transforms/Utils.cpp +++ b/lib/PTO/Transforms/Utils.cpp @@ -131,6 +131,19 @@ void setBaseMemRefTypeScope(Value val, AddressSpaceAttr targetMemScope) { std::optional GetBufferSpaceAttr(Value operand) { + if (auto tileTy = dyn_cast(operand.getType())) { + if (auto memorySpaceAttr = + dyn_cast_or_null(tileTy.getMemorySpace())) + return memorySpaceAttr; + return std::nullopt; + } + if (auto multiTy = dyn_cast(operand.getType())) { + if (auto memorySpaceAttr = dyn_cast_or_null( + multiTy.getSlotType().getMemorySpace())) + return memorySpaceAttr; + return std::nullopt; + } + if (!llvm::isa(operand.getType())) { return std::nullopt; } @@ -148,6 +161,12 @@ std::optional GetBufferSpaceAttr(Value operand) { std::optional> getOperationAliasInfo(Operation *op) { if (auto subViewOp = dyn_cast(op)) { return std::make_pair(subViewOp.getResult(), subViewOp.getViewSource()); + } else if (auto subViewOp = dyn_cast(op)) { + return std::make_pair(subViewOp.getResult(), subViewOp.getSource()); + } else if (auto bitcastOp = dyn_cast(op)) { + return std::make_pair(bitcastOp.getResult(), bitcastOp.getSrc()); + } else if (auto reshapeOp = dyn_cast(op)) { + return std::make_pair(reshapeOp.getResult(), reshapeOp.getSrc()); } else if (auto bindTileOp = dyn_cast(op)) { return std::make_pair(bindTileOp.getResult(), bindTileOp.getSource()); } else if (auto slotMarkerOp = dyn_cast(op)) { @@ -155,6 +174,8 @@ std::optional> getOperationAliasInfo(Operation *op) { // physical slot of a multi-buffer alloc. From an alias-walking // standpoint it behaves like any other view-like op. return std::make_pair(slotMarkerOp.getResult(), slotMarkerOp.getSource()); + } else if (auto multiGetOp = dyn_cast(op)) { + return std::make_pair(multiGetOp.getResult(), multiGetOp.getSource()); } else if (auto extSliceOp = dyn_cast(op)) { return std::make_pair(extSliceOp.getResult(), extSliceOp.getSource()); } else if (auto collapseShapeOp = dyn_cast(op)) { @@ -172,6 +193,9 @@ std::optional> getOperationAliasInfo(Operation *op) { return std::make_pair(reshapeOp.getResult(), reshapeOp.getViewSource()); } else if (auto castOp = dyn_cast(op)) { return std::make_pair(castOp.getResult(), castOp.getViewSource()); + } else if (auto castOp = dyn_cast(op)) { + if (castOp.getNumOperands() == 1 && castOp.getNumResults() == 1) + return std::make_pair(castOp.getResult(0), castOp.getOperand(0)); } else if (auto extractStridedMetadataOp = dyn_cast(op)) { return std::make_pair(extractStridedMetadataOp.getBaseBuffer(), diff --git a/ptodsl/tests/test_docs_as_test.py b/ptodsl/tests/test_docs_as_test.py index 6520d2f296..9d1feda740 100644 --- a/ptodsl/tests/test_docs_as_test.py +++ b/ptodsl/tests/test_docs_as_test.py @@ -67,7 +67,7 @@ class DocTestDirective: symbol: Optional[str] = None compile_kwargs: Optional[dict[str, object]] = None fixture: Optional[str] = None - files: Optional[dict[str, str]] | None = None + files: Optional[dict[str, str]] = None @dataclass(frozen=True) @@ -324,7 +324,7 @@ def parse_test_directive(block: MarkdownCodeBlock) -> DocTestDirective: return DocTestDirective(mode=mode) -def _write_directive_files(snippet_dir: Path, files: dict[str, str] | None) -> None: +def _write_directive_files(snippet_dir: Path, files: Optional[dict[str, str]]) -> None: if not files: return for relative_path, text in files.items(): @@ -351,7 +351,7 @@ def execute_source( symbol: Optional[str] = None, *, extra_namespace: Optional[dict[str, object]] = None, - source_dir: Path | None = None, + source_dir: Optional[Path] = None, ) -> dict[str, object]: source_file = block.path if source_dir is None else source_dir / "case.py" namespace: dict[str, object] = { diff --git a/ptodsl/tests/test_ptoas_frontend_verify.py b/ptodsl/tests/test_ptoas_frontend_verify.py index 57592a8355..47d4f52320 100644 --- a/ptodsl/tests/test_ptoas_frontend_verify.py +++ b/ptodsl/tests/test_ptoas_frontend_verify.py @@ -451,12 +451,13 @@ def main() -> None: "ptr_like_tile_buf_addr_probe frontend verification output should preserve the kernel symbol", ) expect( - "memref address view during PTOViewToMemref", + "pto.alloc_tile addr" in ptr_like_addr_frontend_text + and "pto.tile_buf_addr" in ptr_like_addr_frontend_text, + "ptr-like tile_buf_addr lowering should keep the tile-native address path until PTOPlanMemory materializes alloc_tile addr", ) expect( "call @consume" in ptr_like_addr_frontend_text, - "ptr-like tile_buf_addr lowering should preserve call users after converting pointer-like operands", + "ptr-like tile_buf_addr lowering should preserve call users without half-converting pointer-like operands", ) example_mlir_text = emit_example_mlir(mixed_backend_example) diff --git a/test/lit/pto/a5_unused_tmp_vec_overflow.pto b/test/lit/pto/a5_unused_tmp_vec_overflow.pto index a07f1b9588..500112c2ec 100644 --- a/test/lit/pto/a5_unused_tmp_vec_overflow.pto +++ b/test/lit/pto/a5_unused_tmp_vec_overflow.pto @@ -73,12 +73,12 @@ module { } func.func @TROWEXPANDMAX_TMP_OVERFLOW() { - %src0 = pto.alloc_tile : !pto.tile_buf - %src1 = pto.alloc_tile : !pto.tile_buf - %tmp = pto.alloc_tile : !pto.tile_buf - %dst = pto.alloc_tile : !pto.tile_buf - pto.trowexpandmax ins(%src0, %src1, %tmp : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) - outs(%dst : !pto.tile_buf) + %src0 = pto.alloc_tile : !pto.tile_buf + %src1 = pto.alloc_tile : !pto.tile_buf + %tmp = pto.alloc_tile : !pto.tile_buf + %dst = pto.alloc_tile : !pto.tile_buf + pto.trowexpandmax ins(%src0, %src1, %tmp : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) return } } diff --git a/test/lit/pto/add_carry_tile_native.pto b/test/lit/pto/add_carry_tile_native.pto new file mode 100644 index 0000000000..6b5d337527 --- /dev/null +++ b/test/lit/pto/add_carry_tile_native.pto @@ -0,0 +1,48 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @taddc_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %src2: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.taddc ins(%src0, %src1, %src2 : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @taddsc_arg( + %src0: !pto.tile_buf, + %scalar: f32, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.taddsc ins(%src0, %scalar, %src1 : !pto.tile_buf, f32, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @taddc_arg( +// NATIVE: pto.taddc ins(%arg0, %arg1, %arg2 : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) outs(%arg3 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @taddsc_arg( +// NATIVE: pto.taddsc ins(%arg0, %arg1, %arg2 : !pto.tile_buf, f32, !pto.tile_buf) outs(%arg3 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: taddc_arg( +// EMITC-COUNT-2: TADD( +// EMITC-LABEL: taddsc_arg( +// EMITC: TADDS( +// EMITC: TADD( diff --git a/test/lit/pto/addmul_scalar_tile_native.pto b/test/lit/pto/addmul_scalar_tile_native.pto new file mode 100644 index 0000000000..65f0a83039 --- /dev/null +++ b/test/lit/pto/addmul_scalar_tile_native.pto @@ -0,0 +1,54 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tadds_arg( + %src: !pto.tile_buf, + %scalar: f32, + %dst: !pto.tile_buf) { + pto.tadds ins(%src, %scalar : !pto.tile_buf, f32) + outs(%dst : !pto.tile_buf) + pto.tadds ins(%src, %scalar : !pto.tile_buf, f32) + outs(%src : !pto.tile_buf) + return + } + + func.func private @tmuls_arg( + %src: !pto.tile_buf, + %scalar: f32, + %dst: !pto.tile_buf) { + pto.tmuls ins(%src, %scalar : !pto.tile_buf, f32) + outs(%dst : !pto.tile_buf) + pto.tmuls ins(%src, %scalar : !pto.tile_buf, f32) + outs(%src : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tadds_arg( +// NATIVE-SAME: %arg0: !pto.tile_buf +// NATIVE-SAME: %arg1: f32 +// NATIVE-SAME: %arg2: !pto.tile_buf +// NATIVE: pto.tadds ins(%arg0, %arg1 : !pto.tile_buf, f32) outs(%arg2 : !pto.tile_buf) +// NATIVE: pto.tadds ins(%arg0, %arg1 : !pto.tile_buf, f32) outs(%arg0 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tmuls_arg( +// NATIVE: pto.tmuls ins(%arg0, %arg1 : !pto.tile_buf, f32) outs(%arg2 : !pto.tile_buf) +// NATIVE: pto.tmuls ins(%arg0, %arg1 : !pto.tile_buf, f32) outs(%arg0 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tadds_arg( +// EMITC-COUNT-2: TADDS( +// EMITC-LABEL: tmuls_arg( +// EMITC-COUNT-2: TMULS( diff --git a/test/lit/pto/addmul_tile_native.pto b/test/lit/pto/addmul_tile_native.pto new file mode 100644 index 0000000000..a26a71029e --- /dev/null +++ b/test/lit/pto/addmul_tile_native.pto @@ -0,0 +1,54 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tadd_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tadd ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + pto.tadd ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%src0 : !pto.tile_buf) + return + } + + func.func private @tmul_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tmul ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + pto.tmul ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%src0 : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tadd_arg( +// NATIVE-SAME: %arg0: !pto.tile_buf +// NATIVE-SAME: %arg1: !pto.tile_buf +// NATIVE-SAME: %arg2: !pto.tile_buf +// NATIVE: pto.tadd ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE: pto.tadd ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg0 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tmul_arg( +// NATIVE: pto.tmul ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE: pto.tmul ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg0 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tadd_arg( +// EMITC-COUNT-2: TADD( +// EMITC-LABEL: tmul_arg( +// EMITC-COUNT-2: TMUL( diff --git a/test/lit/pto/alloc_tile_addr_alignment_invalid.pto b/test/lit/pto/alloc_tile_addr_alignment_invalid.pto index 2a94e206e0..a211d08e2f 100644 --- a/test/lit/pto/alloc_tile_addr_alignment_invalid.pto +++ b/test/lit/pto/alloc_tile_addr_alignment_invalid.pto @@ -1,4 +1,5 @@ // RUN: not ptoas --pto-level=level3 %s 2>&1 | FileCheck %s +// RUN: not ptoas --pto-level=level2 %s 2>&1 | FileCheck %s module { func.func @unaligned_alloc_tile_addr() { diff --git a/test/lit/pto/alloc_tile_addr_level12_reject.pto b/test/lit/pto/alloc_tile_addr_level12_reject.pto new file mode 100644 index 0000000000..c40ff6985e --- /dev/null +++ b/test/lit/pto/alloc_tile_addr_level12_reject.pto @@ -0,0 +1,21 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-level=level1 %s 2>&1 | FileCheck %s +// RUN: not ptoas --pto-level=level2 %s 2>&1 | FileCheck %s + +module { + func.func @reject_explicit_addr_below_level3() { + %addr = arith.constant 0 : i64 + %tile = pto.alloc_tile addr = %addr + : !pto.tile_buf + return + } +} + +// CHECK: error: unexpected 'addr' operand: only supported when --pto-level=level3 diff --git a/test/lit/pto/alloc_tile_low_precision_valid.pto b/test/lit/pto/alloc_tile_low_precision_valid.pto index ffa8240309..f385fb0290 100644 --- a/test/lit/pto/alloc_tile_low_precision_valid.pto +++ b/test/lit/pto/alloc_tile_low_precision_valid.pto @@ -18,4 +18,6 @@ module { } // CHECK: func.func @alloc_tile_low_precision_valid() -// CHECK-NOT: pto.alloc_tile +// CHECK: pto.alloc_tile : !pto.tile_buf +// CHECK: pto.alloc_tile : !pto.tile_buf +// CHECK: pto.alloc_tile : !pto.tile_buf diff --git a/test/lit/pto/alloc_tile_plan_memory_no_memref_alloc.pto b/test/lit/pto/alloc_tile_plan_memory_no_memref_alloc.pto new file mode 100644 index 0000000000..55a835120e --- /dev/null +++ b/test/lit/pto/alloc_tile_plan_memory_no_memref_alloc.pto @@ -0,0 +1,35 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level2 --pto-arch=a3 --emit-pto-ir \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=VIEW +// RUN: ptoas --pto-level=level2 --pto-arch=a3 --emit-pto-ir \ +// RUN: --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=PLAN +// RUN: ptoas --pto-level=level2 --pto-arch=a3 %s >/dev/null + +module attributes {"pto.target_arch" = "a3"} { + func.func @alloc_tile_plan_memory_no_memref_alloc() { + %src = pto.alloc_tile : !pto.tile_buf + %dst = pto.alloc_tile : !pto.tile_buf + pto.tmov ins(%src : !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } +} + +// VIEW-LABEL: func.func @alloc_tile_plan_memory_no_memref_alloc +// VIEW: pto.alloc_tile +// VIEW: pto.alloc_tile +// VIEW-NOT: memref.alloc + +// PLAN-LABEL: func.func @alloc_tile_plan_memory_no_memref_alloc +// PLAN: %[[ADDR:.*]] = arith.constant 0 : i64 +// PLAN-NEXT: {{%.*}} = pto.alloc_tile addr = %[[ADDR]] +// PLAN-NOT: pto.pointer_cast +// PLAN-NOT: pto.bind_tile +// PLAN-NOT: memref.alloc diff --git a/test/lit/pto/arg_reduction_tile_native.pto b/test/lit/pto/arg_reduction_tile_native.pto new file mode 100644 index 0000000000..3a54887913 --- /dev/null +++ b/test/lit/pto/arg_reduction_tile_native.pto @@ -0,0 +1,65 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tcolargmax_arg(%src: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tcolargmax ins(%src, %tmp : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tcolargmin_arg(%src: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tcolargmin ins(%src, %tmp : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trowargmax_arg(%src: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trowargmax ins(%src, %tmp : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trowargmin_arg(%src: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trowargmin ins(%src, %tmp : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tpartargmax_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %ai: !pto.tile_buf, %bi: !pto.tile_buf, %dst: !pto.tile_buf, %di: !pto.tile_buf) { + pto.tpartargmax ins(%a, %b, %ai, %bi : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) outs(%dst, %di : !pto.tile_buf, !pto.tile_buf) + return + } + func.func private @tpartargmin_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %ai: !pto.tile_buf, %bi: !pto.tile_buf, %dst: !pto.tile_buf, %di: !pto.tile_buf) { + pto.tpartargmin ins(%a, %b, %ai, %bi : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) outs(%dst, %di : !pto.tile_buf, !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @tcolargmax_arg +// NATIVE: pto.tcolargmax +// NATIVE-LABEL: @tcolargmin_arg +// NATIVE: pto.tcolargmin +// NATIVE-LABEL: @trowargmax_arg +// NATIVE: pto.trowargmax +// NATIVE-LABEL: @trowargmin_arg +// NATIVE: pto.trowargmin +// NATIVE-LABEL: @tpartargmax_arg +// NATIVE: pto.tpartargmax +// NATIVE-LABEL: @tpartargmin_arg +// NATIVE: pto.tpartargmin +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tcolargmax_arg( +// EMITC: TCOLARGMAX( +// EMITC-LABEL: tcolargmin_arg( +// EMITC: TCOLARGMIN( +// EMITC-LABEL: trowargmax_arg( +// EMITC: TROWARGMAX( +// EMITC-LABEL: trowargmin_arg( +// EMITC: TROWARGMIN( +// EMITC-LABEL: tpartargmax_arg( +// EMITC: TPARTARGMAX( +// EMITC-LABEL: tpartargmin_arg( +// EMITC: TPARTARGMIN( diff --git a/test/lit/pto/axpy_dequant_tile_native.pto b/test/lit/pto/axpy_dequant_tile_native.pto new file mode 100644 index 0000000000..d43dbc27da --- /dev/null +++ b/test/lit/pto/axpy_dequant_tile_native.pto @@ -0,0 +1,33 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @taxpy_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf, %scalar: f16) { + pto.taxpy ins(%src, %scalar : !pto.tile_buf, f16) outs(%dst : !pto.tile_buf) + return + } + func.func private @tdequant_arg(%src: !pto.tile_buf, %scale: !pto.tile_buf, %offset: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tdequant ins(%src, %scale, %offset : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @taxpy_arg +// NATIVE: pto.taxpy +// NATIVE-LABEL: @tdequant_arg +// NATIVE: pto.tdequant +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: taxpy_arg( +// EMITC: TAXPY( +// EMITC-LABEL: tdequant_arg( +// EMITC: TDEQUANT< diff --git a/test/lit/pto/basic_float_tile_native.pto b/test/lit/pto/basic_float_tile_native.pto new file mode 100644 index 0000000000..35de5643af --- /dev/null +++ b/test/lit/pto/basic_float_tile_native.pto @@ -0,0 +1,62 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @trelu_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trelu ins(%src : !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trecip_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trecip ins(%src : !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tsqrt_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tsqrt ins(%src : !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tfmod_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tfmod ins(%a, %b : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tfmods_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf, %scalar: f32) { + pto.tfmods ins(%src, %scalar : !pto.tile_buf, f32) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @trelu_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.trelu +// NATIVE-LABEL: @trecip_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.trecip +// NATIVE-LABEL: @tsqrt_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.tsqrt +// NATIVE-LABEL: @tfmod_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.tfmod +// NATIVE-LABEL: @tfmods_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.tfmods +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: trelu_arg( +// EMITC: TRELU( +// EMITC-LABEL: trecip_arg( +// EMITC: TRECIP( +// EMITC-LABEL: tsqrt_arg( +// EMITC: TSQRT( +// EMITC-LABEL: tfmod_arg( +// EMITC: TFMOD( +// EMITC-LABEL: tfmods_arg( +// EMITC: TFMODS( diff --git a/test/lit/pto/bitcast_tile_native.pto b/test/lit/pto/bitcast_tile_native.pto new file mode 100644 index 0000000000..82aa068367 --- /dev/null +++ b/test/lit/pto/bitcast_tile_native.pto @@ -0,0 +1,40 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o - 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE + +module attributes {pto.target_arch = "a5"} { + func.func private @bitcast_arg( + %src: !pto.tile_buf) + -> !pto.tile_buf { + %result = pto.bitcast %src + : !pto.tile_buf + -> !pto.tile_buf + return %result : !pto.tile_buf + } +} + +// NATIVE-LABEL: func.func private @bitcast_arg( +// NATIVE-SAME: !pto.tile_buf +// NATIVE: %[[RESULT:.*]] = pto.bitcast %arg0 +// NATIVE-SAME: !pto.tile_buf -> !pto.tile_buf +// NATIVE: return %[[RESULT]] : !pto.tile_buf +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile diff --git a/test/lit/pto/castptr_emitc_tile_and_int.pto b/test/lit/pto/castptr_emitc_tile_and_int.pto new file mode 100644 index 0000000000..4bd3f4d7ee --- /dev/null +++ b/test/lit/pto/castptr_emitc_tile_and_int.pto @@ -0,0 +1,29 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --cann-output-version=9.0.0 --pto-arch=a5 --pto-level=level3 %s -o - 2>&1 | FileCheck %s + +module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { + func.func @castptr_ptr_to_int(%src: memref<1xf32, #pto.address_space>) + attributes {pto.kernel} { + func.call @helper(%src) : (memref<1xf32, #pto.address_space>) -> i64 + return + } + + func.func private @helper(%src: memref<1xf32, #pto.address_space>) -> i64 { + %ptr = builtin.unrealized_conversion_cast %src + : memref<1xf32, #pto.address_space> to !pto.ptr + %addr = pto.castptr %ptr : !pto.ptr -> i64 + return %addr : i64 + } +} + +// CHECK-LABEL: static AICORE int64_t helper(__ubuf__ float* +// CHECK: reinterpret_cast +// CHECK-LABEL: AICORE void castptr_ptr_to_int(__ubuf__ float* +// CHECK: helper( diff --git a/test/lit/pto/col_minmax_tile_native.pto b/test/lit/pto/col_minmax_tile_native.pto new file mode 100644 index 0000000000..5be86f1bf5 --- /dev/null +++ b/test/lit/pto/col_minmax_tile_native.pto @@ -0,0 +1,43 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tcolmax_arg( + %src: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tcolmax ins(%src : !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tcolmin_arg( + %src: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tcolmin ins(%src : !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tcolmax_arg( +// NATIVE: pto.tcolmax ins(%arg0 : !pto.tile_buf) outs(%arg1 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tcolmin_arg( +// NATIVE: pto.tcolmin ins(%arg0 : !pto.tile_buf) outs(%arg1 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tcolmax_arg( +// EMITC: TCOLMAX( +// EMITC-LABEL: tcolmin_arg( +// EMITC: TCOLMIN( diff --git a/test/lit/pto/colexpand_tile_native.pto b/test/lit/pto/colexpand_tile_native.pto new file mode 100644 index 0000000000..a30a107fdd --- /dev/null +++ b/test/lit/pto/colexpand_tile_native.pto @@ -0,0 +1,111 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tcolexpand_arg( + %src: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tcolexpand ins(%src : !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tcolexpandmul_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tcolexpandmul ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tcolexpandmax_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tcolexpandmax ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tcolexpandmin_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tcolexpandmin ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tcolexpandadd_arg( + %src0: !pto.tile_buf, %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tcolexpandadd ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tcolexpandsub_arg( + %src0: !pto.tile_buf, %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tcolexpandsub ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tcolexpanddiv_arg( + %src0: !pto.tile_buf, %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tcolexpanddiv ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tcolexpandexpdif_arg( + %src0: !pto.tile_buf, %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tcolexpandexpdif ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tcolexpand_arg( +// NATIVE: pto.tcolexpand ins(%arg0 : !pto.tile_buf) outs(%arg1 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tcolexpandmul_arg( +// NATIVE: pto.tcolexpandmul ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tcolexpandmax_arg( +// NATIVE: pto.tcolexpandmax ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tcolexpandmin_arg( +// NATIVE: pto.tcolexpandmin ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tcolexpandadd_arg( +// NATIVE: pto.tcolexpandadd +// NATIVE-LABEL: func.func private @tcolexpandsub_arg( +// NATIVE: pto.tcolexpandsub +// NATIVE-LABEL: func.func private @tcolexpanddiv_arg( +// NATIVE: pto.tcolexpanddiv +// NATIVE-LABEL: func.func private @tcolexpandexpdif_arg( +// NATIVE: pto.tcolexpandexpdif +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tcolexpand_arg( +// EMITC: TCOLEXPAND( +// EMITC-LABEL: tcolexpandmul_arg( +// EMITC: TCOLEXPANDMUL( +// EMITC-LABEL: tcolexpandmax_arg( +// EMITC: TCOLEXPANDMAX( +// EMITC-LABEL: tcolexpandmin_arg( +// EMITC: TCOLEXPANDMIN( +// EMITC-LABEL: tcolexpandadd_arg( +// EMITC: TCOLEXPANDADD( +// EMITC-LABEL: tcolexpandsub_arg( +// EMITC: TCOLEXPANDSUB( +// EMITC-LABEL: tcolexpanddiv_arg( +// EMITC: TCOLEXPANDDIV( +// EMITC-LABEL: tcolexpandexpdif_arg( +// EMITC: TCOLEXPANDEXPDIF( diff --git a/test/lit/pto/colsum_tile_native.pto b/test/lit/pto/colsum_tile_native.pto new file mode 100644 index 0000000000..252428a442 --- /dev/null +++ b/test/lit/pto/colsum_tile_native.pto @@ -0,0 +1,44 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tcolsum_arg( + %src: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tcolsum ins(%src : !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tcolsum_binary_arg( + %src: !pto.tile_buf, + %tmp: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tcolsum ins(%src, %tmp {isBinary = true} : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tcolsum_arg( +// NATIVE: pto.tcolsum ins(%arg0 : !pto.tile_buf) outs(%arg1 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tcolsum_binary_arg( +// NATIVE: pto.tcolsum ins(%arg0, %arg1 {isBinary = true} : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tcolsum_arg( +// EMITC: TCOLSUM( +// EMITC-LABEL: tcolsum_binary_arg( +// EMITC: TCOLSUM( diff --git a/test/lit/pto/compact_left_blayout_parser_a3.pto b/test/lit/pto/compact_left_blayout_parser_a3.pto index b1c7314036..271cdcafd6 100644 --- a/test/lit/pto/compact_left_blayout_parser_a3.pto +++ b/test/lit/pto/compact_left_blayout_parser_a3.pto @@ -8,4 +8,4 @@ module attributes {"pto.target_arch" = "a3"} { } // CHECK-LABEL: func.func @compact_left_blayout_parser_a3() { -// CHECK: #pto.tile_buf_config, slayout=#pto.slayout, s_fractal_size=512, pad=#pto.pad_value, compact=#pto.compact_mode> +// CHECK: pto.alloc_tile : !pto.tile_buf diff --git a/test/lit/pto/compact_left_blayout_parser_a5.pto b/test/lit/pto/compact_left_blayout_parser_a5.pto index 4dad24114e..23a5c1b6e5 100644 --- a/test/lit/pto/compact_left_blayout_parser_a5.pto +++ b/test/lit/pto/compact_left_blayout_parser_a5.pto @@ -8,4 +8,4 @@ module attributes {"pto.target_arch" = "a5"} { } // CHECK-LABEL: func.func @compact_left_blayout_parser_a5() { -// CHECK: #pto.tile_buf_config, slayout=#pto.slayout, s_fractal_size=512, pad=#pto.pad_value, compact=#pto.compact_mode> +// CHECK: pto.alloc_tile : !pto.tile_buf diff --git a/test/lit/pto/compare_tile_native.pto b/test/lit/pto/compare_tile_native.pto new file mode 100644 index 0000000000..2728956cf8 --- /dev/null +++ b/test/lit/pto/compare_tile_native.pto @@ -0,0 +1,45 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tcmp_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tcmp ins(%src0, %src1 {cmpMode = #pto} : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tcmps_arg( + %src: !pto.tile_buf, + %scalar: f32, + %dst: !pto.tile_buf) { + pto.tcmps ins(%src, %scalar {cmpMode = #pto} : !pto.tile_buf, f32) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tcmp_arg( +// NATIVE: pto.tcmp ins(%arg0, %arg1 {cmpMode = #pto} : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tcmps_arg( +// NATIVE: pto.tcmps ins(%arg0, %arg1 {cmpMode = #pto} : !pto.tile_buf, f32) outs(%arg2 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tcmp_arg( +// EMITC: TCMP( +// EMITC-LABEL: tcmps_arg( +// EMITC: TCMPS( diff --git a/test/lit/pto/concat_tile_native.pto b/test/lit/pto/concat_tile_native.pto new file mode 100644 index 0000000000..588116a7bb --- /dev/null +++ b/test/lit/pto/concat_tile_native.pto @@ -0,0 +1,47 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tconcat_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tconcat ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tconcatidx_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %idx0: !pto.tile_buf, + %idx1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tconcatidx ins(%src0, %src1, %idx0, %idx1 : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tconcat_arg( +// NATIVE: pto.tconcat ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tconcatidx_arg( +// NATIVE: pto.tconcatidx ins(%arg0, %arg1, %arg2, %arg3 : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) outs(%arg4 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tconcat_arg( +// EMITC: TCONCAT( +// EMITC-LABEL: tconcatidx_arg( +// EMITC: TCONCAT( diff --git a/test/lit/pto/cvt_tile_native.pto b/test/lit/pto/cvt_tile_native.pto new file mode 100644 index 0000000000..f711431ce6 --- /dev/null +++ b/test/lit/pto/cvt_tile_native.pto @@ -0,0 +1,31 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tcvt_arg( + %src: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tcvt ins(%src {rmode = #pto, satmode = #pto} : !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tcvt_arg( +// NATIVE: pto.tcvt ins(%arg0 {rmode = #pto, satmode = #pto} : !pto.tile_buf) outs(%arg1 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tcvt_arg( +// EMITC: TCVT( diff --git a/test/lit/pto/declare_tile_tile_native.pto b/test/lit/pto/declare_tile_tile_native.pto new file mode 100644 index 0000000000..9a8a40ab50 --- /dev/null +++ b/test/lit/pto/declare_tile_tile_native.pto @@ -0,0 +1,50 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-arch=a3 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-arch=a3 --mlir-print-ir-after=pto-plan-memory %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-arch=a3 --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-arch=a3 --mlir-print-ir-after=pto-materialize-tile-handles %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-arch=a3 %s -o - 2>&1 | FileCheck %s --check-prefix=EMITC + +module { + func.func @declare_tile_native( + %gm_slot_buffer: memref<256xf32, #pto.address_space>, + %consumer_buf: i32) attributes {pto.kernel_kind = #pto.kernel_kind} { + %pipe = pto.initialize_l2g2l_pipe { + dir_mask = 1, + slot_size = 1024, + slot_num = 8, + local_slot_num = 8, + flag_base = 0 + }(%gm_slot_buffer : memref<256xf32, #pto.address_space>, + %consumer_buf : i32) -> !pto.pipe + %tile = pto.declare_tile + -> !pto.tile_buf + pto.tpop(%tile, %pipe + : !pto.tile_buf, !pto.pipe) { + split = 1 + } + return + } +} + +// NATIVE-LABEL: func.func @declare_tile_native +// NATIVE: pto.declare_tile -> !pto.tile_buf +// NATIVE: pto.tpop({{.*}} : !pto.tile_buf +// NATIVE-NOT: pto.declare_tile_memref +// NATIVE-NOT: pto.bind_tile +// NATIVE-NOT: memref.alloc + +// EMITC-LABEL: AICORE void declare_tile_native +// EMITC: Tile&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tdiv_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tdiv ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + {precisionType = #pto} + return + } + + func.func private @tdivs_tile_scalar_arg( + %src: !pto.tile_buf, + %scalar: f32, + %dst: !pto.tile_buf) { + pto.tdivs ins(%src, %scalar : !pto.tile_buf, f32) + outs(%dst : !pto.tile_buf) + {precisionType = #pto} + return + } + + func.func private @tdivs_scalar_tile_arg( + %scalar: f32, + %src: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tdivs ins(%scalar, %src : f32, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + {precisionType = #pto} + return + } +} + +// NATIVE-LABEL: func.func private @tdiv_arg( +// NATIVE: pto.tdiv ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) {precisionType = #pto} +// NATIVE-LABEL: func.func private @tdivs_tile_scalar_arg( +// NATIVE: pto.tdivs ins(%arg0, %arg1 : !pto.tile_buf, f32) outs(%arg2 : !pto.tile_buf) {precisionType = #pto} +// NATIVE-LABEL: func.func private @tdivs_scalar_tile_arg( +// NATIVE: pto.tdivs ins(%arg0, %arg1 : f32, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) {precisionType = #pto} +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tdiv_arg( +// EMITC: TDIV( +// EMITC-LABEL: tdivs_tile_scalar_arg( +// EMITC: TDIVS( +// EMITC-LABEL: tdivs_scalar_tile_arg( +// EMITC: TDIVS( diff --git a/test/lit/pto/emitc_tile_data_sink_after_tassign.pto b/test/lit/pto/emitc_tile_data_sink_after_tassign.pto index 70086a0c89..e8014da937 100644 --- a/test/lit/pto/emitc_tile_data_sink_after_tassign.pto +++ b/test/lit/pto/emitc_tile_data_sink_after_tassign.pto @@ -29,15 +29,6 @@ module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind [[ADDR_BASE:v[0-9]+]]; // CHECK: // pto: %0 // CHECK-NEXT: Tile [[ADDR_TILE:v[0-9]+]] = [[ADDR_BASE]]; -// CHECK-NEXT: TASSIGN([[ADDR_TILE]], v1); // CHECK: // pto: %1 -// CHECK-NEXT: Tile [[SINK_BASE:v[0-9]+]]; -// CHECK: // pto: %1 -// CHECK-NEXT: Tile [[SINK_TILE:v[0-9]+]] = [[SINK_BASE]]; -// CHECK: // pto: %1 -// CHECK-NEXT: __ubuf__ float* [[ADDR_PTR:v[0-9]+]] = [[ADDR_TILE]].data(); -// CHECK: // pto: %1 -// CHECK-NEXT: uint64_t [[ADDR_BITS:v[0-9]+]] = reinterpret_cast([[ADDR_PTR]]); -// CHECK-NEXT: TASSIGN([[SINK_TILE]], [[ADDR_BITS]]); -// CHECK: __ubuf__ float* [[SINK_PTR:v[0-9]+]] = [[SINK_TILE]].data(); +// CHECK-NEXT: __ubuf__ float* [[SINK_PTR:v[0-9]+]] = [[ADDR_TILE]].data(); // CHECK-NEXT: sink_ptr([[SINK_PTR]]); diff --git a/test/lit/pto/expands_tile_native.pto b/test/lit/pto/expands_tile_native.pto new file mode 100644 index 0000000000..680ec473c5 --- /dev/null +++ b/test/lit/pto/expands_tile_native.pto @@ -0,0 +1,31 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @texpands_arg( + %scalar: f32, + %dst: !pto.tile_buf) { + pto.texpands ins(%scalar : f32) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @texpands_arg( +// NATIVE: pto.texpands ins(%arg0 : f32) outs(%arg1 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: texpands_arg( +// EMITC: TEXPANDS( diff --git a/test/lit/pto/extract_insert_fp_tile_native.pto b/test/lit/pto/extract_insert_fp_tile_native.pto new file mode 100644 index 0000000000..e3c740b1ed --- /dev/null +++ b/test/lit/pto/extract_insert_fp_tile_native.pto @@ -0,0 +1,47 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @textract_fp_arg( + %src: !pto.tile_buf, + %fp: !pto.tile_buf, + %row: index, %col: index, + %dst: !pto.tile_buf) { + pto.textract_fp ins(%src, %fp, %row, %col : !pto.tile_buf, !pto.tile_buf, index, index) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tinsert_fp_arg( + %src: !pto.tile_buf, + %fp: !pto.tile_buf, + %row: index, %col: index, + %dst: !pto.tile_buf) { + pto.tinsert_fp ins(%src, %fp, %row, %col : !pto.tile_buf, !pto.tile_buf, index, index) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @textract_fp_arg( +// NATIVE: pto.textract_fp ins(%arg0, %arg1, %arg2, %arg3 +// NATIVE-LABEL: func.func private @tinsert_fp_arg( +// NATIVE: pto.tinsert_fp ins(%arg0, %arg1, %arg2, %arg3 +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: textract_fp_arg( +// EMITC: TEXTRACT_FP( +// EMITC-LABEL: tinsert_fp_arg( +// EMITC: TINSERT_FP( diff --git a/test/lit/pto/extract_insert_tile_native.pto b/test/lit/pto/extract_insert_tile_native.pto new file mode 100644 index 0000000000..86a1d8ecfc --- /dev/null +++ b/test/lit/pto/extract_insert_tile_native.pto @@ -0,0 +1,47 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @textract_arg( + %src: !pto.tile_buf, + %row: index, + %col: index, + %dst: !pto.tile_buf) { + pto.textract ins(%src, %row, %col : !pto.tile_buf, index, index) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tinsert_arg( + %src: !pto.tile_buf, + %row: index, + %col: index, + %dst: !pto.tile_buf) { + pto.tinsert ins(%src, %row, %col : !pto.tile_buf, index, index) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @textract_arg( +// NATIVE: pto.textract ins(%arg0, %arg1, %arg2 : !pto.tile_buf, index, index) outs(%arg3 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tinsert_arg( +// NATIVE: pto.tinsert ins(%arg0, %arg1, %arg2 : !pto.tile_buf, index, index) outs(%arg3 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: textract_arg( +// EMITC: TEXTRACT( +// EMITC-LABEL: tinsert_arg( +// EMITC: TINSERT( diff --git a/test/lit/pto/fillpad_tile_native.pto b/test/lit/pto/fillpad_tile_native.pto new file mode 100644 index 0000000000..081127f5ca --- /dev/null +++ b/test/lit/pto/fillpad_tile_native.pto @@ -0,0 +1,41 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tfillpad_arg( + %tile: !pto.tile_buf) { + pto.tfillpad ins(%tile : !pto.tile_buf) + outs(%tile : !pto.tile_buf) + return + } + + func.func private @tfillpad_inplace_arg( + %tile: !pto.tile_buf) { + pto.tfillpad_inplace ins(%tile : !pto.tile_buf) + outs(%tile : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tfillpad_arg( +// NATIVE: pto.tfillpad ins(%arg0 +// NATIVE-LABEL: func.func private @tfillpad_inplace_arg( +// NATIVE: pto.tfillpad_inplace ins(%arg0 +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tfillpad_arg( +// EMITC: TFILLPAD( +// EMITC-LABEL: tfillpad_inplace_arg( +// EMITC: TFILLPAD_INPLACE( diff --git a/test/lit/pto/fixpipe_frontend_emitc_vector_remat_a5.pto b/test/lit/pto/fixpipe_frontend_emitc_vector_remat_a5.pto index 8af360bcc1..9feaa0fae8 100644 --- a/test/lit/pto/fixpipe_frontend_emitc_vector_remat_a5.pto +++ b/test/lit/pto/fixpipe_frontend_emitc_vector_remat_a5.pto @@ -120,9 +120,9 @@ module { // A5-NEXT: TPUSH<{{.*}}, Pipe0FixpipeConfig>( // PLAN-LABEL: func.func @cube_kernel() -// PLAN: pto.set_quant_vector(%{{.*}} : memref<1x32xf16{{.*}}#pto.address_space>) {id = 0} +// PLAN: pto.set_quant_vector(%{{.*}} : !pto.tile_buf) {id = 0} // PLAN-NEXT: pto.tpush( -// PLAN: pto.set_quant_vector(%{{.*}} : memref<1x32xf16{{.*}}#pto.address_space>) {id = 1} +// PLAN: pto.set_quant_vector(%{{.*}} : !pto.tile_buf) {id = 1} // PLAN-NEXT: pto.tpush( -// PLAN: pto.set_quant_vector(%{{.*}} : memref<1x32xf16{{.*}}#pto.address_space>) {id = 0} +// PLAN: pto.set_quant_vector(%{{.*}} : !pto.tile_buf) {id = 0} // PLAN-NEXT: pto.tpush( diff --git a/test/lit/pto/fixpipe_frontend_quant_state_preserve_ir_a5.pto b/test/lit/pto/fixpipe_frontend_quant_state_preserve_ir_a5.pto index 9ce0ff5db4..13e9c893ff 100644 --- a/test/lit/pto/fixpipe_frontend_quant_state_preserve_ir_a5.pto +++ b/test/lit/pto/fixpipe_frontend_quant_state_preserve_ir_a5.pto @@ -130,7 +130,7 @@ module { // LATE-NEXT: pto.tpush( // LATE: pto.set_quant_scalar(%{{.*}} : f32) {id = 0} // LATE-NEXT: pto.tpush( -// LATE: pto.set_quant_vector(%{{.*}} : memref<1x32xf16{{.*}}#pto.address_space>) {id = 1} +// LATE: pto.set_quant_vector(%{{.*}} : !pto.tile_buf) {id = 1} // LATE-NEXT: pto.tpush( -// LATE: pto.set_quant_vector(%{{.*}} : memref<1x32xf16{{.*}}#pto.address_space>) {id = 1} +// LATE: pto.set_quant_vector(%{{.*}} : !pto.tile_buf) {id = 1} // LATE-NEXT: pto.tpush( diff --git a/test/lit/pto/gather_tile_native.pto b/test/lit/pto/gather_tile_native.pto new file mode 100644 index 0000000000..579e00f151 --- /dev/null +++ b/test/lit/pto/gather_tile_native.pto @@ -0,0 +1,58 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tgather_mask_arg( + %src: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tgather ins(%src, {maskPattern = #pto.mask_pattern} : !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tgather_index_arg( + %src: !pto.tile_buf, + %indices: !pto.tile_buf, + %tmp: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tgather ins(%src, %indices, %tmp : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tgatherb_arg( + %src: !pto.tile_buf, + %offsets: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tgatherb ins(%src, %offsets : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tgather_mask_arg( +// NATIVE: pto.tgather +// NATIVE-LABEL: func.func private @tgather_index_arg( +// NATIVE: pto.tgather +// NATIVE-LABEL: func.func private @tgatherb_arg( +// NATIVE: pto.tgatherb ins(%arg0, %arg1 +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tgather_mask_arg( +// EMITC: TGATHER< +// EMITC-LABEL: tgather_index_arg( +// EMITC: TGATHER( +// EMITC-LABEL: tgatherb_arg( +// EMITC: TGATHERB( diff --git a/test/lit/pto/gemv_tile_native.pto b/test/lit/pto/gemv_tile_native.pto new file mode 100644 index 0000000000..4e506374dc --- /dev/null +++ b/test/lit/pto/gemv_tile_native.pto @@ -0,0 +1,28 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tgemv_arg(%lhs: !pto.tile_buf, %rhs: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tgemv ins(%lhs, %rhs : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @tgemv_arg +// NATIVE-SAME: !pto.tile_buf&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-arch=a3 --mlir-print-ir-after=pto-resolve-buffer-select %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=RESOLVED + +module { + func.func @gm_tensor_view_native_through_sync(%ptr: !pto.ptr) { + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c16 = arith.constant 16 : index + %c32 = arith.constant 32 : index + %view = pto.make_tensor_view %ptr, shape = [%c32, %c16], + strides = [%c16, %c1] : !pto.tensor_view + %dim = pto.get_tensor_view_dim %view, %c0 + : !pto.tensor_view -> index + %stride = pto.get_tensor_view_stride %view, %c1 + : !pto.tensor_view -> index + %part = pto.partition_view %view, offsets = [%c0, %stride], + sizes = [%dim, %c16] + : !pto.tensor_view -> !pto.partition_tensor_view<32x16xf32> + return + } +} + +// NATIVE: IR Dump After PTOViewToMemref +// NATIVE: pto.make_tensor_view +// NATIVE: pto.get_tensor_view_dim +// NATIVE: pto.get_tensor_view_stride +// NATIVE: pto.partition_view +// NATIVE-NOT: memref.reinterpret_cast +// NATIVE-NOT: memref.subview + +// RESOLVED: IR Dump After PTOResolveBufferSelect +// RESOLVED: memref.reinterpret_cast +// RESOLVED: memref.dim +// RESOLVED: memref.subview +// RESOLVED: memref.extract_strided_metadata +// RESOLVED-NOT: pto.make_tensor_view +// RESOLVED-NOT: pto.get_tensor_view_dim +// RESOLVED-NOT: pto.get_tensor_view_stride +// RESOLVED-NOT: pto.partition_view diff --git a/test/lit/pto/graph_sync_solver_basic.pto b/test/lit/pto/graph_sync_solver_basic.pto index 95ebf777e9..dd34184526 100644 --- a/test/lit/pto/graph_sync_solver_basic.pto +++ b/test/lit/pto/graph_sync_solver_basic.pto @@ -148,7 +148,7 @@ module { // CHECK-LABEL: func.func @event_id_fallback // CHECK-COUNT-9: {{pto\.set_flag\[, , \]}} - // CHECK: pto.tmuls ins(%4, %cst : + // CHECK: pto.tmuls ins(%{{[0-9]+}}, %cst : func.func @event_id_fallback(%arg0: !pto.ptr, %arg2: index, %arg3: index, %arg4: index, diff --git a/test/lit/pto/issue531_tprelu_vec_overflow_a5.pto b/test/lit/pto/issue531_tprelu_vec_overflow_a5.pto index c8b9fdcbf3..9c76c24bf8 100644 --- a/test/lit/pto/issue531_tprelu_vec_overflow_a5.pto +++ b/test/lit/pto/issue531_tprelu_vec_overflow_a5.pto @@ -16,13 +16,13 @@ module { func.func @TPRELU_OVERFLOW() { - %src0 = pto.alloc_tile : !pto.tile_buf - %src1 = pto.alloc_tile : !pto.tile_buf - %tmp = pto.alloc_tile : !pto.tile_buf - %dst = pto.alloc_tile : !pto.tile_buf + %src0 = pto.alloc_tile : !pto.tile_buf + %src1 = pto.alloc_tile : !pto.tile_buf + %tmp = pto.alloc_tile : !pto.tile_buf + %dst = pto.alloc_tile : !pto.tile_buf - pto.tprelu ins(%src0, %src1, %tmp : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) - outs(%dst : !pto.tile_buf) + pto.tprelu ins(%src0, %src1, %tmp : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) return } } diff --git a/test/lit/pto/issue686_a5_tmov_treshape_square_dynamic_valid_shape.pto b/test/lit/pto/issue686_a5_tmov_treshape_square_dynamic_valid_shape.pto index 73626db68f..8b41fc5a0e 100644 --- a/test/lit/pto/issue686_a5_tmov_treshape_square_dynamic_valid_shape.pto +++ b/test/lit/pto/issue686_a5_tmov_treshape_square_dynamic_valid_shape.pto @@ -29,10 +29,10 @@ module attributes {pto.target_arch = "a5"} { // CHECK-DAG: [[PART:%[A-Za-z0-9_]+]] = arith.constant 8 : index // CHECK-DAG: [[FULL:%[A-Za-z0-9_]+]] = arith.constant 16 : index -// CHECK: [[SRC_BASE:%[0-9]+]] = pto.bind_tile {{.*}}, [[PART]], [[FULL]] {{.*}}blayout=#pto.blayout -// CHECK: [[DST_BASE:%[0-9]+]] = pto.bind_tile {{.*}}, [[PART]], [[FULL]] {{.*}}blayout=#pto.blayout -// CHECK: [[SRC:%[0-9]+]] = pto.bind_tile [[SRC_BASE]], [[PART]], [[FULL]] {{.*}}blayout=#pto.blayout{{.*}}pto.view_semantics = "treshape" -// CHECK: [[DST:%[0-9]+]] = pto.bind_tile [[DST_BASE]], [[PART]], [[FULL]] {{.*}}blayout=#pto.blayout{{.*}}pto.view_semantics = "treshape" +// CHECK: [[SRC_BASE:%[0-9]+]] = pto.alloc_tile addr = {{.*}} valid_row = [[PART]] valid_col = [[FULL]] : !pto.tile_buf +// CHECK: [[DST_BASE:%[0-9]+]] = pto.alloc_tile addr = {{.*}} valid_row = [[PART]] valid_col = [[FULL]] : !pto.tile_buf +// CHECK: [[SRC:%[0-9]+]] = pto.treshape [[SRC_BASE]] : !pto.tile_buf -> !pto.tile_buf +// CHECK: [[DST:%[0-9]+]] = pto.treshape [[DST_BASE]] : !pto.tile_buf -> !pto.tile_buf // CHECK: [[SRC_ROW:%[A-Za-z0-9_]+]], [[SRC_COL:%[A-Za-z0-9_]+]] = pto.get_validshape [[SRC_BASE]] // CHECK-NEXT: pto.set_validshape [[SRC]], [[SRC_COL]], [[SRC_ROW]] // CHECK-NEXT: [[DST_ROW:%[A-Za-z0-9_]+]], [[DST_COL:%[A-Za-z0-9_]+]] = pto.get_validshape [[DST_BASE]] diff --git a/test/lit/pto/issue708_zero_valid_subview.pto b/test/lit/pto/issue708_zero_valid_subview.pto index 3aa8615180..b4c7be1bc1 100644 --- a/test/lit/pto/issue708_zero_valid_subview.pto +++ b/test/lit/pto/issue708_zero_valid_subview.pto @@ -46,11 +46,9 @@ module attributes {"pto.device-spec" = "Ascend910B3"} { } // CHECK-LABEL: func.func @issue708_zero_valid_subview_row -// CHECK: %[[ROW:[0-9A-Za-z_]+]] = pto.bind_tile {{.*}}, %c0, %c128 -// CHECK-SAME: pto.view_semantics = "subview" +// CHECK: %[[ROW:[0-9A-Za-z_]+]] = pto.alloc_tile addr = %{{.*}} {pto.view_semantics = "subview"} : !pto.tile_buf // CHECK: pto.tstore ins(%[[ROW]] // CHECK-LABEL: func.func @issue708_zero_valid_subview_col -// CHECK: %[[COL:[0-9A-Za-z_]+]] = pto.bind_tile {{.*}}, %c5, %c0 -// CHECK-SAME: pto.view_semantics = "subview" +// CHECK: %[[COL:[0-9A-Za-z_]+]] = pto.alloc_tile addr = %{{.*}} {pto.view_semantics = "subview"} : !pto.tile_buf // CHECK: pto.tstore ins(%[[COL]] diff --git a/test/lit/pto/issue708_zero_valid_subview_inferred.pto b/test/lit/pto/issue708_zero_valid_subview_inferred.pto index 2abc8bae8f..faa4df0930 100644 --- a/test/lit/pto/issue708_zero_valid_subview_inferred.pto +++ b/test/lit/pto/issue708_zero_valid_subview_inferred.pto @@ -91,25 +91,21 @@ module attributes {"pto.device-spec" = "Ascend910B3"} { } } -// The bind_tile that replaces each pto.subview must carry valid operands taken -// from the result type, i.e. 0 for an empty axis -- never the subview size. +// The resolved subview handle must carry valid extents taken from the result +// type, i.e. 0 for an empty axis -- never the subview size. // CHECK-LABEL: func.func @issue708_inferred_empty_store -// CHECK: %[[BOTH:[0-9A-Za-z_]+]] = pto.bind_tile {{.*}}, %c0, %c0 -// CHECK-SAME: pto.view_semantics = "subview" +// CHECK: %[[BOTH:[0-9A-Za-z_]+]] = pto.alloc_tile addr = %{{.*}} {pto.view_semantics = "subview"} : !pto.tile_buf // CHECK: pto.tstore ins(%[[BOTH]] // CHECK-LABEL: func.func @issue708_inferred_empty_cvt -// CHECK: %[[CVTV:[0-9A-Za-z_]+]] = pto.bind_tile {{.*}}, %c0, %c0 -// CHECK-SAME: pto.view_semantics = "subview" +// CHECK: %[[CVTV:[0-9A-Za-z_]+]] = pto.alloc_tile addr = %{{.*}} {pto.view_semantics = "subview"} : !pto.tile_buf // CHECK: pto.tcvt ins(%[[CVTV]] // CHECK-LABEL: func.func @issue708_inferred_row_empty -// CHECK: %[[ROW:[0-9A-Za-z_]+]] = pto.bind_tile {{.*}}, %c0, %c128 -// CHECK-SAME: pto.view_semantics = "subview" +// CHECK: %[[ROW:[0-9A-Za-z_]+]] = pto.alloc_tile addr = %{{.*}} {pto.view_semantics = "subview"} : !pto.tile_buf // CHECK: pto.tstore ins(%[[ROW]] // CHECK-LABEL: func.func @issue708_inferred_partial_valid -// CHECK: %[[PARTIAL:[0-9A-Za-z_]+]] = pto.bind_tile {{.*}}, %c3, %c128 -// CHECK-SAME: pto.view_semantics = "subview" +// CHECK: %[[PARTIAL:[0-9A-Za-z_]+]] = pto.alloc_tile addr = %{{.*}} {pto.view_semantics = "subview"} : !pto.tile_buf // CHECK: pto.tstore ins(%[[PARTIAL]] diff --git a/test/lit/pto/load_store_tile_native.pto b/test/lit/pto/load_store_tile_native.pto new file mode 100644 index 0000000000..ad73bd6915 --- /dev/null +++ b/test/lit/pto/load_store_tile_native.pto @@ -0,0 +1,42 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tload_arg( + %src: !pto.partition_tensor_view<16x16xf32>, + %dst: !pto.tile_buf) { + pto.tload ins(%src : !pto.partition_tensor_view<16x16xf32>) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tstore_arg( + %src: !pto.tile_buf, + %dst: !pto.partition_tensor_view<16x16xf32>) { + pto.tstore ins(%src : !pto.tile_buf) + outs(%dst : !pto.partition_tensor_view<16x16xf32>) + return + } +} + +// NATIVE-LABEL: func.func private @tload_arg( +// NATIVE: pto.tload ins(%arg0 : memref<16x16xf32>) outs(%arg1 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tstore_arg( +// NATIVE: pto.tstore ins(%arg0 : !pto.tile_buf) outs(%arg1 : memref<16x16xf32>) +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tload_arg( +// EMITC: TLOAD( +// EMITC-LABEL: tstore_arg( +// EMITC: TSTORE( diff --git a/test/lit/pto/logic_binary_tile_native.pto b/test/lit/pto/logic_binary_tile_native.pto new file mode 100644 index 0000000000..147bfda9ae --- /dev/null +++ b/test/lit/pto/logic_binary_tile_native.pto @@ -0,0 +1,51 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tand_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tand ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + pto.tand ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%src0 : !pto.tile_buf) + return + } + + func.func private @tor_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tor ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + pto.tor ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%src0 : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tand_arg( +// NATIVE: pto.tand ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE: pto.tand ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg0 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tor_arg( +// NATIVE: pto.tor ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE: pto.tor ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg0 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tand_arg( +// EMITC-COUNT-2: TAND( +// EMITC-LABEL: tor_arg( +// EMITC-COUNT-2: TOR( diff --git a/test/lit/pto/logic_scalar_tile_native.pto b/test/lit/pto/logic_scalar_tile_native.pto new file mode 100644 index 0000000000..dc6c457f68 --- /dev/null +++ b/test/lit/pto/logic_scalar_tile_native.pto @@ -0,0 +1,45 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tands_arg( + %src: !pto.tile_buf, + %scalar: i16, + %dst: !pto.tile_buf) { + pto.tands ins(%src, %scalar : !pto.tile_buf, i16) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tors_arg( + %src: !pto.tile_buf, + %scalar: i16, + %dst: !pto.tile_buf) { + pto.tors ins(%src, %scalar : !pto.tile_buf, i16) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tands_arg( +// NATIVE: pto.tands ins(%arg0, %arg1 : !pto.tile_buf, i16) outs(%arg2 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tors_arg( +// NATIVE: pto.tors ins(%arg0, %arg1 : !pto.tile_buf, i16) outs(%arg2 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tands_arg( +// EMITC: TANDS( +// EMITC-LABEL: tors_arg( +// EMITC: TORS( diff --git a/test/lit/pto/materialize_tile_handles_subkernel_helper_abi.pto b/test/lit/pto/materialize_tile_handles_subkernel_helper_abi.pto index 0fe4c0e375..c5c62816a6 100644 --- a/test/lit/pto/materialize_tile_handles_subkernel_helper_abi.pto +++ b/test/lit/pto/materialize_tile_handles_subkernel_helper_abi.pto @@ -6,8 +6,10 @@ // INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. // See LICENSE in the root of the software repository for the full text of the License. -// Guards PTODSL subkernel helper arguments restored to tile_buf ABI. +// Guards PTODSL subkernel helper arguments staying in tile_buf ABI when the +// helper directly extracts their address views. // RUN: ptoas --pto-arch=a5 --pto-level=level3 --mlir-print-ir-after=pto-materialize-tile-handles %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=IR +// RUN: ptoas --pto-arch=a5 --pto-level=level3 %s -o /dev/null module { func.func @ptodsl_subkernel_helper_tile_abi( @@ -37,12 +39,11 @@ module { } // IR: IR Dump After PTOMaterializeTileHandles -// IR-LABEL: func.func private @ptodsl_subkernel_helper_tile_abi( +// IR-LABEL: func.func @ptodsl_subkernel_helper_tile_abi( // IR-SAME: %{{.*}}: !pto.tile_buf // IR-SAME: %{{.*}}: !pto.tile_buf // IR-SAME: %{{.*}}: f32 // IR-SAME: attributes -// IR: builtin.unrealized_conversion_cast %{{.*}} : !pto.tile_buf to memref<16x1xf32, strided<[1, 16], offset: ?>, #pto.address_space> -// IR: builtin.unrealized_conversion_cast %{{.*}} : !pto.tile_buf to memref<16x16xf16, strided<[16, 1], offset: ?>, #pto.address_space> -// IR: pto.tile_buf_addr %{{.*}} : !pto.tile_buf -// IR: pto.tile_buf_addr %{{.*}} : !pto.tile_buf +// IR: pto.tile_buf_addr %{{.*}} : !pto.tile_buf -> memref<16x16xf16, #pto.address_space> +// IR: pto.tile_buf_addr %{{.*}} : !pto.tile_buf -> memref<16x1xf32, #pto.address_space> +// IR-NOT: builtin.unrealized_conversion_cast diff --git a/test/lit/pto/matmul_tile_native.pto b/test/lit/pto/matmul_tile_native.pto new file mode 100644 index 0000000000..574ff3af1c --- /dev/null +++ b/test/lit/pto/matmul_tile_native.pto @@ -0,0 +1,28 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tmatmul_arg(%lhs: !pto.tile_buf, %rhs: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tmatmul ins(%lhs, %rhs : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @tmatmul_arg +// NATIVE-SAME: !pto.tile_buf([[MGATHER_DST]], {{[_A-Za-z][_A-Za-z0-9]*}}, [[MGATHER_IDX]]); +// CHECK: MGATHER({{[_A-Za-z][_A-Za-z0-9]*}}, {{[_A-Za-z][_A-Za-z0-9]*}}, {{[_A-Za-z][_A-Za-z0-9]*}}); // CHECK-LABEL: AICORE void mscatter_emitc // CHECK: TLOAD({{[_A-Za-z][_A-Za-z0-9]*}}, {{[_A-Za-z][_A-Za-z0-9]*}}); // CHECK: TLOAD({{[_A-Za-z][_A-Za-z0-9]*}}, {{[_A-Za-z][_A-Za-z0-9]*}}); -// CHECK-DAG: __ubuf__ float* [[MSCATTER_SRC:[_A-Za-z][_A-Za-z0-9]*]] = {{.*}}; -// CHECK-DAG: __ubuf__ int32_t* [[MSCATTER_IDX:[_A-Za-z][_A-Za-z0-9]*]] = {{.*}}; -// CHECK: MSCATTER({{[_A-Za-z][_A-Za-z0-9]*}}, [[MSCATTER_SRC]], [[MSCATTER_IDX]]); +// CHECK: MSCATTER({{[_A-Za-z][_A-Za-z0-9]*}}, {{[_A-Za-z][_A-Za-z0-9]*}}, {{[_A-Za-z][_A-Za-z0-9]*}}); diff --git a/test/lit/pto/mgather_mscatter_base_a3_emitc.pto b/test/lit/pto/mgather_mscatter_base_a3_emitc.pto index ac413fa848..e407a8e9db 100644 --- a/test/lit/pto/mgather_mscatter_base_a3_emitc.pto +++ b/test/lit/pto/mgather_mscatter_base_a3_emitc.pto @@ -63,11 +63,7 @@ module attributes {pto.target_arch = "a3"} { } // CHECK-LABEL: AICORE void mgather_emitc_base( -// CHECK: __ubuf__ int32_t* [[MGATHER_IDX:[_A-Za-z][_A-Za-z0-9]*]] = {{[_A-Za-z][_A-Za-z0-9]*}}.data(); -// CHECK: __ubuf__ float* [[MGATHER_DST:[_A-Za-z][_A-Za-z0-9]*]] = {{[_A-Za-z][_A-Za-z0-9]*}}.data(); -// CHECK: MGATHER([[MGATHER_DST]], {{[_A-Za-z][_A-Za-z0-9]*}}, [[MGATHER_IDX]]); +// CHECK: MGATHER({{[_A-Za-z][_A-Za-z0-9]*}}, {{[_A-Za-z][_A-Za-z0-9]*}}, {{[_A-Za-z][_A-Za-z0-9]*}}); // CHECK-LABEL: AICORE void mscatter_emitc_base( -// CHECK: __ubuf__ float* [[MSCATTER_SRC:[_A-Za-z][_A-Za-z0-9]*]] = {{[_A-Za-z][_A-Za-z0-9]*}}.data(); -// CHECK: __ubuf__ int32_t* [[MSCATTER_IDX:[_A-Za-z][_A-Za-z0-9]*]] = {{[_A-Za-z][_A-Za-z0-9]*}}.data(); -// CHECK: MSCATTER({{[_A-Za-z][_A-Za-z0-9]*}}, [[MSCATTER_SRC]], [[MSCATTER_IDX]]); +// CHECK: MSCATTER({{[_A-Za-z][_A-Za-z0-9]*}}, {{[_A-Za-z][_A-Za-z0-9]*}}, {{[_A-Za-z][_A-Za-z0-9]*}}); diff --git a/test/lit/pto/mgather_mscatter_tile_native.pto b/test/lit/pto/mgather_mscatter_tile_native.pto new file mode 100644 index 0000000000..72be822671 --- /dev/null +++ b/test/lit/pto/mgather_mscatter_tile_native.pto @@ -0,0 +1,52 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @mgather_arg( + %mem: memref<1x1x1x8x16xf16, #pto.address_space>, + %idx: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.mgather ins(%mem, %idx : memref<1x1x1x8x16xf16, #pto.address_space>, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + {coalesce = #pto, gatherOob = #pto} + return + } + + func.func private @mscatter_arg( + %src: !pto.tile_buf, + %idx: !pto.tile_buf, + %mem: memref<1x1x1x8x16xf16, #pto.address_space>) { + pto.mscatter ins(%src, %idx : !pto.tile_buf, !pto.tile_buf) + outs(%mem : memref<1x1x1x8x16xf16, #pto.address_space>) + {coalesce = #pto, scatterOob = #pto} + return + } +} + +// NATIVE-LABEL: func.func private @mgather_arg +// NATIVE-SAME: memref<1x1x1x8x16xf16, #pto.address_space> +// NATIVE-SAME: !pto.tile_buf +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.mgather +// NATIVE-LABEL: func.func private @mscatter_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE-SAME: !pto.tile_buf +// NATIVE-SAME: memref<1x1x1x8x16xf16, #pto.address_space> +// NATIVE: pto.mscatter +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: mgather_arg( +// EMITC: MGATHER( +// EMITC-LABEL: mscatter_arg( +// EMITC: MSCATTER( diff --git a/test/lit/pto/minmax_scalar_tile_native.pto b/test/lit/pto/minmax_scalar_tile_native.pto new file mode 100644 index 0000000000..7f053bb34c --- /dev/null +++ b/test/lit/pto/minmax_scalar_tile_native.pto @@ -0,0 +1,54 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tmaxs_arg( + %src: !pto.tile_buf, + %scalar: f32, + %dst: !pto.tile_buf) { + pto.tmaxs ins(%src, %scalar : !pto.tile_buf, f32) + outs(%dst : !pto.tile_buf) + pto.tmaxs ins(%src, %scalar : !pto.tile_buf, f32) + outs(%src : !pto.tile_buf) + return + } + + func.func private @tmins_arg( + %src: !pto.tile_buf, + %scalar: f32, + %dst: !pto.tile_buf) { + pto.tmins ins(%src, %scalar : !pto.tile_buf, f32) + outs(%dst : !pto.tile_buf) + pto.tmins ins(%src, %scalar : !pto.tile_buf, f32) + outs(%src : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tmaxs_arg( +// NATIVE-SAME: %arg0: !pto.tile_buf +// NATIVE-SAME: %arg1: f32 +// NATIVE-SAME: %arg2: !pto.tile_buf +// NATIVE: pto.tmaxs ins(%arg0, %arg1 : !pto.tile_buf, f32) outs(%arg2 : !pto.tile_buf) +// NATIVE: pto.tmaxs ins(%arg0, %arg1 : !pto.tile_buf, f32) outs(%arg0 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tmins_arg( +// NATIVE: pto.tmins ins(%arg0, %arg1 : !pto.tile_buf, f32) outs(%arg2 : !pto.tile_buf) +// NATIVE: pto.tmins ins(%arg0, %arg1 : !pto.tile_buf, f32) outs(%arg0 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tmaxs_arg( +// EMITC-COUNT-2: TMAXS( +// EMITC-LABEL: tmins_arg( +// EMITC-COUNT-2: TMINS( diff --git a/test/lit/pto/minmax_tile_native.pto b/test/lit/pto/minmax_tile_native.pto new file mode 100644 index 0000000000..8b9db4389e --- /dev/null +++ b/test/lit/pto/minmax_tile_native.pto @@ -0,0 +1,54 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tmax_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tmax ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + pto.tmax ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%src0 : !pto.tile_buf) + return + } + + func.func private @tmin_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tmin ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + pto.tmin ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%src0 : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tmax_arg( +// NATIVE-SAME: %arg0: !pto.tile_buf +// NATIVE-SAME: %arg1: !pto.tile_buf +// NATIVE-SAME: %arg2: !pto.tile_buf +// NATIVE: pto.tmax ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE: pto.tmax ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg0 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tmin_arg( +// NATIVE: pto.tmin ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE: pto.tmin ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg0 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tmax_arg( +// EMITC-COUNT-2: TMAX( +// EMITC-LABEL: tmin_arg( +// EMITC-COUNT-2: TMIN( diff --git a/test/lit/pto/mov_tile_native.pto b/test/lit/pto/mov_tile_native.pto new file mode 100644 index 0000000000..b8b77e809e --- /dev/null +++ b/test/lit/pto/mov_tile_native.pto @@ -0,0 +1,26 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tmov_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tmov ins(%src : !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @tmov_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.tmov +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tmov_arg( +// EMITC: TMOV( diff --git a/test/lit/pto/movement_metadata_tile_native.pto b/test/lit/pto/movement_metadata_tile_native.pto new file mode 100644 index 0000000000..4c0e74eee2 --- /dev/null +++ b/test/lit/pto/movement_metadata_tile_native.pto @@ -0,0 +1,33 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tfillpad_expand_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tfillpad_expand ins(%src : !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tget_scale_addr_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tget_scale_addr ins(%src : !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @tfillpad_expand_arg +// NATIVE: pto.tfillpad_expand +// NATIVE-LABEL: @tget_scale_addr_arg +// NATIVE: pto.tget_scale_addr +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tfillpad_expand_arg( +// EMITC: TFILLPAD_EXPAND( +// EMITC-LABEL: tget_scale_addr_arg( +// EMITC: GetScaleAddr diff --git a/test/lit/pto/mrgsort_tile_native.pto b/test/lit/pto/mrgsort_tile_native.pto new file mode 100644 index 0000000000..5fa7f960c4 --- /dev/null +++ b/test/lit/pto/mrgsort_tile_native.pto @@ -0,0 +1,47 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tmrgsort_format1_arg( + %src: !pto.tile_buf, + %dst: !pto.tile_buf, + %block_len: i32) { + pto.tmrgsort ins(%src, %block_len : !pto.tile_buf, i32) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tmrgsort_format2_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %tmp: !pto.tile_buf, + %dst: !pto.tile_buf, + %executed: vector<4xi16>) { + pto.tmrgsort ins(%src0, %src1, %tmp {exhausted = true} : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) + outs(%dst, %executed : !pto.tile_buf, vector<4xi16>) + return + } +} + +// NATIVE-LABEL: func.func private @tmrgsort_format1_arg +// NATIVE: pto.tmrgsort ins(%{{.*}}, %{{.*}} : !pto.tile_buf, i32) outs(%{{.*}} : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tmrgsort_format2_arg +// NATIVE: pto.tmrgsort ins(%{{.*}}, %{{.*}}, %{{.*}} {exhausted = true} : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) outs(%{{.*}}, %{{.*}} : !pto.tile_buf, vector<4xi16>) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tmrgsort_format1_arg( +// EMITC: TMRGSORT( +// EMITC-LABEL: tmrgsort_format2_arg( +// EMITC: TMRGSORT<{{.*}}, true>( diff --git a/test/lit/pto/multi_tile_buf_n3_planmem_e2e.pto b/test/lit/pto/multi_tile_buf_n3_planmem_e2e.pto index 76b9171330..7773a855d9 100644 --- a/test/lit/pto/multi_tile_buf_n3_planmem_e2e.pto +++ b/test/lit/pto/multi_tile_buf_n3_planmem_e2e.pto @@ -7,6 +7,7 @@ // See LICENSE in the root of the software repository for the full text of the License. // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s // Sanity for N == 3 to confirm `ExpandMultiBufferStorageEntry` produces 3 // physical slots and `UpdateBuffer2Offsets` emits them in slot order. @@ -39,4 +40,6 @@ module { // CHECK: IR Dump After PlanMemory // CHECK-LABEL: func.func @n3_three_slots -// CHECK: pto.pointer_cast({{[^,]*}}, {{[^,]*}}, {{[^,]*}}) +// CHECK: pto.alloc_multi_tile {pto.multi_buffer_addrs = array} +// CHECK: pto.multi_tile_get +// CHECK-NOT: pto.pointer_cast diff --git a/test/lit/pto/multi_tile_const_preload_dyn_loop_select.pto b/test/lit/pto/multi_tile_const_preload_dyn_loop_select.pto index c2a1b19d2d..93b2490ee7 100644 --- a/test/lit/pto/multi_tile_const_preload_dyn_loop_select.pto +++ b/test/lit/pto/multi_tile_const_preload_dyn_loop_select.pto @@ -45,21 +45,24 @@ module { // CHECK: IR Dump After PTOViewToMemref // CHECK-LABEL: func.func @const_preload_dyn_loop_select -// CHECK: memref.alloc() {pto.multi_buffer = 2 : i32} -// CHECK: pto.slot_marker %{{.*}}[%c0 +// CHECK: %[[MULTI:.*]] = pto.alloc_multi_tile +// CHECK: pto.multi_tile_get %[[MULTI]][%c0 // CHECK: scf.for // CHECK: %[[IDX:.*]] = arith.remui %arg{{.*}}, %c2 -// CHECK: pto.slot_marker %{{.*}}[%[[IDX]] +// CHECK: pto.multi_tile_get %[[MULTI]][%[[IDX]] +// CHECK-NOT: memref.alloc +// CHECK-NOT: pto.slot_marker -// After ResolveBufferSelect, the const preload slot is a direct single-address -// pointer_cast, while the loop slot is chosen with arith.select. +// After ResolveBufferSelect, the const preload slot becomes a directly +// addressed alloc_tile, while the loop slot is chosen with arith.select. // SELECT: IR Dump After PTOResolveBufferSelect // SELECT-LABEL: func.func @const_preload_dyn_loop_select -// SELECT: %[[ANCHOR:.*]] = pto.pointer_cast(%[[ADDR0:.*]], %[[ADDR1:.*]]) -// SELECT-NOT: pto.slot_marker -// SELECT: pto.pointer_cast(%[[ADDR0]]) +// SELECT-NOT: pto.alloc_multi_tile +// SELECT-NOT: pto.multi_tile_get +// SELECT: pto.alloc_tile addr = %{{.*}} // SELECT: scf.for // SELECT: arith.select +// SELECT: pto.alloc_tile addr = %{{.*}} // MaterializeTileHandles must carry the dynamic selected buffer address into // alloc_tile, otherwise EmitC creates a tile without TASSIGN in the loop. @@ -69,8 +72,10 @@ module { // MAT-DAG: %[[ADDR1:[A-Za-z0-9_]+]] = arith.constant 512 : i64 // MAT: pto.alloc_tile addr = %[[ADDR0]] // MAT: scf.for +// MAT-DAG: %[[LOOP_ADDR0:[A-Za-z0-9_]+]] = arith.constant 0 : i64 +// MAT-DAG: %[[LOOP_ADDR1:[A-Za-z0-9_]+]] = arith.constant 512 : i64 // MAT: %[[COND:[A-Za-z0-9_]+]] = arith.cmpi eq -// MAT: %[[DYN_ADDR:[A-Za-z0-9_]+]] = arith.select %[[COND]], %[[ADDR1]], %[[ADDR0]] : i64 +// MAT: %[[DYN_ADDR:[A-Za-z0-9_]+]] = arith.select %[[COND]], %[[LOOP_ADDR1]], %[[LOOP_ADDR0]] : i64 // MAT: pto.alloc_tile addr = %[[DYN_ADDR]] // EMITC: AICORE void const_preload_dyn_loop_select diff --git a/test/lit/pto/multi_tile_get_const_slot_lowering.pto b/test/lit/pto/multi_tile_get_const_slot_lowering.pto index e8f88edc62..52b5ca5f01 100644 --- a/test/lit/pto/multi_tile_get_const_slot_lowering.pto +++ b/test/lit/pto/multi_tile_get_const_slot_lowering.pto @@ -9,15 +9,13 @@ // RUN: ptoas --mlir-print-ir-after=pto-view-to-memref %s 2>&1 1>/dev/null | FileCheck %s // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=PLAN // RUN: ptoas --mlir-print-ir-after=pto-resolve-buffer-select %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=SELECT +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=PLAN +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-resolve-buffer-select %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=SELECT -// Verifies the full multi-buffer pipeline for constant slots: -// 1. PTOViewToMemref lowers `alloc_multi_tile` to `memref.alloc` with the -// `pto.multi_buffer = N : i32` attribute, and `multi_tile_get [%k]` to -// `pto.slot_marker` carrying the constant slot SSA. -// 2. PTOPlanMemory reserves N physical addresses and emits a multi- -// address `pto.pointer_cast(addr0, addr1)`. -// 3. PTOResolveBufferSelect lowers each `slot_marker` to a single- -// address `pto.pointer_cast(addrK)` -- one per slot use. +// Verifies the tile-native multi-buffer pipeline for constant slots: +// 1. PTOViewToMemref preserves alloc_multi_tile and multi_tile_get. +// 2. PTOPlanMemory records the physical slot addresses on alloc_multi_tile. +// 3. PTOResolveBufferSelect replaces each selection with alloc_tile(addrK). module { func.func @const_slot_two_buffers( @@ -46,21 +44,23 @@ module { // CHECK: IR Dump After PTOViewToMemref // CHECK-LABEL: func.func @const_slot_two_buffers -// CHECK: memref.alloc() {pto.multi_buffer = 2 : i32} -// CHECK: pto.slot_marker %{{.*}}[%c0 -// CHECK: pto.slot_marker %{{.*}}[%c1 +// CHECK: %[[MULTI:.*]] = pto.alloc_multi_tile +// CHECK: pto.multi_tile_get %[[MULTI]][%c0 +// CHECK: pto.multi_tile_get %[[MULTI]][%c1 +// CHECK-NOT: memref.alloc +// CHECK-NOT: pto.slot_marker -// Stage 2 -- PlanMemory: two physical slots reserved -> 2-addr pointer_cast. +// Stage 2 -- PTOPlanMemory records two physical slot addresses. // PLAN: IR Dump After PlanMemory // PLAN-LABEL: func.func @const_slot_two_buffers -// PLAN: pto.pointer_cast({{.*}}, {{.*}}) -// PLAN: pto.slot_marker +// PLAN: pto.alloc_multi_tile {pto.multi_buffer_addrs = array} +// PLAN: pto.multi_tile_get +// PLAN-NOT: pto.pointer_cast -// Stage 3 -- ResolveBufferSelect: each slot_marker becomes a single-addr -// pointer_cast; original 2-addr pointer_cast remains as alloc anchor. +// Stage 3 -- ResolveBufferSelect materializes one addressed tile per use. // SELECT: IR Dump After PTOResolveBufferSelect // SELECT-LABEL: func.func @const_slot_two_buffers -// SELECT: %[[ANCHOR:.*]] = pto.pointer_cast(%{{.*}}, %{{.*}}) -// SELECT-NOT: pto.slot_marker -// SELECT: pto.pointer_cast(%[[ADDR0:.*]]) -// SELECT: pto.pointer_cast(%[[ADDR1:.*]]) +// SELECT-NOT: pto.alloc_multi_tile +// SELECT-NOT: pto.multi_tile_get +// SELECT: pto.alloc_tile addr = %{{.*}} +// SELECT: pto.alloc_tile addr = %{{.*}} diff --git a/test/lit/pto/multi_tile_get_dyn_slot_lowering.pto b/test/lit/pto/multi_tile_get_dyn_slot_lowering.pto index 31b4583b6f..14c055cd44 100644 --- a/test/lit/pto/multi_tile_get_dyn_slot_lowering.pto +++ b/test/lit/pto/multi_tile_get_dyn_slot_lowering.pto @@ -9,11 +9,9 @@ // RUN: ptoas --mlir-print-ir-after=pto-view-to-memref %s 2>&1 1>/dev/null | FileCheck %s // RUN: ptoas --mlir-print-ir-after=pto-resolve-buffer-select %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=SELECT -// Verifies that `multi_tile_get` with a dynamic slot SSA preserves the -// user-supplied slot expression all the way through `pto.slot_marker` and -// the final `arith.select` chain. The lowering must NOT rewrite the slot -// expression into `iv mod N` -- the frontend SSA is the source of truth -// for which slot this use refers to. +// Verifies that tile-native `multi_tile_get` preserves the user-supplied slot +// expression through the final `arith.select` chain. The lowering must not +// synthesize an implicit `iv mod N` expression. module { func.func @dyn_slot_prefetch( @@ -48,17 +46,22 @@ module { // CHECK: IR Dump After PTOViewToMemref // CHECK-LABEL: func.func @dyn_slot_prefetch -// CHECK: memref.alloc() {pto.multi_buffer = 2 : i32} +// CHECK: %[[MULTI:.*]] = pto.alloc_multi_tile // CHECK: %[[NEXT:.*]] = arith.addi %{{.*}}, %c1 // CHECK-DAG: %[[CUR_IDX:.*]] = arith.remui %arg{{.*}}, %c2 // CHECK-DAG: %[[NEXT_IDX:.*]] = arith.remui %[[NEXT]], %c2 -// CHECK: pto.slot_marker %{{.*}}[%[[NEXT_IDX]] -// CHECK: pto.slot_marker %{{.*}}[%[[CUR_IDX]] +// CHECK: pto.multi_tile_get %[[MULTI]][%[[NEXT_IDX]] +// CHECK: pto.multi_tile_get %[[MULTI]][%[[CUR_IDX]] +// CHECK-NOT: memref.alloc +// CHECK-NOT: pto.slot_marker -// After ResolveBufferSelect: slot_marker -> per-slot pointer_cast + -// arith.select chain driven by the user SSA (not iv mod N). +// ResolveBufferSelect creates an addressed alloc_tile from an arith.select +// chain driven by the user SSA (not an implicit iv mod N). // SELECT: IR Dump After PTOResolveBufferSelect // SELECT-LABEL: func.func @dyn_slot_prefetch -// SELECT-NOT: pto.slot_marker -// SELECT: pto.pointer_cast(%{{.*}}, %{{.*}}) +// SELECT-NOT: pto.alloc_multi_tile +// SELECT-NOT: pto.multi_tile_get // SELECT: arith.select +// SELECT: pto.alloc_tile addr = %{{.*}} +// SELECT: arith.select +// SELECT: pto.alloc_tile addr = %{{.*}} diff --git a/test/lit/pto/multi_tile_level3_explicit_addr.pto b/test/lit/pto/multi_tile_level3_explicit_addr.pto index 9efaade82e..6289855628 100644 --- a/test/lit/pto/multi_tile_level3_explicit_addr.pto +++ b/test/lit/pto/multi_tile_level3_explicit_addr.pto @@ -8,15 +8,12 @@ // RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s 2>&1 1>/dev/null | FileCheck %s // RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-resolve-buffer-select %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=SELECT -// RUN: not ptoas --pto-level=level2 --pto-arch=a5 %s 2>&1 | FileCheck %s --check-prefix=LV2ERR +// RUN: not ptoas --pto-level=level1 --pto-arch=a5 %s 2>&1 | FileCheck %s --check-prefix=LVERR +// RUN: not ptoas --pto-level=level2 --pto-arch=a5 %s 2>&1 | FileCheck %s --check-prefix=LVERR // Under --pto-level=level3 the caller owns local memory, PTOPlanMemory does not -// run, and `pto.alloc_multi_tile` carries an explicit base `addr`. The view -// lowering must fan the base address out into the multi-address -// `pto.pointer_cast` anchor PlanMemory would otherwise produce -- slot k at -// `addr + k * slotBytes` (slotBytes = 16*16*2 = 512 here) -- so sync and -// PTOResolveBufferSelect keep working. Without this the slots would all alias -// the same memref (silent single-buffer downgrade). +// run, and `pto.alloc_multi_tile` carries an explicit base `addr`. Slot k is +// resolved as `addr + k * slotBytes` (slotBytes = 16*16*2 = 512 here). module { func.func @l3_two_buffers( @@ -44,23 +41,24 @@ module { } } -// PTOViewToMemref: explicit base address fans out into a 2-address -// pointer_cast; slot 1 = base + 512. No memref.alloc / multi_buffer attr. +// PTOViewToMemref preserves the tile-native allocation and selections. // CHECK: IR Dump After PTOViewToMemref // CHECK-LABEL: func.func @l3_two_buffers -// CHECK-NOT: pto.multi_buffer -// CHECK-DAG: %[[OFF1:.*]] = arith.constant 512 : i64 -// CHECK-DAG: %[[A1:.*]] = arith.addi %{{.*}}, %[[OFF1]] : i64 -// CHECK: pto.pointer_cast(%{{.*}}, %[[A1]]) -// CHECK: pto.slot_marker %{{.*}}[%c0 -// CHECK: pto.slot_marker %{{.*}}[%c1 +// CHECK: %[[BASE:.*]] = arith.constant 0 : i64 +// CHECK: %[[MULTI:.*]] = pto.alloc_multi_tile addr = %[[BASE]] +// CHECK: pto.multi_tile_get %[[MULTI]][%c0 +// CHECK: pto.multi_tile_get %[[MULTI]][%c1 +// CHECK-NOT: pto.multi_buffer_addrs +// CHECK-NOT: memref.alloc -// ResolveBufferSelect: each slot_marker becomes a single-address pointer_cast; -// the 2-address anchor is preserved. +// ResolveBufferSelect materializes one addressed alloc_tile per selection. // SELECT: IR Dump After PTOResolveBufferSelect // SELECT-LABEL: func.func @l3_two_buffers -// SELECT: pto.pointer_cast(%{{.*}}, %{{.*}}) -// SELECT-NOT: pto.slot_marker +// SELECT-NOT: pto.alloc_multi_tile +// SELECT-NOT: pto.multi_tile_get +// SELECT: pto.alloc_tile addr = %{{.*}} +// SELECT: arith.addi +// SELECT: pto.alloc_tile addr = %{{.*}} // Outside level3 an explicit addr on alloc_multi_tile is rejected. -// LV2ERR: unexpected 'addr' operand on pto.alloc_multi_tile: only supported when --pto-level=level3 +// LVERR: unexpected 'addr' operand on pto.alloc_multi_tile: only supported when --pto-level=level3 diff --git a/test/lit/pto/multi_tile_level3_missing_addr.pto b/test/lit/pto/multi_tile_level3_missing_addr.pto index 9784b9db9d..f581df3ebd 100644 --- a/test/lit/pto/multi_tile_level3_missing_addr.pto +++ b/test/lit/pto/multi_tile_level3_missing_addr.pto @@ -8,9 +8,9 @@ // RUN: not ptoas --pto-level=level3 --pto-arch=a5 %s 2>&1 | FileCheck %s -// In level3 PTOPlanMemory is skipped, so multi-buffer slots have no physical -// addresses unless the caller provides a base `addr`. Reject the op instead of -// silently collapsing every slot onto one memref. +// In level3 automatic address planning is unavailable, so multi-buffer slots +// have no physical addresses unless the caller provides a base `addr`. Reject +// the op instead of silently collapsing every slot onto one memref. module { func.func @l3_missing_addr( diff --git a/test/lit/pto/multi_tile_n4_planmem_e2e.pto b/test/lit/pto/multi_tile_n4_planmem_e2e.pto index 361e06da7e..d44d97b3dd 100644 --- a/test/lit/pto/multi_tile_n4_planmem_e2e.pto +++ b/test/lit/pto/multi_tile_n4_planmem_e2e.pto @@ -8,11 +8,11 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=PLAN // RUN: ptoas --mlir-print-ir-after=pto-resolve-buffer-select %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=SELECT +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=PLAN +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-resolve-buffer-select %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=SELECT -// End-to-end test for the N == 4 multi-buffer path: PlanMemory must -// reserve 4 physical slots and emit a 4-address `pto.pointer_cast`. -// PTOResolveBufferSelect then lowers each constant-slot `pto.slot_marker` -// to a single-address pointer_cast picking the right slot. +// End-to-end test for N == 4: PlanMemory records four physical slot addresses, +// then PTOResolveBufferSelect materializes each selected tile handle. module { func.func @n4_rotate( @@ -50,18 +50,18 @@ module { } } -// PlanMemory: 4 physical addresses, one `pto.pointer_cast` with 4 i64 args. +// PlanMemory records four physical addresses on alloc_multi_tile. // PLAN: IR Dump After PlanMemory // PLAN-LABEL: func.func @n4_rotate -// PLAN: pto.pointer_cast({{[^,]*}}, {{[^,]*}}, {{[^,]*}}, {{[^,]*}}) +// PLAN: pto.alloc_multi_tile {pto.multi_buffer_addrs = array} +// PLAN-NOT: pto.pointer_cast -// PTOResolveBufferSelect: four single-addr casts (one per slot), no leftover -// slot_marker. The original 4-addr cast remains as the alloc anchor. +// PTOResolveBufferSelect: four addressed alloc_tile ops, no multi-buffer ops. // SELECT: IR Dump After PTOResolveBufferSelect // SELECT-LABEL: func.func @n4_rotate -// SELECT-NOT: pto.slot_marker -// SELECT: pto.pointer_cast({{[^,]*}}, {{[^,]*}}, {{[^,]*}}, {{[^,]*}}) -// SELECT: pto.pointer_cast(%{{[^,)]+}}) -// SELECT: pto.pointer_cast(%{{[^,)]+}}) -// SELECT: pto.pointer_cast(%{{[^,)]+}}) -// SELECT: pto.pointer_cast(%{{[^,)]+}}) +// SELECT-NOT: pto.alloc_multi_tile +// SELECT-NOT: pto.multi_tile_get +// SELECT: pto.alloc_tile addr = %{{.*}} +// SELECT: pto.alloc_tile addr = %{{.*}} +// SELECT: pto.alloc_tile addr = %{{.*}} +// SELECT: pto.alloc_tile addr = %{{.*}} diff --git a/test/lit/pto/multi_tile_no_loop_unroll.pto b/test/lit/pto/multi_tile_no_loop_unroll.pto index e7b4373e96..fe9c70ee80 100644 --- a/test/lit/pto/multi_tile_no_loop_unroll.pto +++ b/test/lit/pto/multi_tile_no_loop_unroll.pto @@ -8,11 +8,8 @@ // RUN: ptoas --mlir-print-ir-after=pto-view-to-memref %s 2>&1 1>/dev/null | FileCheck %s -// Verifies that multi-buffer expression and slot selection work WITHOUT an -// enclosing `scf.for`. Two constant slots are tagged via `pto.slot_marker` -// at the memref layer and live independently of any loop induction -// variable. This is something the PR615 fully-automatic `iv mod N` path -// could not express. +// Verifies that tile-native multi-buffer selection works without an enclosing +// `scf.for`; the two constant slots do not depend on a loop induction value. module { func.func @unroll_two_slots( @@ -42,6 +39,8 @@ module { // CHECK: IR Dump After PTOViewToMemref // CHECK-LABEL: func.func @unroll_two_slots // CHECK-NOT: scf.for -// CHECK: memref.alloc() {pto.multi_buffer = 2 : i32} -// CHECK: pto.slot_marker %{{.*}}[%c0 -// CHECK: pto.slot_marker %{{.*}}[%c1 +// CHECK: %[[MULTI:.*]] = pto.alloc_multi_tile +// CHECK: pto.multi_tile_get %[[MULTI]][%c0 +// CHECK: pto.multi_tile_get %[[MULTI]][%c1 +// CHECK-NOT: memref.alloc +// CHECK-NOT: pto.slot_marker diff --git a/test/lit/pto/multi_tile_reject_internal_addrs.pto b/test/lit/pto/multi_tile_reject_internal_addrs.pto new file mode 100644 index 0000000000..b547f8d48f --- /dev/null +++ b/test/lit/pto/multi_tile_reject_internal_addrs.pto @@ -0,0 +1,23 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas %s 2>&1 | FileCheck %s + +// Planner-generated slot addresses are an internal contract. Users must not +// bypass level validation or memory planning by attaching the attribute. + +module { + func.func @reject_internal_addrs() { + %multi = pto.alloc_multi_tile { + pto.multi_buffer_addrs = array + } : !pto.multi_tile_buf, count=2> + return + } +} + +// CHECK: attribute 'pto.multi_buffer_addrs' is reserved for pto-plan-memory diff --git a/test/lit/pto/part_arithmetic_tile_native.pto b/test/lit/pto/part_arithmetic_tile_native.pto new file mode 100644 index 0000000000..2282b5573e --- /dev/null +++ b/test/lit/pto/part_arithmetic_tile_native.pto @@ -0,0 +1,71 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tpartadd_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tpartadd ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tpartmul_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tpartmul ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tpartmax_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tpartmax ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tpartmin_arg( + %src0: !pto.tile_buf, + %src1: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tpartmin ins(%src0, %src1 : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tpartadd_arg( +// NATIVE: pto.tpartadd ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tpartmul_arg( +// NATIVE: pto.tpartmul ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tpartmax_arg( +// NATIVE: pto.tpartmax ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tpartmin_arg( +// NATIVE: pto.tpartmin ins(%arg0, %arg1 : !pto.tile_buf, !pto.tile_buf) outs(%arg2 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tpartadd_arg( +// EMITC: TPARTADD( +// EMITC-LABEL: tpartmul_arg( +// EMITC: TPARTMUL( +// EMITC-LABEL: tpartmax_arg( +// EMITC: TPARTMAX( +// EMITC-LABEL: tpartmin_arg( +// EMITC: TPARTMIN( diff --git a/test/lit/pto/plan_memory_bind_tile_alias_liveness.pto b/test/lit/pto/plan_memory_bind_tile_alias_liveness.pto index 32af387daf..ff075d063d 100644 --- a/test/lit/pto/plan_memory_bind_tile_alias_liveness.pto +++ b/test/lit/pto/plan_memory_bind_tile_alias_liveness.pto @@ -1,4 +1,5 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s module { func.func @bind_tile_alias_liveness(%arg0: memref<16x16x16xf16, #pto.address_space>, @@ -30,6 +31,5 @@ module { // CHECK-NOT: memref.alloc // CHECK-DAG: %c0_i64 = arith.constant 0 : i64 // CHECK-DAG: %c8192_i64 = arith.constant 8192 : i64 -// CHECK-DAG: pto.pointer_cast(%c0_i64) : memref<16x16x16xf16, #pto.address_space<{{vec|ub}}>> -// CHECK-DAG: pto.pointer_cast(%c8192_i64) {{.*}} : memref<16x16x16xf16, #pto.address_space<{{vec|ub}}>> - +// CHECK-DAG: pto.pointer_cast(%c0_i64){{.*}} : memref<16x16x16xf16, #pto.address_space<{{vec|ub}}>> +// CHECK-DAG: pto.pointer_cast(%c8192_i64){{.*}} : memref<16x16x16xf16, #pto.address_space<{{vec|ub}}>> diff --git a/test/lit/pto/plan_memory_five_gates_lifetime_overlap.pto b/test/lit/pto/plan_memory_five_gates_lifetime_overlap.pto new file mode 100644 index 0000000000..3bf501afe1 --- /dev/null +++ b/test/lit/pto/plan_memory_five_gates_lifetime_overlap.pto @@ -0,0 +1,40 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level2 --pto-arch=a3 --plan-memory-impl=modern \ +// RUN: --emit-pto-ir --mlir-print-ir-after=pto-plan-memory %s 2>&1 \ +// RUN: | FileCheck %s + +module attributes {"pto.target_arch" = "a3"} { + func.func @five_gates_lifetime_overlap( + %gm0 : memref<16x16xf16, #pto.address_space>, + %gm1 : memref<16x16xf16, #pto.address_space>, + %out0 : memref<16x16xf16, #pto.address_space>, + %out1 : memref<16x16xf16, #pto.address_space>) { + %buf0 = memref.alloc() : memref<16x16xf16, #pto.address_space> + %buf1 = memref.alloc() : memref<16x16xf16, #pto.address_space> + + pto.tload ins(%gm0 : memref<16x16xf16, #pto.address_space>) + outs(%buf0 : memref<16x16xf16, #pto.address_space>) {layout = #pto.layout} + pto.tload ins(%gm1 : memref<16x16xf16, #pto.address_space>) + outs(%buf1 : memref<16x16xf16, #pto.address_space>) {layout = #pto.layout} + + pto.tstore ins(%buf0 : memref<16x16xf16, #pto.address_space>) + outs(%out0 : memref<16x16xf16, #pto.address_space>) {layout = #pto.layout} + pto.tstore ins(%buf1 : memref<16x16xf16, #pto.address_space>) + outs(%out1 : memref<16x16xf16, #pto.address_space>) {layout = #pto.layout} + return + } +} + +// CHECK-LABEL: func.func @five_gates_lifetime_overlap +// CHECK-DAG: %[[ADDR0:.*]] = arith.constant 0 : i64 +// CHECK-DAG: %[[ADDR1:.*]] = arith.constant 512 : i64 +// CHECK: pto.pointer_cast(%[[ADDR0]]) +// CHECK: pto.pointer_cast(%[[ADDR1]]) +// CHECK-NOT: memref.alloc diff --git a/test/lit/pto/plan_memory_five_gates_phi_family.pto b/test/lit/pto/plan_memory_five_gates_phi_family.pto new file mode 100644 index 0000000000..ddf295fd80 --- /dev/null +++ b/test/lit/pto/plan_memory_five_gates_phi_family.pto @@ -0,0 +1,43 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level2 --pto-arch=a3 --plan-memory-impl=modern \ +// RUN: --emit-pto-ir --mlir-print-ir-after=pto-plan-memory %s 2>&1 \ +// RUN: | FileCheck %s + +module attributes {"pto.target_arch" = "a3"} { + func.func @five_gates_phi_family( + %gm0 : memref<16x16xf16, #pto.address_space>, + %gm1 : memref<16x16xf16, #pto.address_space>, + %out : memref<16x16xf16, #pto.address_space>) { + %cond = arith.constant true + %merged = scf.if %cond -> (memref<16x16xf16, #pto.address_space>) { + %then = memref.alloc() : memref<16x16xf16, #pto.address_space> + pto.tload ins(%gm0 : memref<16x16xf16, #pto.address_space>) + outs(%then : memref<16x16xf16, #pto.address_space>) {layout = #pto.layout} + scf.yield %then : memref<16x16xf16, #pto.address_space> + } else { + %else = memref.alloc() : memref<16x16xf16, #pto.address_space> + pto.tload ins(%gm1 : memref<16x16xf16, #pto.address_space>) + outs(%else : memref<16x16xf16, #pto.address_space>) {layout = #pto.layout} + scf.yield %else : memref<16x16xf16, #pto.address_space> + } + + pto.tstore ins(%merged : memref<16x16xf16, #pto.address_space>) + outs(%out : memref<16x16xf16, #pto.address_space>) {layout = #pto.layout} + return + } +} + +// CHECK-LABEL: func.func @five_gates_phi_family +// CHECK: %[[ADDR0:.*]] = arith.constant 0 : i64 +// CHECK: scf.if +// CHECK: pto.pointer_cast(%[[ADDR0]]) +// CHECK: } else { +// CHECK: pto.pointer_cast(%[[ADDR0]]) +// CHECK-NOT: memref.alloc diff --git a/test/lit/pto/plan_memory_for_iter_args_yield.pto b/test/lit/pto/plan_memory_for_iter_args_yield.pto index bb85772ad4..6f1578aac7 100644 --- a/test/lit/pto/plan_memory_for_iter_args_yield.pto +++ b/test/lit/pto/plan_memory_for_iter_args_yield.pto @@ -1,4 +1,5 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s module { func.func @for_iter_args_yield(%arg0: memref<16x16x16xf16, #pto.address_space>, diff --git a/test/lit/pto/plan_memory_fragmentation_hole_fit.pto b/test/lit/pto/plan_memory_fragmentation_hole_fit.pto index aac3aa7640..39bb3b8494 100644 --- a/test/lit/pto/plan_memory_fragmentation_hole_fit.pto +++ b/test/lit/pto/plan_memory_fragmentation_hole_fit.pto @@ -1,4 +1,5 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s module { func.func @fragmentation_hole_fit(%arg0: memref<16x16x16xf16, #pto.address_space>, diff --git a/test/lit/pto/plan_memory_fragmentation_two_holes.pto b/test/lit/pto/plan_memory_fragmentation_two_holes.pto index 04f7437e41..4a5c8be91a 100644 --- a/test/lit/pto/plan_memory_fragmentation_two_holes.pto +++ b/test/lit/pto/plan_memory_fragmentation_two_holes.pto @@ -1,4 +1,5 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s module { func.func @fragmentation_two_holes(%arg0: memref<16x16x16xf16, #pto.address_space>, diff --git a/test/lit/pto/plan_memory_if_in_loop.pto b/test/lit/pto/plan_memory_if_in_loop.pto index f831840b20..1d902eedce 100644 --- a/test/lit/pto/plan_memory_if_in_loop.pto +++ b/test/lit/pto/plan_memory_if_in_loop.pto @@ -1,4 +1,5 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s module { func.func @if_in_loop(%arg0: memref<16x16x16xf16, #pto.address_space>, @@ -31,7 +32,7 @@ module { // CHECK: func.func @if_in_loop // CHECK-NOT: memref.alloc // CHECK-DAG: %c0_i64 = arith.constant 0 : i64 -// CHECK-DAG: %c8192_i64 = arith.constant 8192 : i64 // CHECK: scf.for // CHECK: scf.if -// CHECK: pto.pointer_cast +// CHECK: pto.pointer_cast(%c0_i64) +// CHECK: pto.pointer_cast(%c0_i64) diff --git a/test/lit/pto/plan_memory_if_yield.pto b/test/lit/pto/plan_memory_if_yield.pto index 522464fc8e..d3f1e4a578 100644 --- a/test/lit/pto/plan_memory_if_yield.pto +++ b/test/lit/pto/plan_memory_if_yield.pto @@ -1,4 +1,5 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s module { func.func @if_yield(%arg0: memref<16x16x16xf16, #pto.address_space>, diff --git a/test/lit/pto/plan_memory_inplace_forbid_alias.pto b/test/lit/pto/plan_memory_inplace_forbid_alias.pto new file mode 100644 index 0000000000..54e65a0fd9 --- /dev/null +++ b/test/lit/pto/plan_memory_inplace_forbid_alias.pto @@ -0,0 +1,60 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level2 --pto-arch=a3 --plan-memory-impl=modern \ +// RUN: --emit-pto-ir --mlir-print-ir-after=pto-plan-memory %s 2>&1 \ +// RUN: | FileCheck %s + +module attributes {"pto.target_arch" = "a3"} { + func.func @inplace_forbid_alias_ttrans() { + %src = memref.alloc() : memref<16x16xf16, #pto.address_space> + %tmp = memref.alloc() : memref<16x16xf16, #pto.address_space> + %dst = memref.alloc() : memref<16x16xf16, #pto.address_space> + + pto.ttrans ins(%src, %tmp : memref<16x16xf16, #pto.address_space>, + memref<16x16xf16, #pto.address_space>) + outs(%dst : memref<16x16xf16, #pto.address_space>) + return + } + + func.func @inplace_forbid_alias_tsel() { + %mask = memref.alloc() : memref<16x16xi16, #pto.address_space> + %src0 = memref.alloc() : memref<16x16xf16, #pto.address_space> + %src1 = memref.alloc() : memref<16x16xf16, #pto.address_space> + %tmp = memref.alloc() : memref<16x16xui8, #pto.address_space> + %dst = memref.alloc() : memref<16x16xf16, #pto.address_space> + + pto.tsel ins(%mask, %src0, %src1, %tmp + : memref<16x16xi16, #pto.address_space>, + memref<16x16xf16, #pto.address_space>, + memref<16x16xf16, #pto.address_space>, + memref<16x16xui8, #pto.address_space>) + outs(%dst : memref<16x16xf16, #pto.address_space>) + return + } +} + +// CHECK-LABEL: func.func @inplace_forbid_alias_ttrans +// CHECK-DAG: %[[A0:.*]] = arith.constant 0 : i64 +// CHECK-DAG: %[[A512:.*]] = arith.constant 512 : i64 +// CHECK-DAG: %[[A1024:.*]] = arith.constant 1024 : i64 +// CHECK: pto.pointer_cast(%[[A0]]) +// CHECK: pto.pointer_cast(%[[A512]]) +// CHECK: pto.pointer_cast(%[[A1024]]) +// CHECK-LABEL: func.func @inplace_forbid_alias_tsel +// CHECK-DAG: %[[B0:.*]] = arith.constant 0 : i64 +// CHECK-DAG: %[[B512:.*]] = arith.constant 512 : i64 +// CHECK-DAG: %[[B1024:.*]] = arith.constant 1024 : i64 +// CHECK-DAG: %[[B1536:.*]] = arith.constant 1536 : i64 +// CHECK-DAG: %[[B2048:.*]] = arith.constant 2048 : i64 +// CHECK: pto.pointer_cast(%[[B0]]) +// CHECK: pto.pointer_cast(%[[B512]]) +// CHECK: pto.pointer_cast(%[[B1024]]) +// CHECK: pto.pointer_cast(%[[B2048]]) +// CHECK: pto.pointer_cast(%[[B1536]]) +// CHECK-NOT: memref.alloc diff --git a/test/lit/pto/plan_memory_loop_in_if.pto b/test/lit/pto/plan_memory_loop_in_if.pto index 558401ec1c..2b4e06be93 100644 --- a/test/lit/pto/plan_memory_loop_in_if.pto +++ b/test/lit/pto/plan_memory_loop_in_if.pto @@ -1,4 +1,5 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s module { func.func @loop_in_if(%arg0: memref<16x16x16xf16, #pto.address_space>, @@ -34,4 +35,3 @@ module { // CHECK: scf.for // CHECK: } else { // CHECK: pto.pointer_cast - diff --git a/test/lit/pto/plan_memory_loop_no_reuse_outer_live.pto b/test/lit/pto/plan_memory_loop_no_reuse_outer_live.pto index 8e122bffeb..82131666c6 100644 --- a/test/lit/pto/plan_memory_loop_no_reuse_outer_live.pto +++ b/test/lit/pto/plan_memory_loop_no_reuse_outer_live.pto @@ -1,4 +1,5 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s module { func.func @loop_outer_live(%arg0: memref<16x16x16xf16, #pto.address_space>, diff --git a/test/lit/pto/plan_memory_nested_loops.pto b/test/lit/pto/plan_memory_nested_loops.pto index 9bfb14ed07..f4a6710512 100644 --- a/test/lit/pto/plan_memory_nested_loops.pto +++ b/test/lit/pto/plan_memory_nested_loops.pto @@ -1,4 +1,5 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s module { func.func @nested_loops(%arg0: memref<16x16x16xf16, #pto.address_space>, diff --git a/test/lit/pto/plan_memory_no_reuse_overlap.pto b/test/lit/pto/plan_memory_no_reuse_overlap.pto index 5a5d50d2ad..8e04a9e601 100644 --- a/test/lit/pto/plan_memory_no_reuse_overlap.pto +++ b/test/lit/pto/plan_memory_no_reuse_overlap.pto @@ -1,4 +1,5 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s module { func.func @no_reuse_overlap(%arg0: memref<16x16x16xf16, #pto.address_space>, diff --git a/test/lit/pto/plan_memory_order_by_size_noreuse.pto b/test/lit/pto/plan_memory_order_by_size_noreuse.pto index 6e9c2d3cfa..88ef992031 100644 --- a/test/lit/pto/plan_memory_order_by_size_noreuse.pto +++ b/test/lit/pto/plan_memory_order_by_size_noreuse.pto @@ -8,7 +8,9 @@ // contract that the option means the same thing whether or not reuse kicks in. // // RUN: ptoas --pto-arch=a3 --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=DEFAULT +// RUN: ptoas --pto-arch=a3 --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=BYSIZE // RUN: ptoas --pto-arch=a3 --plan-memory-order-by-size --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=BYSIZE +// RUN: ptoas --pto-arch=a3 --plan-memory-impl=modern --plan-memory-order-by-size --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=BYSIZE module { func.func @order_by_size_noreuse(%src_ptr: !pto.ptr, %idx_ptr: !pto.ptr, %dst_ptr: !pto.ptr) attributes {pto.kernel} { @@ -65,5 +67,5 @@ module { // On the no-reuse fast path the largest tile (dst, 1x8192xf32) gets offset 0 // ONLY with order-by-size; the default order leaves offset 0 to the smaller // first-generated input tile. -// BYSIZE: pto.pointer_cast(%c0_i64){{.*}} : memref<1x8192xf32 -// DEFAULT-NOT: pto.pointer_cast(%c0_i64){{.*}} : memref<1x8192xf32 +// BYSIZE: pto.alloc_tile addr = %c0_i64 : !pto.tile_buf +// DEFAULT-NOT: pto.alloc_tile addr = %c0_i64 : !pto.tile_buf diff --git a/test/lit/pto/plan_memory_order_by_size_reuse.pto b/test/lit/pto/plan_memory_order_by_size_reuse.pto index 05541cfd07..b83ff45090 100644 --- a/test/lit/pto/plan_memory_order_by_size_reuse.pto +++ b/test/lit/pto/plan_memory_order_by_size_reuse.pto @@ -6,7 +6,9 @@ // (first-fit-decreasing) the largest tile is allocated first and gets offset 0. // // RUN: ptoas --pto-arch=a3 --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=DEFAULT +// RUN: ptoas --pto-arch=a3 --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=BYSIZE // RUN: ptoas --pto-arch=a3 --plan-memory-order-by-size --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=BYSIZE +// RUN: ptoas --pto-arch=a3 --plan-memory-impl=modern --plan-memory-order-by-size --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s --check-prefix=BYSIZE module { func.func @order_by_size_reuse(%src_ptr: !pto.ptr, %idx_ptr: !pto.ptr, %dst_ptr: !pto.ptr) attributes {pto.kernel} { @@ -62,5 +64,5 @@ module { // The largest tile (dst, 1x32768xf32) is placed at offset 0 ONLY with // order-by-size; the default order leaves offset 0 to a smaller input tile. -// BYSIZE: pto.pointer_cast(%c0_i64){{.*}} : memref<1x32768xf32 -// DEFAULT-NOT: pto.pointer_cast(%c0_i64){{.*}} : memref<1x32768xf32 +// BYSIZE: pto.alloc_tile addr = %c0_i64 : !pto.tile_buf +// DEFAULT-NOT: pto.alloc_tile addr = %c0_i64 : !pto.tile_buf diff --git a/test/lit/pto/plan_memory_peak_8_overlapping.pto b/test/lit/pto/plan_memory_peak_8_overlapping.pto index 48d46f4537..b42f8fd997 100644 --- a/test/lit/pto/plan_memory_peak_8_overlapping.pto +++ b/test/lit/pto/plan_memory_peak_8_overlapping.pto @@ -1,4 +1,5 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s module { func.func @peak_8_overlapping(%arg0: memref<16x16x16xf16, #pto.address_space>, @@ -56,4 +57,3 @@ module { // 8 live buffers implies a max offset of 7*8192 = 57344 bytes. // CHECK: %[[O57344:.*]] = arith.constant 57344 : i64 // CHECK: pto.pointer_cast(%[[O57344]]) : memref<16x16x16xf16, #pto.address_space<{{vec|ub}}>> - diff --git a/test/lit/pto/plan_memory_peak_exact_capacity.pto b/test/lit/pto/plan_memory_peak_exact_capacity.pto index 4ef8148e1a..8ecb828a99 100644 --- a/test/lit/pto/plan_memory_peak_exact_capacity.pto +++ b/test/lit/pto/plan_memory_peak_exact_capacity.pto @@ -1,4 +1,5 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s module { func.func @peak_exact_capacity(%arg0: memref<16x16x16xf16, #pto.address_space>, @@ -138,4 +139,3 @@ module { // 24 live buffers implies a max offset of 23*8192 = 188416 bytes. // CHECK: %[[O188416:.*]] = arith.constant 188416 : i64 // CHECK: pto.pointer_cast(%[[O188416]]) : memref<16x16x16xf16, #pto.address_space<{{vec|ub}}>> - diff --git a/test/lit/pto/plan_memory_reserve_buffer_manual_reject_nested_alloc.pto b/test/lit/pto/plan_memory_reserve_buffer_manual_reject_nested_alloc.pto index 06ae763576..b89bb80022 100644 --- a/test/lit/pto/plan_memory_reserve_buffer_manual_reject_nested_alloc.pto +++ b/test/lit/pto/plan_memory_reserve_buffer_manual_reject_nested_alloc.pto @@ -1,4 +1,5 @@ // RUN: not ptoas %s 2>&1 1>/dev/null | FileCheck %s +// RUN: not ptoas --plan-memory-impl=modern %s 2>&1 1>/dev/null | FileCheck %s module { func.func @manual_nested_alloc(%cond: i1) { @@ -22,4 +23,4 @@ module { } } -// CHECK: error: 'pto.reserve_buffer' op pto.reserve_buffer with explicit 'base' (auto = false) is not supported in PlanMemory; use --pto-level=level3 or set auto = true +// CHECK: error: pto.reserve_buffer with explicit 'base' (auto = false) is not supported when --pto-level=level1 or level2; use --pto-level=level3 or set auto = true diff --git a/test/lit/pto/plan_memory_reserve_buffer_prefix.pto b/test/lit/pto/plan_memory_reserve_buffer_prefix.pto index fd4a577eb4..a5bcc2d249 100644 --- a/test/lit/pto/plan_memory_reserve_buffer_prefix.pto +++ b/test/lit/pto/plan_memory_reserve_buffer_prefix.pto @@ -1,4 +1,5 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s module { func.func @reserve_prefix(%arg0: memref<16x16x16xf16, #pto.address_space>, diff --git a/test/lit/pto/plan_memory_reuse_sequential.pto b/test/lit/pto/plan_memory_reuse_sequential.pto index 6ed1a49599..425a3b399f 100644 --- a/test/lit/pto/plan_memory_reuse_sequential.pto +++ b/test/lit/pto/plan_memory_reuse_sequential.pto @@ -1,4 +1,5 @@ // RUN: ptoas --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s +// RUN: ptoas --plan-memory-impl=modern --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s module { func.func @reuse_sequential(%arg0: memref<16x16x16xf16, #pto.address_space>, diff --git a/test/lit/pto/plan_memory_reused_tstore_sync_level2.pto b/test/lit/pto/plan_memory_reused_tstore_sync_level2.pto index 91fd32e8b4..06a47e100b 100644 --- a/test/lit/pto/plan_memory_reused_tstore_sync_level2.pto +++ b/test/lit/pto/plan_memory_reused_tstore_sync_level2.pto @@ -8,7 +8,7 @@ // End-to-end level2 coverage for PlanMemory reuse followed by InsertSync. The // source uses logical alloc_tile operations without explicit addresses, so the -// physical pointer_cast addresses checked below are produced by PlanMemory. +// physical alloc_tile addresses checked below are produced by PlanMemory. // `stored` occupies [8192, 16384), while `reused` occupies [8448, 16640). // // RUN: ptoas --pto-arch=a3 --pto-level=level2 --enable-insert-sync --emit-pto-ir --mlir-print-ir-after=pto-plan-memory %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=PLAN @@ -70,13 +70,11 @@ module attributes {pto.target_arch = "a2a3"} { } // PLAN-LABEL: func.func @plan_memory_reused_tstore_sync_level2( -// PLAN-DAG: %[[STORED_ADDR:[A-Za-z0-9_]+]] = arith.constant 8192 : i64 // PLAN-DAG: %[[REUSED_ADDR:[A-Za-z0-9_]+]] = arith.constant 8448 : i64 -// PLAN: %[[STORED_CAST:[0-9]+]] = pto.pointer_cast(%[[STORED_ADDR]]){{.*}} : memref<16x128xf32 -// PLAN-NEXT: %[[STORED:[0-9]+]] = pto.bind_tile %[[STORED_CAST]] +// PLAN-DAG: %[[STORED_ADDR:[A-Za-z0-9_]+]] = arith.constant 8192 : i64 +// PLAN: %[[STORED:[0-9]+]] = pto.alloc_tile addr = %[[STORED_ADDR]] : !pto.tile_buf // PLAN: pto.tstore ins(%[[STORED]] -// PLAN: %[[REUSED_CAST:[0-9]+]] = pto.pointer_cast(%[[REUSED_ADDR]]){{.*}} : memref<16x128xf32 -// PLAN-NEXT: %[[REUSED:[0-9]+]] = pto.bind_tile %[[REUSED_CAST]] +// PLAN: %[[REUSED:[0-9]+]] = pto.alloc_tile addr = %[[REUSED_ADDR]] : !pto.tile_buf // PLAN: pto.tmuls {{.*}} outs(%[[REUSED]] // SYNC-LABEL: AICORE void plan_memory_reused_tstore_sync_level2( diff --git a/test/lit/pto/plan_memory_spec_level0_no_reuse_overlap.pto b/test/lit/pto/plan_memory_spec_level0_no_reuse_overlap.pto new file mode 100644 index 0000000000..fe520f08bc --- /dev/null +++ b/test/lit/pto/plan_memory_spec_level0_no_reuse_overlap.pto @@ -0,0 +1,31 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level2 --pto-arch=a3 --plan-memory-impl=modern --emit-pto-ir \ +// RUN: --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s + +module attributes {"pto.target_arch" = "a3"} { + func.func @spec_level0_no_reuse_overlap( + %gm : memref<32x32xf16, #pto.address_space>) { + %src = memref.alloc() : memref<32x32xf16, #pto.address_space> + %dst = memref.alloc() : memref<32x32xf16, #pto.address_space> + + pto.tload ins(%gm : memref<32x32xf16, #pto.address_space>) + outs(%src : memref<32x32xf16, #pto.address_space>) {layout = #pto.layout} + pto.tmov ins(%src : memref<32x32xf16, #pto.address_space>) + outs(%dst : memref<32x32xf16, #pto.address_space>) + return + } +} + +// CHECK-LABEL: func.func @spec_level0_no_reuse_overlap +// CHECK-DAG: %[[ADDR0:.*]] = arith.constant 0 : i64 +// CHECK-DAG: %[[ADDR1:.*]] = arith.constant 2048 : i64 +// CHECK: pto.pointer_cast(%[[ADDR0]]) +// CHECK: pto.pointer_cast(%[[ADDR1]]) +// CHECK-NOT: memref.alloc diff --git a/test/lit/pto/plan_memory_spec_level0_reuse.pto b/test/lit/pto/plan_memory_spec_level0_reuse.pto new file mode 100644 index 0000000000..b55d457086 --- /dev/null +++ b/test/lit/pto/plan_memory_spec_level0_reuse.pto @@ -0,0 +1,31 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level2 --pto-arch=a3 --plan-memory-impl=modern --emit-pto-ir \ +// RUN: --mlir-print-ir-after=pto-plan-memory %s 2>&1 1>/dev/null | FileCheck %s + +module attributes {"pto.target_arch" = "a3"} { + func.func @spec_level0_reuse( + %gm0 : memref<32x32xf16, #pto.address_space>, + %gm1 : memref<32x32xf16, #pto.address_space>) { + %buf0 = memref.alloc() : memref<32x32xf16, #pto.address_space> + pto.tload ins(%gm0 : memref<32x32xf16, #pto.address_space>) + outs(%buf0 : memref<32x32xf16, #pto.address_space>) {layout = #pto.layout} + + %buf1 = memref.alloc() : memref<32x32xf16, #pto.address_space> + pto.tload ins(%gm1 : memref<32x32xf16, #pto.address_space>) + outs(%buf1 : memref<32x32xf16, #pto.address_space>) {layout = #pto.layout} + return + } +} + +// CHECK-LABEL: func.func @spec_level0_reuse +// CHECK: %[[ADDR0:.*]] = arith.constant 0 : i64 +// CHECK: %[[BUF0:.*]] = pto.pointer_cast(%[[ADDR0]]) +// CHECK: %[[BUF1:.*]] = pto.pointer_cast(%[[ADDR0]]) +// CHECK-NOT: memref.alloc diff --git a/test/lit/pto/pow_rsqrt_tile_native.pto b/test/lit/pto/pow_rsqrt_tile_native.pto new file mode 100644 index 0000000000..b2a99514e6 --- /dev/null +++ b/test/lit/pto/pow_rsqrt_tile_native.pto @@ -0,0 +1,41 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tpow_arg(%base: !pto.tile_buf, %exp: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tpow ins(%base, %exp, %tmp : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tpows_arg(%src: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf, %scalar: f32) { + pto.tpows ins(%src, %scalar, %tmp : !pto.tile_buf, f32, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trsqrt_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trsqrt ins(%src : !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @tpow_arg +// NATIVE: pto.tpow +// NATIVE-LABEL: @tpows_arg +// NATIVE: pto.tpows +// NATIVE-LABEL: @trsqrt_arg +// NATIVE: pto.trsqrt +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tpow_arg( +// EMITC: TPOW( +// EMITC-LABEL: tpows_arg( +// EMITC: TPOWS( +// EMITC-LABEL: trsqrt_arg( +// EMITC: TRSQRT( diff --git a/test/lit/pto/print_tile_native.pto b/test/lit/pto/print_tile_native.pto new file mode 100644 index 0000000000..84247d2a5f --- /dev/null +++ b/test/lit/pto/print_tile_native.pto @@ -0,0 +1,40 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a5 %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tprint_arg( + %src: !pto.tile_buf) { + pto.tprint ins(%src : !pto.tile_buf) + return + } + + func.func private @tprint_tmp_arg( + %src: !pto.tile_buf, + %tmp: !pto.partition_tensor_view<16x16xf32>) { + pto.tprint ins(%src, %tmp : !pto.tile_buf, !pto.partition_tensor_view<16x16xf32>) + {printFormat = #pto} + return + } +} + +// NATIVE-LABEL: func.func private @tprint_arg( +// NATIVE: pto.tprint ins(%arg0 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tprint_tmp_arg( +// NATIVE: pto.tprint ins(%arg0, %arg1 : !pto.tile_buf, memref<16x16xf32>) +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tprint_arg( +// EMITC: TPRINT( +// EMITC-LABEL: tprint_tmp_arg( +// EMITC: TPRINT( diff --git a/test/lit/pto/ptodsl_subkernel_call_autosync.pto b/test/lit/pto/ptodsl_subkernel_call_autosync.pto index 798c67b3df..8cd65ca69a 100644 --- a/test/lit/pto/ptodsl_subkernel_call_autosync.pto +++ b/test/lit/pto/ptodsl_subkernel_call_autosync.pto @@ -42,6 +42,8 @@ module { // CHECK: set_flag(PIPE_MTE2, PIPE_V, EVENT_ID0); // CHECK: wait_flag(PIPE_MTE2, PIPE_V, EVENT_ID0); // CHECK: TNEG( +// CHECK: pipe_barrier(PIPE_V); +// CHECK: TNEG( // CHECK: set_flag(PIPE_V, PIPE_MTE3, EVENT_ID0); // CHECK: wait_flag(PIPE_V, PIPE_MTE3, EVENT_ID0); // CHECK: TSTORE( diff --git a/test/lit/pto/quant_mx_tile_native.pto b/test/lit/pto/quant_mx_tile_native.pto new file mode 100644 index 0000000000..2d154a36d5 --- /dev/null +++ b/test/lit/pto/quant_mx_tile_native.pto @@ -0,0 +1,26 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tquant_mx_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf, %exp: !pto.tile_buf, %max: !pto.tile_buf, %scaling: !pto.tile_buf) { + pto.tquant.mx ins(%src : !pto.tile_buf) outs(%dst, %exp, %max, %scaling : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) {quant_type = #pto} + return + } +} + +// NATIVE-LABEL: @tquant_mx_arg +// NATIVE-SAME: !pto.tile_buf< +// NATIVE: pto.tquant.mx +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tquant_mx_arg( +// EMITC: TQUANT&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @trowmax_arg(%src: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trowmax ins(%src, %tmp : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trowmin_arg(%src: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trowmin ins(%src, %tmp : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trowsum_arg(%src: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trowsum ins(%src, %tmp : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trowprod_arg(%src: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trowprod ins(%src, %tmp : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tcolprod_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tcolprod ins(%src : !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @trowmax_arg +// NATIVE: pto.trowmax +// NATIVE-LABEL: @trowmin_arg +// NATIVE: pto.trowmin +// NATIVE-LABEL: @trowsum_arg +// NATIVE: pto.trowsum +// NATIVE-LABEL: @trowprod_arg +// NATIVE: pto.trowprod +// NATIVE-LABEL: @tcolprod_arg +// NATIVE: pto.tcolprod +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: trowmax_arg( +// EMITC: TROWMAX( +// EMITC-LABEL: trowmin_arg( +// EMITC: TROWMIN( +// EMITC-LABEL: trowsum_arg( +// EMITC: TROWSUM( +// EMITC-LABEL: trowprod_arg( +// EMITC: TROWPROD( +// EMITC-LABEL: tcolprod_arg( +// EMITC: TCOLPROD( diff --git a/test/lit/pto/reserve_buffer_level12_auto_base_reject.pto b/test/lit/pto/reserve_buffer_level12_auto_base_reject.pto new file mode 100644 index 0000000000..cc0c462343 --- /dev/null +++ b/test/lit/pto/reserve_buffer_level12_auto_base_reject.pto @@ -0,0 +1,25 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-level=level1 --pto-arch=a3 %s 2>&1 | FileCheck %s +// RUN: not ptoas --pto-level=level2 --pto-arch=a3 %s 2>&1 | FileCheck %s + +module { + func.func @auto_base_reject() { + %fifo = pto.reserve_buffer { + name = "fifo", + size = 8192, + location = #pto.address_space, + auto = true, + base = 0 + } -> i32 + return + } +} + +// CHECK: error: unexpected 'base' on auto reserve_buffer diff --git a/test/lit/pto/reserve_buffer_level12_manual_base_reject.pto b/test/lit/pto/reserve_buffer_level12_manual_base_reject.pto new file mode 100644 index 0000000000..3f1069f574 --- /dev/null +++ b/test/lit/pto/reserve_buffer_level12_manual_base_reject.pto @@ -0,0 +1,25 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-level=level1 --pto-arch=a3 %s 2>&1 | FileCheck %s +// RUN: not ptoas --pto-level=level2 --pto-arch=a3 %s 2>&1 | FileCheck %s + +module { + func.func @manual_base_reject() { + %fifo = pto.reserve_buffer { + name = "fifo", + size = 8192, + location = #pto.address_space, + auto = false, + base = 0 + } -> i32 + return + } +} + +// CHECK: error: pto.reserve_buffer with explicit 'base' (auto = false) is not supported when --pto-level=level1 or level2 diff --git a/test/lit/pto/reserve_buffer_level3_auto_reject.pto b/test/lit/pto/reserve_buffer_level3_auto_reject.pto new file mode 100644 index 0000000000..cb258ef978 --- /dev/null +++ b/test/lit/pto/reserve_buffer_level3_auto_reject.pto @@ -0,0 +1,23 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: not ptoas --pto-level=level3 --pto-arch=a3 %s 2>&1 | FileCheck %s + +module { + func.func @level3_auto_reject() { + %fifo = pto.reserve_buffer { + name = "fifo", + size = 8192, + location = #pto.address_space, + auto = true + } -> i32 + return + } +} + +// CHECK: error: pto.reserve_buffer requires 'auto = false' and explicit 'base' when --pto-level=level3 diff --git a/test/lit/pto/reserve_buffer_level3_manual_base.pto b/test/lit/pto/reserve_buffer_level3_manual_base.pto new file mode 100644 index 0000000000..0164b8474a --- /dev/null +++ b/test/lit/pto/reserve_buffer_level3_manual_base.pto @@ -0,0 +1,27 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --emit-pto-ir \ +// RUN: --mlir-print-ir-after=pto-resolve-reserved-buffers %s 2>&1 1>/dev/null | FileCheck %s + +module { + func.func @manual_base_level3() { + %fifo = pto.reserve_buffer { + name = "fifo", + size = 8192, + location = #pto.address_space, + auto = false, + base = 0 + } -> i32 + return + } +} + +// CHECK-LABEL: func.func @manual_base_level3 +// CHECK: arith.constant 0 : i32 +// CHECK-NOT: pto.reserve_buffer diff --git a/test/lit/pto/rowexpand_tile_native.pto b/test/lit/pto/rowexpand_tile_native.pto new file mode 100644 index 0000000000..1eb14eaeaf --- /dev/null +++ b/test/lit/pto/rowexpand_tile_native.pto @@ -0,0 +1,82 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @trowexpand_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trowexpand ins(%src : !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trowexpandadd_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trowexpandadd ins(%a, %b : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trowexpandsub_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trowexpandsub ins(%a, %b : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trowexpandmul_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trowexpandmul ins(%a, %b : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trowexpanddiv_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trowexpanddiv ins(%a, %b : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trowexpandmax_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trowexpandmax ins(%a, %b : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trowexpandmin_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trowexpandmin ins(%a, %b : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trowexpandexpdif_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trowexpandexpdif ins(%a, %b : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @trowexpand_arg +// NATIVE-SAME: !pto.tile_buf< +// NATIVE: pto.trowexpand +// NATIVE-LABEL: @trowexpandadd_arg +// NATIVE: pto.trowexpandadd +// NATIVE-LABEL: @trowexpandsub_arg +// NATIVE: pto.trowexpandsub +// NATIVE-LABEL: @trowexpandmul_arg +// NATIVE: pto.trowexpandmul +// NATIVE-LABEL: @trowexpanddiv_arg +// NATIVE: pto.trowexpanddiv +// NATIVE-LABEL: @trowexpandmax_arg +// NATIVE: pto.trowexpandmax +// NATIVE-LABEL: @trowexpandmin_arg +// NATIVE: pto.trowexpandmin +// NATIVE-LABEL: @trowexpandexpdif_arg +// NATIVE: pto.trowexpandexpdif +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: trowexpand_arg( +// EMITC: TROWEXPAND( +// EMITC-LABEL: trowexpandadd_arg( +// EMITC: TROWEXPANDADD( +// EMITC-LABEL: trowexpandsub_arg( +// EMITC: TROWEXPANDSUB( +// EMITC-LABEL: trowexpandmul_arg( +// EMITC: TROWEXPANDMUL( +// EMITC-LABEL: trowexpanddiv_arg( +// EMITC: TROWEXPANDDIV( +// EMITC-LABEL: trowexpandmax_arg( +// EMITC: TROWEXPANDMAX( +// EMITC-LABEL: trowexpandmin_arg( +// EMITC: TROWEXPANDMIN( +// EMITC-LABEL: trowexpandexpdif_arg( +// EMITC: TROWEXPANDEXPDIF( diff --git a/test/lit/pto/scratch_elementwise_tile_native.pto b/test/lit/pto/scratch_elementwise_tile_native.pto new file mode 100644 index 0000000000..af45853b52 --- /dev/null +++ b/test/lit/pto/scratch_elementwise_tile_native.pto @@ -0,0 +1,41 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tprelu_arg(%src: !pto.tile_buf, %slope: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tprelu ins(%src, %slope, %tmp : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trem_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.trem ins(%a, %b, %tmp : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trems_arg(%src: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf, %scalar: f32) { + pto.trems ins(%src, %scalar, %tmp : !pto.tile_buf, f32, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @tprelu_arg +// NATIVE: pto.tprelu +// NATIVE-LABEL: @trem_arg +// NATIVE: pto.trem +// NATIVE-LABEL: @trems_arg +// NATIVE: pto.trems +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tprelu_arg( +// EMITC: TPRELU( +// EMITC-LABEL: trem_arg( +// EMITC: TREM( +// EMITC-LABEL: trems_arg( +// EMITC: TREMS( diff --git a/test/lit/pto/select_tile_native.pto b/test/lit/pto/select_tile_native.pto new file mode 100644 index 0000000000..c79fde6bc4 --- /dev/null +++ b/test/lit/pto/select_tile_native.pto @@ -0,0 +1,33 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tsel_arg(%mask: !pto.tile_buf, %a: !pto.tile_buf, %b: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tsel ins(%mask, %a, %b, %tmp : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tsels_arg(%mask: !pto.tile_buf, %src: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf, %scalar: i16) { + pto.tsels ins(%mask, %src, %tmp, %scalar : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf, i16) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @tsel_arg +// NATIVE: pto.tsel +// NATIVE-LABEL: @tsels_arg +// NATIVE: pto.tsels +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tsel_arg( +// EMITC: TSEL( +// EMITC-LABEL: tsels_arg( +// EMITC: TSELS( diff --git a/test/lit/pto/shift_tile_native.pto b/test/lit/pto/shift_tile_native.pto new file mode 100644 index 0000000000..0470930f7f --- /dev/null +++ b/test/lit/pto/shift_tile_native.pto @@ -0,0 +1,53 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tshl_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tshl ins(%a, %b : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tshr_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tshr ins(%a, %b : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tshls_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf, %shift: i16) { + pto.tshls ins(%src, %shift : !pto.tile_buf, i16) outs(%dst : !pto.tile_buf) + return + } + func.func private @tshrs_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf, %shift: i16) { + pto.tshrs ins(%src, %shift : !pto.tile_buf, i16) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @tshl_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.tshl +// NATIVE-LABEL: @tshr_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.tshr +// NATIVE-LABEL: @tshls_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.tshls +// NATIVE-LABEL: @tshrs_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.tshrs +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tshl_arg( +// EMITC: TSHL( +// EMITC-LABEL: tshr_arg( +// EMITC: TSHR( +// EMITC-LABEL: tshls_arg( +// EMITC: TSHLS( +// EMITC-LABEL: tshrs_arg( +// EMITC: TSHRS( diff --git a/test/lit/pto/sort32_tile_native.pto b/test/lit/pto/sort32_tile_native.pto new file mode 100644 index 0000000000..d3d0648a85 --- /dev/null +++ b/test/lit/pto/sort32_tile_native.pto @@ -0,0 +1,33 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tsort32_arg(%src: !pto.tile_buf, %idx: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tsort32 ins(%src, %idx : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tsort32_tmp_arg(%src: !pto.tile_buf, %idx: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tsort32 ins(%src, %idx, %tmp : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @tsort32_arg +// NATIVE: pto.tsort32 +// NATIVE-LABEL: @tsort32_tmp_arg +// NATIVE: pto.tsort32 +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tsort32_arg( +// EMITC: TSORT32( +// EMITC-LABEL: tsort32_tmp_arg( +// EMITC: TSORT32( diff --git a/test/lit/pto/store_fp_tile_native.pto b/test/lit/pto/store_fp_tile_native.pto new file mode 100644 index 0000000000..72b30a402d --- /dev/null +++ b/test/lit/pto/store_fp_tile_native.pto @@ -0,0 +1,27 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tstore_fp_arg(%src: !pto.tile_buf, %fp: !pto.tile_buf, %dst: memref<1x32xf16, #pto.address_space>) { + pto.tstore_fp ins(%src, %fp : !pto.tile_buf, !pto.tile_buf) outs(%dst : memref<1x32xf16, #pto.address_space>) + return + } +} + +// NATIVE-LABEL: @tstore_fp_arg +// NATIVE-SAME: !pto.tile_buf> +// NATIVE: pto.tstore_fp +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tstore_fp_arg( +// EMITC: TSTORE_FP( diff --git a/test/lit/pto/sub_tile_native.pto b/test/lit/pto/sub_tile_native.pto new file mode 100644 index 0000000000..676fa74573 --- /dev/null +++ b/test/lit/pto/sub_tile_native.pto @@ -0,0 +1,58 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tsub_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tsub ins(%a, %b : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tsubs_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf, %scalar: i16) { + pto.tsubs ins(%src, %scalar : !pto.tile_buf, i16) outs(%dst : !pto.tile_buf) + return + } + func.func private @tsubc_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %c: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tsubc ins(%a, %b, %c : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @tsubsc_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %dst: !pto.tile_buf, %scalar: i16) { + pto.tsubsc ins(%a, %scalar, %b : !pto.tile_buf, i16, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tsub_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.tsub +// NATIVE-LABEL: func.func private @tsubs_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.tsubs +// NATIVE-LABEL: func.func private @tsubc_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.tsubc +// NATIVE-LABEL: func.func private @tsubsc_arg +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.tsubsc +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tsub_arg( +// EMITC: TSUB( +// EMITC-LABEL: tsubs_arg( +// EMITC: TSUBS( +// EMITC-LABEL: tsubc_arg( +// EMITC: TSUB( +// EMITC: TADD( +// EMITC-LABEL: tsubsc_arg( +// EMITC: TSUBS( +// EMITC: TADD( diff --git a/test/lit/pto/subview_bind_tile_preserve_stride.pto b/test/lit/pto/subview_bind_tile_preserve_stride.pto index c611c540e2..9e2ca7168f 100644 --- a/test/lit/pto/subview_bind_tile_preserve_stride.pto +++ b/test/lit/pto/subview_bind_tile_preserve_stride.pto @@ -47,5 +47,4 @@ module { // CHECK-DAG: TSTORE( // CHECK-DAG: TSTORE( // CHECK-DAG: TSTORE( -// CHECK-NOT: Tile // CHECK-NOT: copy_ubuf_to_gm_align_b32( diff --git a/test/lit/pto/subview_boxed_row_major_column_slice_preserve_parent_shape_a5.pto b/test/lit/pto/subview_boxed_row_major_column_slice_preserve_parent_shape_a5.pto index 1ecda2915c..d4fd3668d2 100644 --- a/test/lit/pto/subview_boxed_row_major_column_slice_preserve_parent_shape_a5.pto +++ b/test/lit/pto/subview_boxed_row_major_column_slice_preserve_parent_shape_a5.pto @@ -34,8 +34,7 @@ module { // IR: %[[ELEM_BYTES:.*]] = arith.constant 4 : i64 // IR: %[[BYTE_OFFSET:.*]] = arith.muli %[[ELEM_OFFSET]], %[[ELEM_BYTES]] : i64 // IR: %[[ADDR:.*]] = arith.addi %{{.*}}, %[[BYTE_OFFSET]] : i64 -// IR: pto.alloc_tile addr = %[[ADDR]] : !pto.tile_buf -// IR-NOT: pto.view_semantics +// IR: pto.alloc_tile addr = %[[ADDR]] {pto.view_semantics = "subview"} : !pto.tile_buf // IR-NOT: pto.materialize_tile // EMITC: Tile diff --git a/test/lit/pto/subview_col_major_compact_keeps_normal_shape.pto b/test/lit/pto/subview_col_major_compact_keeps_normal_shape.pto index 4f635b6225..108c0b070a 100644 --- a/test/lit/pto/subview_col_major_compact_keeps_normal_shape.pto +++ b/test/lit/pto/subview_col_major_compact_keeps_normal_shape.pto @@ -20,5 +20,5 @@ module { } } -// CHECK: Tile -// CHECK-NOT: Tile +// CHECK: Tile +// CHECK-NOT: Tile diff --git a/test/lit/pto/subview_col_major_noncompact_preserve_stride.pto b/test/lit/pto/subview_col_major_noncompact_preserve_stride.pto index ef374d41d1..644bbe8386 100644 --- a/test/lit/pto/subview_col_major_noncompact_preserve_stride.pto +++ b/test/lit/pto/subview_col_major_noncompact_preserve_stride.pto @@ -22,4 +22,3 @@ module { } // CHECK: Tile -// CHECK-NOT: Tile diff --git a/test/lit/pto/subview_col_major_row_plus_one_stride_offset.pto b/test/lit/pto/subview_col_major_row_plus_one_stride_offset.pto index b83bb4bdfd..847484b8ca 100644 --- a/test/lit/pto/subview_col_major_row_plus_one_stride_offset.pto +++ b/test/lit/pto/subview_col_major_row_plus_one_stride_offset.pto @@ -21,6 +21,7 @@ module { } // ColMajor + RowPlusOne: major stride should be 17 (not 16). -// CHECK: pto.pointer_cast(%c0_i64) {{.*}} : memref<16x16xf32, strided<[1, 17]>, #pto.address_space> -// CHECK: compact=#pto.compact_mode -// CHECK: memref.subview {{.*}} to memref<8x8xf32, strided<[1, 17], offset: ?>, #pto.address_space> +// CHECK: pto.alloc_tile addr = %c0_i64 : !pto.tile_buf +// CHECK: %c17_i64 = arith.constant 17 : i64 +// CHECK: arith.muli %{{.*}}, %c17_i64 : i64 +// CHECK: pto.alloc_tile addr = %{{.*}} {pto.view_semantics = "subview"} : !pto.tile_buf diff --git a/test/lit/pto/subview_compact_keeps_normal_shape.pto b/test/lit/pto/subview_compact_keeps_normal_shape.pto index 82092c7745..ae18825a05 100644 --- a/test/lit/pto/subview_compact_keeps_normal_shape.pto +++ b/test/lit/pto/subview_compact_keeps_normal_shape.pto @@ -20,5 +20,5 @@ module { } } -// CHECK: Tile -// CHECK-NOT: Tile +// CHECK: Tile +// CHECK-NOT: Tile diff --git a/test/lit/pto/subview_dynamic_offset_static_valid_regression.pto b/test/lit/pto/subview_dynamic_offset_static_valid_regression.pto index 9826fc294f..f9f92a74b3 100644 --- a/test/lit/pto/subview_dynamic_offset_static_valid_regression.pto +++ b/test/lit/pto/subview_dynamic_offset_static_valid_regression.pto @@ -1,4 +1,5 @@ // RUN: ptoas --emit-pto-ir %s -o - | FileCheck %s +// RUN: ptoas %s --mlir-print-ir-after=pto-view-to-memref -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE module { func.func @subview_dynamic_offset_static_valid_regression( @@ -24,5 +25,11 @@ module { } // CHECK: func.func @subview_dynamic_offset_static_valid_regression -// CHECK: memref.subview %{{.*}}[0, %{{.*}}] [1, 64] [1, 1] -// CHECK: pto.bind_tile %{{.*}}, %c1, %c64 +// CHECK: %[[BYTE_OFFSET:.*]] = arith.muli %{{.*}}, %c4_i64 : i64 +// CHECK: %[[ADDR:.*]] = arith.addi %c0_i64, %[[BYTE_OFFSET]] : i64 +// CHECK: pto.alloc_tile addr = %[[ADDR]] {pto.view_semantics = "subview"} : !pto.tile_buf + +// NATIVE: IR Dump After PTOViewToMemref +// NATIVE: pto.subview %{{.*}}[%c0, %{{.*}}] sizes [1, 64] : !pto.tile_buf -> !pto.tile_buf +// NATIVE-NOT: memref.subview +// NATIVE-NOT: pto.bind_tile diff --git a/test/lit/pto/subview_row_plus_one_stride_offset.pto b/test/lit/pto/subview_row_plus_one_stride_offset.pto index c2587b3a6e..80d1625663 100644 --- a/test/lit/pto/subview_row_plus_one_stride_offset.pto +++ b/test/lit/pto/subview_row_plus_one_stride_offset.pto @@ -21,6 +21,7 @@ module { } // RowPlusOne: major stride should be 17 (not 16). -// CHECK: pto.pointer_cast(%c0_i64) {{.*}} : memref<16x16xf32, strided<[17, 1]>, #pto.address_space> -// CHECK: compact=#pto.compact_mode -// CHECK: memref.subview {{.*}} to memref<8x8xf32, strided<[17, 1], offset: ?>, #pto.address_space> +// CHECK: pto.alloc_tile addr = %c0_i64 : !pto.tile_buf +// CHECK: %c17_i64 = arith.constant 17 : i64 +// CHECK: arith.muli %{{.*}}, %c17_i64 : i64 +// CHECK: pto.alloc_tile addr = %{{.*}} {pto.view_semantics = "subview"} : !pto.tile_buf diff --git a/test/lit/pto/subview_tmatmul_acc_slices_a5.pto b/test/lit/pto/subview_tmatmul_acc_slices_a5.pto index e2cf4f04c4..102c03a4cf 100644 --- a/test/lit/pto/subview_tmatmul_acc_slices_a5.pto +++ b/test/lit/pto/subview_tmatmul_acc_slices_a5.pto @@ -24,5 +24,5 @@ module { } } -// CHECK-COUNT-2: Tile +// CHECK-COUNT-2: Tile // CHECK-COUNT-2: TMATMUL( diff --git a/test/lit/pto/subview_validshape_guard.pto b/test/lit/pto/subview_validshape_guard.pto index fafcdea660..2720df1937 100644 --- a/test/lit/pto/subview_validshape_guard.pto +++ b/test/lit/pto/subview_validshape_guard.pto @@ -67,10 +67,10 @@ module { // Default non-compact subview keeps parent physical shape (4x16) but valid // defaults to subview shape (2x8), not clipped from parent valid shape. // CHECK: Tile -// CHECK-NOT: Tile +// CHECK-NOT: Tile -// Default row-major full-col subview can use the compact child physical shape (2x8). -// CHECK: Tile +// Default row-major full-col subview keeps parent physical shape (4x8). +// CHECK: Tile // Explicit valid override takes effect. // CHECK: Tile diff --git a/test/lit/pto/tassign_tile_native_arg.pto b/test/lit/pto/tassign_tile_native_arg.pto new file mode 100644 index 0000000000..29b945d619 --- /dev/null +++ b/test/lit/pto/tassign_tile_native_arg.pto @@ -0,0 +1,32 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a5 %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tassign_arg(%tile: !pto.tile_buf) + -> !pto.tile_buf { + %addr = arith.constant 0 : i64 + %rebound = pto.tassign %tile, %addr + : !pto.tile_buf -> !pto.tile_buf + return %rebound : !pto.tile_buf + } +} + +// NATIVE-LABEL: func.func private @tassign_arg +// NATIVE-SAME: (%[[TILE:.*]]: !pto.tile_buf) +// NATIVE: %{{.*}} = pto.tassign %[[TILE]], %{{.*}} : !pto.tile_buf -> !pto.tile_buf +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tassign_arg( +// EMITC: TASSIGN( diff --git a/test/lit/pto/tcvt_low_precision_a5_valid.pto b/test/lit/pto/tcvt_low_precision_a5_valid.pto index 7424b7c508..94c02082cd 100644 --- a/test/lit/pto/tcvt_low_precision_a5_valid.pto +++ b/test/lit/pto/tcvt_low_precision_a5_valid.pto @@ -28,11 +28,12 @@ module { } // CHECK: func.func @tcvt_low_precision_a5_valid() attributes {pto.kernel_kind = #pto.kernel_kind} -// CHECK: pto.declare_tile_memref -> memref<16x8x!pto.f4E1M2x2 -// CHECK: pto.declare_tile_memref -> memref<16x16x!pto.hif8 -// CHECK: pto.tcvt ins(%1 {rmode = #pto, satmode = #pto} : memref<16x16xf32 -// CHECK: outs(%3 : memref<16x16xf8E4M3FN -// CHECK: pto.tcvt ins(%5 {rmode = #pto, satmode = #pto} : memref<16x16xbf16 -// CHECK: outs(%7 : memref<16x8x!pto.f4E1M2x2 -// CHECK: pto.tcvt ins(%9 {rmode = #pto, satmode = #pto} : memref<16x16x!pto.hif8 -// CHECK: outs(%11 : memref<16x16xf32 +// CHECK: pto.declare_tile -> !pto.tile_buf +// CHECK: pto.declare_tile -> !pto.tile_buf +// CHECK-NOT: pto.declare_tile_memref +// CHECK: pto.tcvt ins(%{{.*}} {rmode = #pto, satmode = #pto} : !pto.tile_buf +// CHECK: outs(%{{.*}} : !pto.tile_buf) +// CHECK: pto.tcvt ins(%{{.*}} {rmode = #pto, satmode = #pto} : !pto.tile_buf +// CHECK: outs(%{{.*}} : !pto.tile_buf) +// CHECK: pto.tcvt ins(%{{.*}} {rmode = #pto, satmode = #pto} : !pto.tile_buf +// CHECK: outs(%{{.*}} : !pto.tile_buf) diff --git a/test/lit/pto/tile_assemble_level3_subview_tmov_emitc.pto b/test/lit/pto/tile_assemble_level3_subview_tmov_emitc.pto new file mode 100644 index 0000000000..85a92aff72 --- /dev/null +++ b/test/lit/pto/tile_assemble_level3_subview_tmov_emitc.pto @@ -0,0 +1,43 @@ +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-insert-sync %s | FileCheck %s + +module attributes {"pto.target_arch" = "a5"} { + func.func @tile_assemble_level3_subview_tmov_emitc( + %x: memref<32x32xf32, #pto.address_space>, + %src: memref<32x16xf32, #pto.address_space>, + %out: memref<32x32xf32, #pto.address_space>) { + %c0 = arith.constant 0 : index + %c16 = arith.constant 16 : index + %c32 = arith.constant 32 : index + %addr0 = arith.constant 0 : i64 + %addr1 = arith.constant 4096 : i64 + + %tile_x = pto.alloc_tile addr = %addr0 : + !pto.tile_buf + %tile_src = pto.alloc_tile addr = %addr1 : + !pto.tile_buf + + pto.tload ins(%x : memref<32x32xf32, #pto.address_space>) + outs(%tile_x : !pto.tile_buf) + pto.tload ins(%src : memref<32x16xf32, #pto.address_space>) + outs(%tile_src : !pto.tile_buf) + %view = pto.subview %tile_x[%c0, %c16] sizes [32, 16] valid [%c32, %c16] : + !pto.tile_buf -> + !pto.tile_buf + pto.tmov ins(%tile_src : + !pto.tile_buf) + outs(%view : !pto.tile_buf) + pto.tstore ins(%tile_x : + !pto.tile_buf) + outs(%out : memref<32x32xf32, #pto.address_space>) + return + } +} + +// CHECK-LABEL: AICORE void tile_assemble_level3_subview_tmov_emitc +// CHECK: set_flag(PIPE_MTE2, PIPE_V, EVENT_ID0); +// CHECK: Tile&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-arch=a5 --pto-level=level3 %s -o - 2>&1 | FileCheck %s --check-prefix=EMITC +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --pto-backend=vpto --emit-vpto %s -o - 2>&1 | FileCheck %s --check-prefix=VPTO + +module attributes {pto.kernel_kind = #pto.kernel_kind} { + func.func @tile_buf_addr_tile_native_arg( + %tile: !pto.tile_buf) { + %ptr = pto.tile_buf_addr %tile + : !pto.tile_buf -> !pto.ptr + func.call @consume(%ptr) : (!pto.ptr) -> () + return + } + + func.func private @consume(!pto.ptr) +} + +// NATIVE: IR Dump After PTOViewToMemref +// NATIVE-LABEL: func.func @tile_buf_addr_tile_native_arg( +// NATIVE-SAME: %[[TILE:.*]]: !pto.tile_buf +// NATIVE: %[[PTR:.*]] = pto.tile_buf_addr %[[TILE]] : !pto.tile_buf -> !pto.ptr +// NATIVE: call @consume(%[[PTR]]) : (!pto.ptr) -> () +// NATIVE-NOT: memref.reinterpret_cast +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: AICORE void tile_buf_addr_tile_native_arg( +// EMITC: __ubuf__ float* {{.*}} = {{.*}}.data(); +// EMITC: consume( + +// VPTO-LABEL: func.func @tile_buf_addr_tile_native_arg( +// VPTO-SAME: !pto.tile_buf +// VPTO: pto.tile_buf_addr %{{.*}} : !pto.tile_buf -> !pto.ptr diff --git a/test/lit/pto/tile_compact_mode_emitc.pto b/test/lit/pto/tile_compact_mode_emitc.pto index f03d81b79f..7c827be7e4 100644 --- a/test/lit/pto/tile_compact_mode_emitc.pto +++ b/test/lit/pto/tile_compact_mode_emitc.pto @@ -25,9 +25,9 @@ module { } } -// A3-DAG: memref.alloc() : memref<1x16xf16, strided<[16, 1]>, #pto.address_space> -// A3-DAG: memref.alloc() : memref<1x16xf16, strided<[16, 1]>, #pto.address_space> -// A3-DAG: memref.alloc() : memref<1x16xf16, strided<[17, 1]>, #pto.address_space> +// A3-DAG: pto.alloc_tile : !pto.tile_buf +// A3-DAG: pto.alloc_tile : !pto.tile_buf +// A3-DAG: pto.alloc_tile : !pto.tile_buf // A3: Tile [[DEFAULT:[_A-Za-z][_A-Za-z0-9]*]]; // A3: Tile [[COMPACT:[_A-Za-z][_A-Za-z0-9]*]]; // A3: Tile [[ROWP1:[_A-Za-z][_A-Za-z0-9]*]]; diff --git a/test/lit/pto/tile_scalar_access_tile_native.pto b/test/lit/pto/tile_scalar_access_tile_native.pto new file mode 100644 index 0000000000..d201f2c382 --- /dev/null +++ b/test/lit/pto/tile_scalar_access_tile_native.pto @@ -0,0 +1,46 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tsetval_arg( + %dst: !pto.tile_buf, + %offset: index, + %val: f32) { + pto.tsetval ins(%offset, %val : index, f32) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tgetval_arg( + %src: !pto.tile_buf, + %offset: index) -> f32 { + %val = pto.tgetval ins(%src, %offset : !pto.tile_buf, index) + outs : f32 + return %val : f32 + } +} + +// NATIVE-LABEL: func.func private @tsetval_arg( +// NATIVE: pto.tsetval ins(%arg1, %arg2 : index, f32) outs(%arg0 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tgetval_arg( +// NATIVE-SAME: -> f32 +// NATIVE: %0 = pto.tgetval ins(%arg0, %arg1 : !pto.tile_buf, index) outs : f32 +// NATIVE: return %0 : f32 +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tsetval_arg( +// EMITC: .SetValue( +// EMITC-LABEL: tgetval_arg( +// EMITC: .GetValue( diff --git a/test/lit/pto/tinsert_level3_tile_native_sync.pto b/test/lit/pto/tinsert_level3_tile_native_sync.pto new file mode 100644 index 0000000000..255c4ab7e0 --- /dev/null +++ b/test/lit/pto/tinsert_level3_tile_native_sync.pto @@ -0,0 +1,38 @@ +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-insert-sync --emit-pto-ir %s | FileCheck %s + +module attributes {"pto.target_arch" = "a5"} { + func.func @tinsert_level3_tile_native_sync( + %x: memref<32x32xf32, #pto.address_space>, + %src: memref<32x16xf32, #pto.address_space>, + %out: memref<32x32xf32, #pto.address_space>) { + %c0 = arith.constant 0 : index + %addr0 = arith.constant 0 : i64 + %addr1 = arith.constant 4096 : i64 + + %tile_x = pto.alloc_tile addr = %addr0 : + !pto.tile_buf + %tile_src = pto.alloc_tile addr = %addr1 : + !pto.tile_buf + + pto.tload ins(%x : memref<32x32xf32, #pto.address_space>) + outs(%tile_x : !pto.tile_buf) + pto.tload ins(%src : memref<32x16xf32, #pto.address_space>) + outs(%tile_src : !pto.tile_buf) + pto.tinsert ins(%tile_src, %c0, %c0 : + !pto.tile_buf, index, index) + outs(%tile_x : !pto.tile_buf) + pto.tstore ins(%tile_x : + !pto.tile_buf) + outs(%out : memref<32x32xf32, #pto.address_space>) + return + } +} + +// CHECK-LABEL: func.func @tinsert_level3_tile_native_sync +// CHECK: pto.tload +// CHECK: pto.tload +// CHECK: pto.set_flag[, , ] +// CHECK: pto.wait_flag[, , ] +// CHECK: pto.tinsert +// CHECK: pto.set_flag[, , ] +// CHECK: pto.wait_flag[, , ] diff --git a/test/lit/pto/tload_tprefetch_low_precision_a5_valid.pto b/test/lit/pto/tload_tprefetch_low_precision_a5_valid.pto index affb448722..0b5c48df87 100644 --- a/test/lit/pto/tload_tprefetch_low_precision_a5_valid.pto +++ b/test/lit/pto/tload_tprefetch_low_precision_a5_valid.pto @@ -24,6 +24,7 @@ module { } // CHECK: func.func @tload_tprefetch_low_precision_a5_valid(%arg0: memref<16x16xf8E4M3FN>, %arg1: memref<16x16x!pto.hif8>) -// CHECK: pto.declare_tile_memref -> memref<16x16x!pto.hif8 +// CHECK: pto.declare_tile -> !pto.tile_buf +// CHECK-NOT: pto.declare_tile_memref // CHECK: pto.tload ins(%arg0 : memref<16x16xf8E4M3FN>) outs( // CHECK: pto.tprefetch ins(%arg1 : memref<16x16x!pto.hif8>) outs( diff --git a/test/lit/pto/tlrelu_tile_native.pto b/test/lit/pto/tlrelu_tile_native.pto new file mode 100644 index 0000000000..9324743b06 --- /dev/null +++ b/test/lit/pto/tlrelu_tile_native.pto @@ -0,0 +1,35 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tlrelu_arg( + %src: !pto.tile_buf, + %slope: f32, + %dst: !pto.tile_buf) { + pto.tlrelu ins(%src, %slope : !pto.tile_buf, f32) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tlrelu_arg( +// NATIVE-SAME: %arg0: !pto.tile_buf +// NATIVE-SAME: %arg1: f32 +// NATIVE-SAME: %arg2: !pto.tile_buf +// NATIVE: pto.tlrelu ins(%arg0, %arg1 : !pto.tile_buf, f32) outs(%arg2 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tlrelu_arg( +// EMITC: TLRELU( diff --git a/test/lit/pto/tmov_fp_tile_native.pto b/test/lit/pto/tmov_fp_tile_native.pto new file mode 100644 index 0000000000..4e2822406c --- /dev/null +++ b/test/lit/pto/tmov_fp_tile_native.pto @@ -0,0 +1,32 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tmov_fp_arg( + %src: !pto.tile_buf, + %fp: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tmov.fp ins(%src, %fp : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tmov_fp_arg( +// NATIVE: pto.tmov.fp ins(%arg0, %arg1 +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tmov_fp_arg( +// EMITC: TMOV_FP< diff --git a/test/lit/pto/tpow_emitc.pto b/test/lit/pto/tpow_emitc.pto index 186ce1c57c..d252177805 100644 --- a/test/lit/pto/tpow_emitc.pto +++ b/test/lit/pto/tpow_emitc.pto @@ -19,6 +19,6 @@ module { } } -// A3: pto.tpow ins([[BASE:%[0-9]+]], [[EXP:%[0-9]+]], [[TMP:%[0-9]+]] : memref<1x16xf32 -// A3: outs([[DST:%[0-9]+]] : memref<1x16xf32 +// A3: pto.tpow ins([[BASE:%[0-9]+]], [[EXP:%[0-9]+]], [[TMP:%[0-9]+]] : !pto.tile_buf, diff --git a/test/lit/pto/trans_tile_native.pto b/test/lit/pto/trans_tile_native.pto new file mode 100644 index 0000000000..af88a74480 --- /dev/null +++ b/test/lit/pto/trans_tile_native.pto @@ -0,0 +1,33 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @ttrans_arg( + %src: !pto.tile_buf, + %tmp: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.ttrans ins(%src, %tmp : !pto.tile_buf, !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @ttrans_arg +// NATIVE-SAME: (%[[SRC:[^:]+]]: !pto.tile_buf, %[[TMP:[^:]+]]: !pto.tile_buf, %[[DST:[^:]+]]: !pto.tile_buf) +// NATIVE: pto.ttrans ins(%[[SRC]], %[[TMP]] : !pto.tile_buf, !pto.tile_buf) outs(%[[DST]] : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: ttrans_arg( +// EMITC: TTRANS( diff --git a/test/lit/pto/transcendental_unary_tile_native.pto b/test/lit/pto/transcendental_unary_tile_native.pto new file mode 100644 index 0000000000..37e90e4527 --- /dev/null +++ b/test/lit/pto/transcendental_unary_tile_native.pto @@ -0,0 +1,47 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @texp_arg( + %src: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.texp ins(%src : !pto.tile_buf) + outs(%dst : !pto.tile_buf) + {precisionType = #pto} + return + } + + func.func private @tlog_arg( + %src: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tlog ins(%src : !pto.tile_buf) + outs(%dst : !pto.tile_buf) + {precisionType = #pto} + return + } +} + +// NATIVE-LABEL: func.func private @texp_arg( +// NATIVE-SAME: %arg0: !pto.tile_buf +// NATIVE-SAME: %arg1: !pto.tile_buf +// NATIVE: pto.texp ins(%arg0 : !pto.tile_buf) outs(%arg1 : !pto.tile_buf) {precisionType = #pto} +// NATIVE-LABEL: func.func private @tlog_arg( +// NATIVE: pto.tlog ins(%arg0 : !pto.tile_buf) outs(%arg1 : !pto.tile_buf) {precisionType = #pto} +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: texp_arg( +// EMITC: TEXP( +// EMITC-LABEL: tlog_arg( +// EMITC: TLOG( diff --git a/test/lit/pto/trem_emitc.pto b/test/lit/pto/trem_emitc.pto index a5aa39f7ee..36736b0102 100644 --- a/test/lit/pto/trem_emitc.pto +++ b/test/lit/pto/trem_emitc.pto @@ -11,8 +11,8 @@ module { } } -// A3: pto.trem ins([[SRC0:%[0-9]+]], [[SRC1:%[0-9]+]], [[TMP:%[0-9]+]] : memref<1x16xf32 -// A3-SAME: memref<1x16xf32 -// A3-SAME: memref<2x16xf32 -// A3: outs([[DST:%[0-9]+]] : memref<1x16xf32 +// A3: pto.trem ins([[SRC0:%[0-9]+]], [[SRC1:%[0-9]+]], [[TMP:%[0-9]+]] : !pto.tile_buf{{.*}}pto.view_semantics = "treshape" +// CHECK: pto.alloc_tile addr = {{.*}} valid_row = [[PART]] valid_col = [[FULL]] : !pto.tile_buf +// CHECK: pto.treshape {{.*}} : !pto.tile_buf -> !pto.tile_buf diff --git a/test/lit/pto/treshape_tile_native_arg.pto b/test/lit/pto/treshape_tile_native_arg.pto new file mode 100644 index 0000000000..bbf649ad92 --- /dev/null +++ b/test/lit/pto/treshape_tile_native_arg.pto @@ -0,0 +1,41 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o - 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE + +module attributes {pto.target_arch = "a5"} { + func.func private @reshape_arg( + %src: !pto.tile_buf) + -> !pto.tile_buf { + %result = pto.treshape %src + : !pto.tile_buf + -> !pto.tile_buf + return %result : !pto.tile_buf + } +} + +// NATIVE-LABEL: func.func private @reshape_arg( +// NATIVE-SAME: !pto.tile_buf +// NATIVE: %[[RESULT:.*]] = pto.treshape %arg0 +// NATIVE-SAME: !pto.tile_buf +// NATIVE-SAME: -> !pto.tile_buf +// NATIVE: return %[[RESULT]] : !pto.tile_buf +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile diff --git a/test/lit/pto/trsqrt_emitc.pto b/test/lit/pto/trsqrt_emitc.pto index b9b1b55bf3..7fc664730b 100644 --- a/test/lit/pto/trsqrt_emitc.pto +++ b/test/lit/pto/trsqrt_emitc.pto @@ -13,9 +13,9 @@ module { } } -// A3: pto.trsqrt ins([[SRC0:%[0-9]+]] : memref<1x16xf16 -// A3: outs([[DST0:%[0-9]+]] : memref<1x16xf16 -// A3: pto.trsqrt ins([[SRC1:%[0-9]+]], [[TMP:%[0-9]+]] : memref<1x16xf16 -// A3: outs([[DST1:%[0-9]+]] : memref<1x16xf16 +// A3: pto.trsqrt ins([[SRC0:%[0-9]+]] : !pto.tile_buf, %arg1: memref<16x16x!pto.hif8>, %arg2: i64) -// CHECK: pto.bind_tile +// CHECK: %[[VEC:.*]] = pto.declare_tile -> !pto.tile_buf +// CHECK: %[[ACC:.*]] = pto.declare_tile -> !pto.tile_buf +// CHECK-NOT: pto.declare_tile_memref +// CHECK-NOT: pto.bind_tile // CHECK: pto.section.vector { -// CHECK: pto.tstore ins( +// CHECK: pto.tstore ins(%[[VEC]] : !pto.tile_buf) // CHECK: outs(%arg0 : memref<16x16xf8E4M3FN>) // CHECK: } // CHECK: pto.section.cube { -// CHECK: pto.tstore ins( +// CHECK: pto.tstore ins(%[[ACC]] : !pto.tile_buf, %arg2 : i64) // CHECK: outs(%arg1 : memref<16x16x!pto.hif8>) // CHECK: } diff --git a/test/lit/pto/unary_compute_tile_native.pto b/test/lit/pto/unary_compute_tile_native.pto new file mode 100644 index 0000000000..d73a152dde --- /dev/null +++ b/test/lit/pto/unary_compute_tile_native.pto @@ -0,0 +1,57 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s \ +// RUN: | FileCheck %s --check-prefix=EMITC + +module { + func.func private @tabs_arg( + %src: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tabs ins(%src : !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tneg_arg( + %src: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tneg ins(%src : !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } + + func.func private @tnot_arg( + %src: !pto.tile_buf, + %dst: !pto.tile_buf) { + pto.tnot ins(%src : !pto.tile_buf) + outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: func.func private @tabs_arg( +// NATIVE-SAME: %arg0: !pto.tile_buf +// NATIVE-SAME: %arg1: !pto.tile_buf +// NATIVE: pto.tabs ins(%arg0 : !pto.tile_buf) outs(%arg1 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tneg_arg( +// NATIVE: pto.tneg ins(%arg0 : !pto.tile_buf) outs(%arg1 : !pto.tile_buf) +// NATIVE-LABEL: func.func private @tnot_arg( +// NATIVE: pto.tnot ins(%arg0 : !pto.tile_buf) outs(%arg1 : !pto.tile_buf) +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: tabs_arg( +// EMITC: TABS( +// EMITC-LABEL: tneg_arg( +// EMITC: TNEG( +// EMITC-LABEL: tnot_arg( +// EMITC: TNOT( diff --git a/test/lit/pto/validshape_tile_native.pto b/test/lit/pto/validshape_tile_native.pto new file mode 100644 index 0000000000..17b4609ddf --- /dev/null +++ b/test/lit/pto/validshape_tile_native.pto @@ -0,0 +1,43 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 \ +// RUN: --mlir-print-ir-after=pto-view-to-memref %s -o - 2>&1 \ +// RUN: | FileCheck %s --check-prefix=NATIVE + +module attributes {pto.target_arch = "a5"} { + func.func private @get_validshape_arg( + %src: !pto.tile_buf) { + %c0_i64 = arith.constant 0 : i64 + %c16 = arith.constant 16 : index + %local = pto.alloc_tile addr = %c0_i64 valid_row = %c16 valid_col = %c16 + : !pto.tile_buf + %row, %col = pto.get_validshape %src + : !pto.tile_buf + pto.set_validshape %local, %row, %col + : !pto.tile_buf + return + } +} + +// NATIVE-LABEL: func.func private @get_validshape_arg( +// NATIVE-SAME: !pto.tile_buf) +// NATIVE: %[[ROW:.*]], %[[COL:.*]] = pto.get_validshape %arg0 +// NATIVE-SAME: !pto.tile_buf +// NATIVE: pto.set_validshape {{.*}}, %[[ROW]], %[[COL]] +// NATIVE: return +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile diff --git a/test/lit/pto/vector_specialty_tile_native.pto b/test/lit/pto/vector_specialty_tile_native.pto new file mode 100644 index 0000000000..601c6ce412 --- /dev/null +++ b/test/lit/pto/vector_specialty_tile_native.pto @@ -0,0 +1,49 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @thistogram_arg(%src: !pto.tile_buf, %idx: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.thistogram ins(%src, %idx : !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @trandom_arg(%dst: !pto.tile_buf, %k0: i32, %k1: i32, %c0: i32, %c1: i32, %c2: i32, %c3: i32) { + pto.trandom ins(%k0, %k1, %c0, %c1, %c2, %c3 {rounds = 7 : i32} : i32, i32, i32, i32, i32, i32) outs(%dst : !pto.tile_buf) + return + } + func.func private @tscatter_arg(%src: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.tscatter ins(%src, {maskPattern = #pto.mask_pattern} : !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @ttri_arg(%dst: !pto.tile_buf, %diag: i32) { + pto.ttri ins(%diag {upperOrLower = 1 : i32} : i32) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @thistogram_arg +// NATIVE: pto.thistogram +// NATIVE-LABEL: @trandom_arg +// NATIVE: pto.trandom +// NATIVE-LABEL: @tscatter_arg +// NATIVE: pto.tscatter +// NATIVE-LABEL: @ttri_arg +// NATIVE: pto.ttri +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: thistogram_arg( +// EMITC: THISTOGRAM< +// EMITC-LABEL: trandom_arg( +// EMITC: PTOAS__TRANDOM<7>( +// EMITC-LABEL: tscatter_arg( +// EMITC: TSCATTER( +// EMITC-LABEL: ttri_arg( +// EMITC: TTRI< diff --git a/test/lit/pto/xor_tile_native.pto b/test/lit/pto/xor_tile_native.pto new file mode 100644 index 0000000000..c728792ef4 --- /dev/null +++ b/test/lit/pto/xor_tile_native.pto @@ -0,0 +1,33 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// RUN: ptoas --pto-level=level3 --pto-arch=a5 --mlir-print-ir-after=pto-view-to-memref %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=NATIVE +// RUN: ptoas --pto-level=level3 --pto-arch=a3 --enable-insert-sync %s | FileCheck %s --check-prefix=EMITC + +module { + func.func private @txor_arg(%a: !pto.tile_buf, %b: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf) { + pto.txor ins(%a, %b, %tmp : !pto.tile_buf, !pto.tile_buf, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } + func.func private @txors_arg(%src: !pto.tile_buf, %tmp: !pto.tile_buf, %dst: !pto.tile_buf, %scalar: i16) { + pto.txors ins(%src, %scalar, %tmp : !pto.tile_buf, i16, !pto.tile_buf) outs(%dst : !pto.tile_buf) + return + } +} + +// NATIVE-LABEL: @txor_arg +// NATIVE: pto.txor +// NATIVE-LABEL: @txors_arg +// NATIVE: pto.txors +// NATIVE-NOT: memref< +// NATIVE-NOT: pto.bind_tile + +// EMITC-LABEL: txor_arg( +// EMITC: TXOR( +// EMITC-LABEL: txors_arg( +// EMITC: TXORS( diff --git a/test/lit/tile_fusion/op_fusion_adapter_placement_level2_tadd.pto b/test/lit/tile_fusion/op_fusion_adapter_placement_level2_tadd.pto index 5c2f81d4ee..1d493ea705 100644 --- a/test/lit/tile_fusion/op_fusion_adapter_placement_level2_tadd.pto +++ b/test/lit/tile_fusion/op_fusion_adapter_placement_level2_tadd.pto @@ -53,12 +53,9 @@ module { // SEAM-LABEL: func.func @fusion_adapter_placement_level2_tadd( // SEAM-NOT: pto.fusion_region -// SEAM: memref.alloc() -// SEAM: pto.bind_tile -// SEAM: memref.alloc() -// SEAM: pto.bind_tile -// SEAM: memref.alloc() -// SEAM: pto.bind_tile +// SEAM: pto.alloc_tile +// SEAM: pto.alloc_tile +// SEAM: pto.alloc_tile // SEAM: pto.tadd{{.*}}{pto.fusion.group_id = 0 : i64, pto.fusion.order = 0 : i64{{.*}}} // SEAM: pto.tadd{{.*}}{pto.fusion.group_id = 0 : i64, pto.fusion.order = 1 : i64{{.*}}} // SEAM: pto.tadd{{.*}}{pto.fusion.group_id = 0 : i64, pto.fusion.order = 2 : i64{{.*}}} diff --git a/test/lit/tile_fusion/op_fusion_adapter_placement_level3_tadd.pto b/test/lit/tile_fusion/op_fusion_adapter_placement_level3_tadd.pto index 09d319e076..9c948530f4 100644 --- a/test/lit/tile_fusion/op_fusion_adapter_placement_level3_tadd.pto +++ b/test/lit/tile_fusion/op_fusion_adapter_placement_level3_tadd.pto @@ -63,13 +63,13 @@ module { // SEAM: // -----// IR Dump After PTOViewToMemref (pto-view-to-memref) //----- // // SEAM-LABEL: func.func @fusion_adapter_placement_level3_tadd( // SEAM-NOT: pto.fusion_region -// SEAM: pto.pointer_cast(%c0_i64) -// SEAM: pto.pointer_cast(%c4096_i64) -// SEAM: pto.pointer_cast(%c8192_i64) -// SEAM: pto.pointer_cast(%c12288_i64) -// SEAM: pto.pointer_cast(%c16384_i64) -// SEAM: pto.pointer_cast(%c20480_i64) -// SEAM: pto.pointer_cast(%c24576_i64) +// SEAM: pto.alloc_tile addr = %c0_i64 +// SEAM: pto.alloc_tile addr = %c4096_i64 +// SEAM: pto.alloc_tile addr = %c8192_i64 +// SEAM: pto.alloc_tile addr = %c12288_i64 +// SEAM: pto.alloc_tile addr = %c16384_i64 +// SEAM: pto.alloc_tile addr = %c20480_i64 +// SEAM: pto.alloc_tile addr = %c24576_i64 // SEAM: pto.tadd{{.*}}{pto.fusion.group_id = 0 : i64, pto.fusion.order = 0 : i64{{.*}}} // SEAM: pto.tadd{{.*}}{pto.fusion.group_id = 0 : i64, pto.fusion.order = 1 : i64{{.*}}} // SEAM: pto.tadd{{.*}}{pto.fusion.group_id = 0 : i64, pto.fusion.order = 2 : i64{{.*}}} diff --git a/test/lit/tile_fusion/op_fusion_region_pipeline_level2.pto b/test/lit/tile_fusion/op_fusion_region_pipeline_level2.pto index 47ca3e773c..dafa62885a 100644 --- a/test/lit/tile_fusion/op_fusion_region_pipeline_level2.pto +++ b/test/lit/tile_fusion/op_fusion_region_pipeline_level2.pto @@ -70,7 +70,7 @@ module { // LEVEL2-PLAIN-NEXT: pto.trowexpandmul // LEVEL2-PLAIN-NEXT: pto.tadd // LEVEL2-PLAIN-NEXT: pto.yield( -// LEVEL2-PLAIN: } {pto.fusion.group_id = 0 : i64} : memref<32x32xf32 +// LEVEL2-PLAIN: } {pto.fusion.group_id = 0 : i64} : !pto.tile_buf // LEVEL2-PLAIN: pto.tadd ins( // LEVEL2-PLAIN-NEXT: pto.tmul ins( // LEVEL2-PLAIN: pto.tstore ins( @@ -83,7 +83,7 @@ module { // LEVEL2-SYNC-NEXT: pto.trowexpandmul // LEVEL2-SYNC-NEXT: pto.tadd // LEVEL2-SYNC: pto.yield( -// LEVEL2-SYNC: } {pto.fusion.group_id = 0 : i64} : memref<32x32xf32 +// LEVEL2-SYNC: } {pto.fusion.group_id = 0 : i64} : !pto.tile_buf // LEVEL2-SYNC: pto.tadd ins( // LEVEL2-SYNC-NEXT: pto.tmul ins( // LEVEL2-SYNC: pto.tstore ins( diff --git a/test/lit/vpto/mte_ub_gm_l2_cache_ctl.pto b/test/lit/vpto/mte_ub_gm_l2_cache_ctl.pto index f91b6a0389..a0381aa7ae 100644 --- a/test/lit/vpto/mte_ub_gm_l2_cache_ctl.pto +++ b/test/lit/vpto/mte_ub_gm_l2_cache_ctl.pto @@ -6,8 +6,10 @@ // INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. // See LICENSE in the root of the software repository for the full text of the License. -// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-before=vpto-expand-wrapper-ops %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=ROUNDTRIP -// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-after=vpto-expand-wrapper-ops %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=EXPAND +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-before=vpto-expand-wrapper-ops %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=ROUNDTRIP-CTL +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-before=vpto-expand-wrapper-ops %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=ROUNDTRIP-DEFAULT +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-after=vpto-expand-wrapper-ops %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=EXPAND-CTL +// RUN: ptoas --pto-arch=a5 --pto-backend=vpto --emit-vpto --mlir-print-ir-after=vpto-expand-wrapper-ops %s -o /dev/null 2>&1 | FileCheck %s --check-prefix=EXPAND-DEFAULT module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind} { func.func @mte_ub_gm_l2_cache_ctl(%out: !pto.ptr) attributes {pto.aicore} { @@ -36,19 +38,19 @@ module attributes {pto.target_arch = "a5", pto.kernel_kind = #pto.kernel_kind, !pto.ptr, i64, i64, i64, i64, i64{{$}} -// ROUNDTRIP-LABEL: func.func @mte_ub_gm_l2_cache_ctl_default -// ROUNDTRIP: pto.mte_ub_gm %{{[^,]+}}, %arg0, %{{[^ ]+}} nburst(%{{[^,]+}}, %{{[^,]+}}, %{{[^)]+}}) -// ROUNDTRIP-SAME: : !pto.ptr, !pto.ptr, i64, i64, i64, i64{{$}} +// ROUNDTRIP-CTL-LABEL: func.func @mte_ub_gm_l2_cache_ctl( +// ROUNDTRIP-CTL: %[[L2:.*]] = arith.constant 5 : i64 +// ROUNDTRIP-CTL: pto.mte_ub_gm %{{[^,]+}}, %arg0, %{{[^ ]+}} nburst(%{{[^,]+}}, %{{[^,]+}}, %{{[^)]+}}) l2_cache_ctl(%[[L2]]) +// ROUNDTRIP-CTL-SAME: : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64{{$}} +// ROUNDTRIP-DEFAULT-LABEL: func.func @mte_ub_gm_l2_cache_ctl_default( +// ROUNDTRIP-DEFAULT: pto.mte_ub_gm %{{[^,]+}}, %arg0, %{{[^ ]+}} nburst(%{{[^,]+}}, %{{[^,]+}}, %{{[^)]+}}) +// ROUNDTRIP-DEFAULT-SAME: : !pto.ptr, !pto.ptr, i64, i64, i64, i64{{$}} -// EXPAND-LABEL: func.func @mte_ub_gm_l2_cache_ctl -// EXPAND: %[[L2:.*]] = arith.constant 5 : i64 -// EXPAND: pto.copy_ubuf_to_gm %{{[^,]+}}, %arg0, %{{[^,]+}}, %{{[^,]+}}, %{{[^,]+}}, %[[L2]], %{{[^,]+}}, %{{[^:]+}} -// EXPAND-SAME: : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64, i64 -// EXPAND-LABEL: func.func @mte_ub_gm_l2_cache_ctl_default -// EXPAND: %[[ZERO:.*]] = arith.constant 0 : i64 -// EXPAND: pto.copy_ubuf_to_gm %{{[^,]+}}, %arg0, %[[ZERO]], %{{[^,]+}}, %{{[^,]+}}, %[[ZERO]], %{{[^,]+}}, %{{[^:]+}} -// EXPAND-SAME: : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64, i64 +// EXPAND-CTL-LABEL: func.func @mte_ub_gm_l2_cache_ctl( +// EXPAND-CTL: %[[L2:.*]] = arith.constant 5 : i64 +// EXPAND-CTL: pto.copy_ubuf_to_gm %{{[^,]+}}, %arg0, %{{[^,]+}}, %{{[^,]+}}, %{{[^,]+}}, %[[L2]], %{{[^,]+}}, %{{[^:]+}} +// EXPAND-CTL-SAME: : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64, i64 +// EXPAND-DEFAULT-LABEL: func.func @mte_ub_gm_l2_cache_ctl_default( +// EXPAND-DEFAULT: %[[ZERO:.*]] = arith.constant 0 : i64 +// EXPAND-DEFAULT: pto.copy_ubuf_to_gm %{{[^,]+}}, %arg0, %[[ZERO]], %{{[^,]+}}, %{{[^,]+}}, %[[ZERO]], %{{[^,]+}}, %{{[^:]+}} +// EXPAND-DEFAULT-SAME: : !pto.ptr, !pto.ptr, i64, i64, i64, i64, i64, i64 diff --git a/test/lit/vpto/normalize_uncovered_tile_sections_vector.pto b/test/lit/vpto/normalize_uncovered_tile_sections_vector.pto index 665d08b682..728c2a2e5c 100644 --- a/test/lit/vpto/normalize_uncovered_tile_sections_vector.pto +++ b/test/lit/vpto/normalize_uncovered_tile_sections_vector.pto @@ -39,7 +39,7 @@ module attributes {pto.target_arch = "a5"} { // CHECK: module attributes // CHECK-SAME: pto.backend = "vpto", pto.kernel_kind = #pto.kernel_kind, pto.target_arch = "a5" // CHECK: func.func @auto_vector_segment -// CHECK: pto.bind_tile +// CHECK: pto.alloc_tile addr = // CHECK: pto.tload // CHECK: pto.tadds // CHECK: pto.tstore diff --git a/tools/ptoas/ptoas.cpp b/tools/ptoas/ptoas.cpp index 2c62a9005c..17fcfbb0a8 100644 --- a/tools/ptoas/ptoas.cpp +++ b/tools/ptoas/ptoas.cpp @@ -8,6 +8,7 @@ #include "ptoas.h" #include "PTO/IR/PTO.h" +#include "PTO/IR/PTOMultiBuffer.h" #include "PTO/Transforms/VPTOLLVMEmitter.h" #include "PTO/Transforms/Passes.h" #include "PTO/Transforms/BufferizableOpInterfaceImpl.h" @@ -326,11 +327,18 @@ static llvm::cl::opt enableInsertSync("enable-insert-sync", static llvm::cl::opt planMemoryOrderBySize( "plan-memory-order-by-size", - llvm::cl::desc("PlanMemory: allocate buffers largest-first " - "(first-fit-decreasing) instead of the default DMA-first " - "order"), + llvm::cl::desc("Plan larger local buffers first inside one AddressSpace " + "before applying the basic SPEC_LEVEL_0 reuse strategy. " + "Defaults to true when --plan-memory-impl=modern is " + "explicitly selected"), llvm::cl::init(false)); +static llvm::cl::opt planMemoryImpl( + "plan-memory-impl", + llvm::cl::desc("Select local memory planner implementation: legacy or " + "modern"), + llvm::cl::init("legacy")); + static llvm::cl::opt enableBufidSync( "enable-bufid_sync", llvm::cl::desc("Enable A5 buffer-id synchronization insertion pass"), @@ -628,6 +636,97 @@ static bool parseBuildLevel(llvm::StringRef levelStr, PTOBuildLevel &out) { return false; } +struct ReserveBufferMemSpec { + uint64_t capacityBytes = 0; + uint64_t alignmentBytes = 1; +}; + +static ReserveBufferMemSpec getReserveBufferMemSpec(PTOArch arch, + AddressSpace space) { + switch (space) { + case AddressSpace::VEC: + return {arch == PTOArch::A5 ? 253952ull : 196608ull, 256}; + case AddressSpace::MAT: + return {524288ull, 256}; + case AddressSpace::LEFT: + case AddressSpace::RIGHT: + case AddressSpace::ACC: + case AddressSpace::BIAS: + case AddressSpace::SCALING: + case AddressSpace::GM: + case AddressSpace::Zero: + break; + } + return {}; +} + +static LogicalResult validateReserveBufferBase(pto::ReserveBufferOp op, + PTOArch arch) { + auto baseAttr = op.getBaseAttr(); + if (!baseAttr) + return op.emitError("expects explicit 'base'"); + + int64_t signedBase = baseAttr.getInt(); + if (signedBase < 0) + return op.emitError("expects 'base' to be non-negative when present"); + + ReserveBufferMemSpec spec = + getReserveBufferMemSpec(arch, op.getLocation().getAddressSpace()); + uint64_t base = static_cast(signedBase); + if (base % spec.alignmentBytes != 0) { + return op.emitError("expects 'base' to be aligned to ") + << spec.alignmentBytes << " bytes for " + << stringifyEnum(op.getLocation().getAddressSpace()); + } + + uint64_t size = static_cast(op.getSize()); + if (base > spec.capacityBytes || size > spec.capacityBytes - base) { + return op.emitError("reserved range exceeds ") + << stringifyEnum(op.getLocation().getAddressSpace()) + << " capacity: base " << base << " + size " << size + << " > " << spec.capacityBytes << " bytes"; + } + + return success(); +} + +static bool validateReserveBufferLevelRules(ModuleOp module, + PTOBuildLevel level) { + bool failed = false; + PTOArch arch = getTargetArch(module); + module.walk([&](pto::ReserveBufferOp op) { + if (level != PTOBuildLevel::Level3) { + if (op.getAutoAlloc()) { + if (op.getBaseAttr()) { + op.emitError("unexpected 'base' on auto reserve_buffer: " + "level1/level2 assign it in pto-plan-memory"); + failed = true; + } + return; + } + + if (op.getBaseAttr()) + (void)validateReserveBufferBase(op, arch); + op.emitError("pto.reserve_buffer with explicit 'base' (auto = false) is " + "not supported when --pto-level=level1 or level2; use " + "--pto-level=level3 or set auto = true"); + failed = true; + return; + } + + if (op.getAutoAlloc() || !op.getBaseAttr()) { + op.emitError("pto.reserve_buffer requires 'auto = false' and explicit " + "'base' when --pto-level=level3"); + failed = true; + return; + } + + if (mlir::failed(validateReserveBufferBase(op, arch))) + failed = true; + }); + return !failed; +} + static constexpr llvm::StringLiteral kAutoSyncTailPolicyBarrierAll = "barrier_all"; static constexpr llvm::StringLiteral kAutoSyncTailPolicyMte3ToSEvent0 = @@ -2867,6 +2966,17 @@ int mlir::pto::compilePTOASModule( return 1; } + bool hasUserPlannedMultiAddrs = false; + module->walk([&](pto::AllocMultiTileOp op) { + if (!op->hasAttr(pto::kPtoMultiBufferAddrsAttrName)) + return; + op.emitError() << "attribute '" << pto::kPtoMultiBufferAddrsAttrName + << "' is reserved for pto-plan-memory"; + hasUserPlannedMultiAddrs = true; + }); + if (hasUserPlannedMultiAddrs) + return 1; + if (effectiveLevel == PTOBuildLevel::Level3) { // In level3 the caller owns local memory and PTOPlanMemory is skipped, so // every allocation must carry an explicit physical address. For @@ -2909,6 +3019,9 @@ int mlir::pto::compilePTOASModule( return 1; } + if (!validateReserveBufferLevelRules(*module, effectiveLevel)) + return 1; + { PassManager preBackendPM(module->getContext()); preBackendPM.enableVerifier(); @@ -3002,12 +3115,23 @@ int mlir::pto::compilePTOASModule( pto::createPTORematerializeFixpipeVectorQuantPass()); if (effectiveLevel != PTOBuildLevel::Level3) { - PlanMemoryOptions planMemoryOption; - planMemoryOption.memMode = MemPlanMode::LOCAL_MEM_PLAN; - planMemoryOption.enableGlobalReuse = false; - planMemoryOption.enablePrintMemoryAllocatedSize = false; - planMemoryOption.orderBySize = planMemoryOrderBySize; - pm.addPass(pto::createPlanMemoryPass(planMemoryOption)); + pto::PlanMemoryOptions planMemoryOptions; + planMemoryOptions.memMode = "local"; + bool effectivePlanMemoryOrderBySize = planMemoryOrderBySize; + if (planMemoryImpl == "modern" && + planMemoryOrderBySize.getNumOccurrences() == 0) { + effectivePlanMemoryOrderBySize = true; + } + planMemoryOptions.orderBySize = effectivePlanMemoryOrderBySize; + if (planMemoryImpl == "legacy") { + pm.addPass(pto::createPlanMemoryPass(planMemoryOptions)); + } else if (planMemoryImpl == "modern") { + pm.addPass(pto::createPlanMemoryModernPass(planMemoryOptions)); + } else { + llvm::errs() << "Error: invalid --plan-memory-impl='" << planMemoryImpl + << "', expected 'legacy' or 'modern'.\n"; + return 1; + } } pm.addPass(pto::createPTOResolveReservedBuffersPass()); pm.addNestedPass(pto::createPTORemoveIdentityTMovPass()); @@ -3039,7 +3163,8 @@ int mlir::pto::compilePTOASModule( // Materialize per-slot single-address `pto.pointer_cast` (constant slot) // or an `arith.select` chain (dynamic slot). The multi-address cast - // produced by PlanMemory survives as the alloc anchor. + // produced by explicit-address lowering or PTOPlanMemory survives as + // the alloc anchor. pm.addPass(pto::createPTOResolveBufferSelectPass()); if (effectiveBackend == PTOBackend::EmitC) pm.addPass(createNarrowUnusedMultiResultProvenancePass());