diff --git a/docs/api/runtime.md b/docs/api/runtime.md index 0505777..7a1cd4c 100644 --- a/docs/api/runtime.md +++ b/docs/api/runtime.md @@ -50,6 +50,9 @@ otherwise. runtime::Error runtime::Stream runtime::Event +runtime::Graph +runtime::GraphExec +runtime::StreamCaptureMode runtime::MemcpyKind runtime::kSuccess runtime::kMemcpyHostToHost @@ -58,6 +61,15 @@ runtime::kMemcpyDeviceToHost runtime::kMemcpyDeviceToDevice ``` +`StreamCaptureMode` supports the same capture scopes used by CUDA-style stream +capture: + +```cpp +runtime::StreamCaptureMode::kStreamCaptureModeGlobal +runtime::StreamCaptureMode::kStreamCaptureModeThreadLocal +runtime::StreamCaptureMode::kStreamCaptureModeRelaxed +``` + ## Device Functions ```cpp @@ -107,6 +119,58 @@ runtime::Error EventElapsedTime(float* ms, runtime::Event start, runtime::Event end); ``` +## Graph Capture and Replay Functions + +When the configured build includes a graph-capable backend, the generated +runtime API also includes graph capture and replay entry points. + +The graph API captures stream work into a reusable `Graph`, instantiates it as a +launchable `GraphExec`, and replays it on a stream: + +```cpp +runtime::Error StreamBeginCapture(runtime::Stream stream, + runtime::StreamCaptureMode mode); +runtime::Error StreamEndCapture(runtime::Stream stream, runtime::Graph* graph); +runtime::Error GraphDestroy(runtime::Graph graph); +runtime::Error GraphInstantiate(runtime::GraphExec* graph_exec, + runtime::Graph graph); +runtime::Error GraphExecDestroy(runtime::GraphExec graph_exec); +runtime::Error GraphLaunch(runtime::GraphExec graph_exec, + runtime::Stream stream); +``` + +Use the same stream for `StreamBeginCapture` and `StreamEndCapture`. Operations +issued to that stream between the two calls become part of the captured graph. +After `GraphInstantiate` succeeds, the executable graph can be launched multiple +times while inputs and outputs remain at the recorded addresses. + +```cpp +namespace rt = infini::rt::runtime; + +rt::Stream stream = nullptr; +rt::Graph graph = nullptr; +rt::GraphExec graph_exec = nullptr; + +rt::StreamCreate(&stream); + +rt::StreamBeginCapture( + stream, rt::StreamCaptureMode::kStreamCaptureModeRelaxed); +rt::MemcpyAsync(dst, src, count, rt::kMemcpyDeviceToDevice, stream); +rt::StreamEndCapture(stream, &graph); + +rt::GraphInstantiate(&graph_exec, graph); +rt::GraphLaunch(graph_exec, stream); +rt::StreamSynchronize(stream); + +rt::GraphExecDestroy(graph_exec); +rt::GraphDestroy(graph); +rt::StreamDestroy(stream); +``` + +Backends that do not support graph capture return a non-success status for the +graph entry points when those functions are present in the configured public +header. See [Backends](../backends.md) for current support. + ## Backend-Specific Runtime Templates Backend runtime specializations are available as implementation-facing headers, diff --git a/docs/backends.md b/docs/backends.md index 5b62dd5..26ad1ff 100644 --- a/docs/backends.md +++ b/docs/backends.md @@ -28,16 +28,16 @@ status on backends that do not support them. Current test expectations are: -| Backend | Async memcpy | Host memory | Async memory | Memory info | Async memset | Events | -| --- | --- | --- | --- | --- | --- | --- | -| CPU | No | Yes | No | Yes | No | Yes | -| NVIDIA | Yes | Yes | Yes | Yes | Yes | Yes | -| Iluvatar | Yes | Yes | Yes | Yes | Yes | Yes | -| MetaX | Yes | Yes | Yes | Yes | Yes | Yes | -| Moore | Yes | Yes | No | Yes | Yes | Yes | -| Hygon | Yes | Yes | Yes | Yes | Yes | Yes | -| Cambricon | Yes | No | No | No | No | No | -| Ascend | Yes | No | No | No | No | No | +| Backend | Async memcpy | Host memory | Async memory | Memory info | Async memset | Events | Graph capture | +| --- | --- | --- | --- | --- | --- | --- | --- | +| CPU | No | Yes | No | Yes | No | Yes | No | +| NVIDIA | Yes | Yes | Yes | Yes | Yes | Yes | Yes | +| Iluvatar | Yes | Yes | Yes | Yes | Yes | Yes | Yes | +| MetaX | Yes | Yes | Yes | Yes | Yes | Yes | No | +| Moore | Yes | Yes | No | Yes | Yes | Yes | No | +| Hygon | Yes | Yes | Yes | Yes | Yes | Yes | No | +| Cambricon | Yes | No | No | No | No | No | No | +| Ascend | Yes | No | No | No | No | No | Yes | Treat a non-`kSuccess` status as the portable way to detect unsupported operations. diff --git a/docs/compatibility.md b/docs/compatibility.md index 2621d74..c700c95 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -18,7 +18,7 @@ The stable user-facing surface is: - `infini::rt::TensorView` - `infini::rt::set_runtime_device_type` - `infini::rt::runtime_device_type` -- `infini::rt::runtime` dispatching functions and constants documented in +- `infini::rt::runtime` types, dispatching functions, and constants documented in [Runtime API](api/runtime.md) ## Implementation-Facing Headers