Skip to content

Adding DirectML fallback support in heuristics in EP - #84

Draft
urpetkov-amd wants to merge 1 commit into
mainfrom
add-heuristic-fallback
Draft

Adding DirectML fallback support in heuristics in EP#84
urpetkov-amd wants to merge 1 commit into
mainfrom
add-heuristic-fallback

Conversation

@urpetkov-amd

Copy link
Copy Markdown
Collaborator

Move AMDGPU automatic DirectML fallback into the execution provider

Summary

This work moves the existing ModelBench-only automatic DirectML fallback policy
into the AMDGPU umbrella execution provider.

The new implementation:

  • Keeps ModelBench in auto mode.
  • Uses model-content metadata instead of model names as the routing identity.
  • Passes the identity through the existing model_arch provider option.
  • Makes the AMDGPU EP responsible for selecting MIGraphX or DirectML/DirectX.
  • Preserves all existing measured per-GPU fallback decisions.
  • Leaves unknown models on the normal architecture default.
  • Keeps explicit profiles authoritative.

This is whole-session backend routing. It is not graph-level, node-level, or
per-operation fallback.

Motivation

The original fallback was implemented specifically in ModelBench:

  1. ModelBench looked up the requested model name in a GPU-specific fallback
    list.
  2. If the name matched, ModelBench changed the requested AMDGPU profile from
    auto to directml.
  3. The ORT session was then created with DirectML already selected.

Why the caller supplies the identity

The AMDGPU umbrella EP chooses and constructs one backend when the provider is
created. At that point ONNX Runtime has not yet exposed the model graph through
GetCapability().

The EP therefore cannot inspect the complete graph before deciding whether its
kernel registry and implementation should come from MIGraphX or DirectML
without a substantially larger composite-EP redesign.

ModelBench already knows the model before session creation and owns the
registry containing its expected SHA256 checksum. It can supply a stable,
name-independent hint while the EP remains the owner of the actual routing
policy.

Model identity

For a registered ModelBench model:

  1. ModelBench looks up the model in ModelUrls.
  2. It reads the expected SHA256 checksum metadata.
  3. It decodes the checksum and takes the first 16 hexadecimal characters,
    representing a 64-bit content signature.
  4. In AMDGPU auto mode, it sends that 16-character string through the
    existing model_arch provider option.
  5. The EP normalizes the string and computes the existing 64-bit FNV-1a
    model_arch_hash().
  6. The EP matches (GPU architecture prefix, model_arch_hash) against
    kArchModelBackend.

The generated routing table performs the same FNV-1a calculation offline for
the same 16-character SHA prefix. Runtime and generated values therefore match
without adding a new provider option or a separate signature parser.

ModelBench changes

Remove name-based routing

The large directMLFallbackModels tables and the
shouldFallbackToDirectML(model_name, architecture) decision are removed from
the runtime path.

ModelBench no longer changes auto to directml based on the model name.

AMDGPU EP changes

Generated routing data

The generated header defines kArchModelBackend as an array of:

struct arch_model_backend {
    std::string_view arch_prefix;
    std::uint64_t model_arch_hash;
    Profile backend;
};

Each row contains:

  • A gfx architecture prefix.
  • The FNV-1a hash of the normalized 16-character SHA prefix supplied through
    model_arch.
  • Profile::DirectX.

Model names do not appear in the generated table or shipped EP binary.

Backend selection

select_backend() checks the generated table before applying general
architecture defaults. A matching row therefore reproduces the legacy
ModelBench fallback while keeping the decision inside the EP.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants