Explain the Metal teardown abort and disable residency sets in smoke runs - #78
Merged
Conversation
…runs
On Metal, a full suite could print `263 passed, 0 failures` and then abort
the VM with exit 134:
ggml-metal-device.m:622: GGML_ASSERT([rsets->data count] == 0) failed
The backtrace puts the abort after the suite is over, in static teardown:
exit(3) -> __cxa_finalize_ranges
-> ~vector<unique_ptr<ggml_metal_device>> (ggml-metal-device.cpp)
-> ggml_metal_device_free -> ggml_metal_rsets_free -> ggml_abort
ggml_metal_device_get keeps its devices in a function-local
`static std::vector`, so the device outlives the run and is destroyed by
__cxa_finalize_ranges after the BEAM calls exit(3). Its destructor asserts
the process-global MTLResidencySet collection is empty. A residency set is
added when a Metal buffer is created and removed in ggml_metal_buffer_free,
so a non-empty collection means a llama model or context resource has not
been destructed — and the BEAM makes no promise that NIF resource
destructors run before halt. Hence the race: 3 of 12 all-green full-suite
runs aborted (3B model + 9B MTP, real GGUF models, smoke and slow
included), and it fires on a 6-test run as readily as on the whole suite.
Set GGML_METAL_NO_RESIDENCY=1 in the documented smoke-test commands.
With it, dev->rsets is never allocated and ggml_metal_rsets_free returns
early, so the assert is unreachable rather than merely unlikely: 12 of 12
runs clean, same wall clock (~30s), 263 passed each. Residency sets are an
OS memory-residency hint, so buffer allocation and compute are unchanged.
Two things it deliberately is not:
* not `backend_free/0` in an after_suite hook — llama_backend_free() is
just ggml_quantize_free(); it frees no models, contexts or buffers and
cannot empty the collection.
* not System.put_env/2 in test_helper.exs — verified ineffective: the
device still logged `use residency sets = true` and a run still
aborted. os:putenv does not reach the C getenv ggml reads, so the
variable has to come from the shell.
The library is untouched: production keeps the upstream default. CI is
unaffected either way, since it runs on Linux with LLAMA_BACKEND=cpu and
loads no models, so no Metal device is ever created.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On Metal, a full suite could print
263 passed, 0 failuresand then abort the VMwith exit 134:
Root cause
The backtrace puts the abort after the suite is over, in static teardown:
ggml_metal_device_getkeeps its devices in a function-localstatic std::vector,so the device outlives the run and is destroyed by
__cxa_finalize_rangesafterthe BEAM calls
exit(3). Its destructor asserts that the process-globalMTLResidencySetcollection is empty. A residency set is added when a Metalbuffer is created and removed in
ggml_metal_buffer_free, so a non-emptycollection means a
llama_model/llama_contextNIF resource has not beendestructed — and the BEAM makes no promise that resource destructors run before
halt.
Hence the race: 3 of 12 all-green full-suite runs aborted (3B model + 9B MTP,
real GGUF models, smoke and slow included), and it fires on a 6-test run as
readily as on the whole suite — it is not accumulation, and not a failing test.
Fix
Set
GGML_METAL_NO_RESIDENCY=1in the documented smoke-test commands, and writethe mechanism down in
test/test_helper.exswhere the next person will hit it.With the variable set,
dev->rsetsis never allocated andggml_metal_rsets_freereturns early, so the assert is unreachable rather thanmerely unlikely. Residency sets are an OS memory-residency hint, so buffer
allocation and compute are unchanged.
What this deliberately is not
backend_free/0in anafter_suitehook.llama_backend_free()isjust
ggml_quantize_free()— it frees no models, contexts or buffers andcannot empty the collection.
System.put_env/2intest_helper.exs. Tried and verifiedineffective: the device still logged
use residency sets = trueand a runstill aborted.
os:putenvdoes not reach the Cgetenvggml reads, so thevariable has to come from the shell.
Verification
263 passed, 0 failures263 passed, 0 failuresEvery run after the change logs
use residency sets = false. Defaultmix testis green (189 passed, 4 skipped, 8 excluded) andmix format --check-formattedis clean.Scope
The library is untouched — production keeps the upstream default. CI is
unaffected either way: it runs on Linux with
LLAMA_BACKEND=cpuand loads nomodels, so no Metal device is ever created. The same exit-ordering race does
apply to any node that halts while a model or context is still referenced, which
the CHANGELOG entry notes; fixing that would need explicit release of those
resources before halt.
The v0.8.39 CHANGELOG entry is corrected in place (its full-suite figure now
reads 263 passed over 12 consecutive runs) rather than annotated retroactively,
because v0.8.39 is not tagged yet — remote tags stop at v0.8.36.