feat(chat): 0.9.0 Expose ContextShiftPolicy in EngineChat.generate - #8
Closed
sebasbad wants to merge 5 commits into
Closed
feat(chat): 0.9.0 Expose ContextShiftPolicy in EngineChat.generate#8sebasbad wants to merge 5 commits into
sebasbad wants to merge 5 commits into
Conversation
ffigen 20.1.1 -> 21.0.0 and lints 5.0.0 -> 6.1.0. Resolved versions also pull image 4.9.1 and test 1.31.2. `dart analyze` is clean under lints 6 — no new rules fire on this codebase. The ffigen compiler-opts pointed -resource-dir at clang 17, which no longer exists in the installed Command Line Tools (only 21 ships now), so `dart run ffigen` was using a dead resource dir. Point it at 21. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Submodule d6d0ce821 (b9581, 2026-06-09) -> afeebe103 (b10182). Bindings regenerated with ffigen 21; version.dart regenerated; README pin reference updated (it still claimed b9360, which was already stale). Upstream e6dd0e29a collapsed the use_mmap / use_direct_io / use_mlock booleans in llama_model_params into a single llama_load_mode enum. The ModelParams public API keeps its three booleans and now maps them at the FFI boundary, so this is not a breaking change for callers. One semantic caveat: the new enum has no direct-I/O-plus-mlock value, so when both are requested direct I/O wins and mlock is dropped. useDirectIo keeps its documented precedence over useMmap. mtmd_encode is deprecated upstream in favor of mtmd_encode_chunk, but this package reaches multimodal via mtmd_helper_eval_chunks and never called it, so no change was needed there. Verified: dart analyze clean; full model-backed suite green with -j 1 (43 passed, 2 skipped) against SmolVLM2-256M on macOS/Metal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LlamaLog.silence() installs a Pointer.fromFunction bound to the isolate that registered it, but the slot it occupies lives in process-global llama.cpp/ggml state and outlives that isolate. dispose() only dropped Dart-side references, so the stale pointer stayed installed: the next isolate to emit a log line invoked a callback owned by a dead isolate and the VM aborted with "Cannot invoke native callback from a different isolate". This reproduced as `dart test` dying in smoke_test whenever a suite that calls silence() (batch_embed, generation, speculative) ran first in the same process. Each file passed in isolation, which is why it went unnoticed — and CI never ran the model-backed tests at all. dispose() now calls LlamaLog.useDefault() while _bindings is still valid. Full suite with -j 1 goes from aborting to 43 passed, 2 skipped. Not a llama.cpp bump regression — it surfaced under Dart 3.12's stricter cross-isolate callback check. Parallel `dart test` still aborts on the concurrent variant of this race, where one isolate holds a live callback while another loads a model; that needs silence() to stop using a Dart callback at all, and is left for a separate change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nistic dispose & context shift support Consolidate mobile & Apple ecosystem enhancements for llama_cpp_dart 0.9.0 release: - Platform Support: Declare Flutter ffiPlugin support for iOS and macOS in pubspec.yaml. - Apple XCFramework: Build script fixes for self-contained dynamic Llama.xcframework (universal simulator arm64+x86_64, executable name fix, bundle id normalization). - Mobile-First API: Added ContextParams.mobile preset, estimateVramBytes(), model.isDisposed, and engine worker chat cancel. - Resource Safety: Deterministic C++ VRAM teardown for model, context, and multimodal instances on isolate shutdown. - Reasoning Models: Exposed ContextShiftPolicy in EngineChat.generate() for auto context sliding on reasoning/thinking models.
sebasbad
force-pushed
the
feat/0.9.0-context-shift-policy-chat
branch
from
July 31, 2026 23:17
5ced252 to
060f171
Compare
Owner
Author
|
Superseded by upstream PR netdur#109 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Exposes
shiftPolicyandshiftparameters through the fullEngineChatstack so callers can opt-in to automatic context sliding when running reasoning models (DeepSeek R1, Qwen-Thinking, etc.) that produce long<think>streams.Without this,
chat.generate()always ran withContextShiftPolicy.off, causing a hardLlamaDecodeException: context full at pos=N / nCtx=1024crash whenever a reasoning model's thinking stream exceeded the mobile context window.Changes
lib/src/isolate/messages.dartshiftPolicyandshiftfields toGenerateChatCommand(defaulting toContextShiftPolicy.off/ContextShift.defaults— fully backward-compatible).lib/src/isolate/engine.dartshiftPolicyandshiftnamed params toEngineChat.generate()and the privateLlamaEngine._generateChat(), forwarding them intoGenerateChatCommand.lib/src/isolate/worker.dartcmd.shiftPolicyandcmd.shiftinto the existing_streamSessionGenerate()call inside theGenerateChatCommandhandler (the underlying session-level shifting was already implemented).Usage