When using ChatOllama in chatlas to talk to local models (especially models with reasoning capabilities like gemma4 or larger parameter models), responses are easily truncated due to Ollama's default context window limit of 2048 tokens.
Because chatlas (seemingly) interfaces with Ollama via the OpenAI-compatible /v1/chat/completions API and strictly filters parameters in translate_model_params(), there is currently no way for users to increase the context window size (num_ctx) on-the-fly.
In contrast, the R equivalent (ellmer::chat_ollama) handles this seamlessly by communicating directly with Ollama's native /api/chat API, which natively supports setting num_ctx and num_predict via the request options.
Reproducible Examples (Reprex)
1. R (ellmer) - Success
Using ellmer, we can configure both max output tokens and a large context window, avoiding truncation:
library(ellmer)
# ellmer maps these parameters directly to Ollama's options block
chat <- chat_ollama(
model = "gemma4",
params = params(
max_tokens = 2048,
num_ctx = 8192
)
)
response <- chat$chat("Write a very long essay about the history of statistics.")
cat(nchar(response)) # Successfully generates a long, complete response
2. Python (chatlas) - Truncation (Failure)
Using chatlas, setting max_tokens is supported, but we cannot pass num_ctx to the client. When the prompt + response (including thinking traces) exceeds 2048 tokens, it gets truncated:
from chatlas import ChatOllama
# Initialize client (no way to configure context window)
chat = ChatOllama(model="gemma4")
chat.set_model_params(max_tokens=2048)
# Writing a long prompt/response will hit Ollama's default 2048 context limit
response = chat.chat("Write a very long essay about the history of statistics.")
print(chat.get_turns()[-1].finish_reason) # prints "length" (truncated early)
Proposed Solutions
-
Allow custom parameters in ChatOllama:
Expose num_ctx (or a generic options payload dictionary) in the ChatOllama client initialization, and have the underlying OllamaProvider pass it along as part of the request payload to Ollama's completion endpoint (Ollama's /v1/chat/completions accepts custom parameters like num_ctx in the JSON request body).
-
Expose provider-specific parameters in set_model_params():
Instead of raising a TypeError for non-standard parameters, allow provider-specific keys like num_ctx to be passed through to the underlying request payload.
When using
ChatOllamainchatlasto talk to local models (especially models with reasoning capabilities likegemma4or larger parameter models), responses are easily truncated due to Ollama's default context window limit of2048tokens.Because
chatlas(seemingly) interfaces with Ollama via the OpenAI-compatible/v1/chat/completionsAPI and strictly filters parameters intranslate_model_params(), there is currently no way for users to increase the context window size (num_ctx) on-the-fly.In contrast, the R equivalent (
ellmer::chat_ollama) handles this seamlessly by communicating directly with Ollama's native/api/chatAPI, which natively supports settingnum_ctxandnum_predictvia the requestoptions.Reproducible Examples (Reprex)
1. R (
ellmer) - SuccessUsing
ellmer, we can configure both max output tokens and a large context window, avoiding truncation:2. Python (
chatlas) - Truncation (Failure)Using
chatlas, settingmax_tokensis supported, but we cannot passnum_ctxto the client. When the prompt + response (including thinking traces) exceeds 2048 tokens, it gets truncated:Proposed Solutions
Allow custom parameters in
ChatOllama:Expose
num_ctx(or a genericoptionspayload dictionary) in theChatOllamaclient initialization, and have the underlyingOllamaProviderpass it along as part of the request payload to Ollama's completion endpoint (Ollama's/v1/chat/completionsaccepts custom parameters likenum_ctxin the JSON request body).Expose provider-specific parameters in
set_model_params():Instead of raising a
TypeErrorfor non-standard parameters, allow provider-specific keys likenum_ctxto be passed through to the underlying request payload.