Skip to content

Support configuring context window size (num_ctx) for local models in ChatOllama #327

Description

@bgreenwell

When using ChatOllama in chatlas to talk to local models (especially models with reasoning capabilities like gemma4 or larger parameter models), responses are easily truncated due to Ollama's default context window limit of 2048 tokens.

Because chatlas (seemingly) interfaces with Ollama via the OpenAI-compatible /v1/chat/completions API and strictly filters parameters in translate_model_params(), there is currently no way for users to increase the context window size (num_ctx) on-the-fly.

In contrast, the R equivalent (ellmer::chat_ollama) handles this seamlessly by communicating directly with Ollama's native /api/chat API, which natively supports setting num_ctx and num_predict via the request options.


Reproducible Examples (Reprex)

1. R (ellmer) - Success

Using ellmer, we can configure both max output tokens and a large context window, avoiding truncation:

library(ellmer)

# ellmer maps these parameters directly to Ollama's options block
chat <- chat_ollama(
  model = "gemma4",
  params = params(
    max_tokens = 2048,
    num_ctx = 8192
  )
)

response <- chat$chat("Write a very long essay about the history of statistics.")
cat(nchar(response)) # Successfully generates a long, complete response

2. Python (chatlas) - Truncation (Failure)

Using chatlas, setting max_tokens is supported, but we cannot pass num_ctx to the client. When the prompt + response (including thinking traces) exceeds 2048 tokens, it gets truncated:

from chatlas import ChatOllama

# Initialize client (no way to configure context window)
chat = ChatOllama(model="gemma4")
chat.set_model_params(max_tokens=2048)

# Writing a long prompt/response will hit Ollama's default 2048 context limit
response = chat.chat("Write a very long essay about the history of statistics.")
print(chat.get_turns()[-1].finish_reason) # prints "length" (truncated early)

Proposed Solutions

  1. Allow custom parameters in ChatOllama:
    Expose num_ctx (or a generic options payload dictionary) in the ChatOllama client initialization, and have the underlying OllamaProvider pass it along as part of the request payload to Ollama's completion endpoint (Ollama's /v1/chat/completions accepts custom parameters like num_ctx in the JSON request body).

  2. Expose provider-specific parameters in set_model_params():
    Instead of raising a TypeError for non-standard parameters, allow provider-specific keys like num_ctx to be passed through to the underlying request payload.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Priority: MediumValid bug or well-defined request with moderate impact or a workaround.ai-triage:doneMarks an issue whose AI triage workflow is complete.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions