feat: Add structured output / JSON mode support across all LLM providers

[JNK234]

Feature Request

Add provider-native structured output / JSON mode support as a general capability across all LLM primitives, not just llm:choose. This would let model authors constrain LLM responses to specific formats (JSON objects, enums, typed fields) at the API level rather than relying on prompt engineering and NetLogo string parsing.

Related: #21 (llm:choose parsing reliability)

---

Motivation

The problem is broader than llm:choose

Every LLM primitive currently returns raw free-text with no structural guarantees:

Primitive	Returns	Parsing
llm:chat	responseMessage.content	None — raw string
llm:chat-async	responseMessage.content	None — raw string
llm:chat-with-template	responseMessage.content	None — raw string
llm:choose	fuzzy-matched choice	Fuzzy substring + number + random fallback

This means any NetLogo model that needs structured LLM output — JSON, numbers, yes/no, lists, key-value pairs — must either:

1. Hope the LLM follows prompt instructions (unreliable)
2. Parse free-text in NetLogo (extremely limited string manipulation)
3. Accept silent failures when the LLM returns unexpected formats

Real-world use cases that need structured output

• llm:choose — pick from a fixed list (current: fuzzy matching, should be: enum constraint)
• Agent decisions — return a JSON object like {"action": "move", "direction": "north", "confidence": 0.8}
• Numeric extraction — "How many resources?" should return just a number, not "I think there are about 5 resources"
• Boolean decisions — yes/no without preamble
• Structured analysis — return categorized data that NetLogo code can decompose

---

Provider API Research (March 2026)

All four supported providers now have native structured output capabilities. The common denominator is JSON Schema — each provider accepts a standard JSON Schema object, but wraps it differently in the request.

Provider Comparison Matrix

Feature	OpenAI	Anthropic (Claude)	Gemini	Ollama
API Parameter	response_format	output_config.format	generationConfig.responseJsonSchema	format
Schema wrapping	{type: "json_schema", json_schema: {name, strict, schema: {...}}}	{type: "json_schema", schema: {...}}	Raw schema object	Raw schema object
Simple JSON mode	{type: "json_object"}	N/A (use schema)	responseMimeType: "application/json" (no schema)	format: "json"
additionalProperties: false required?	Yes (mandatory)	Yes (mandatory)	No	No
All fields must be in required?	Yes (nullable for optional)	Yes (similar)	No	No
$ref / $defs support	Yes	No	No	N/A
Enum support	Yes (via enum in schema)	Yes	Yes (also text/x.enum MIME type)	Yes
Max nesting depth	5 levels	Not documented	Large schemas may 400	Model-dependent
Max properties	100 total	20 strict tools, 24 optional params	Not documented	No limit
Works with thinking mode?	Yes (o-series)	Yes (native format only, NOT with forced tool_choice)	Likely (not explicitly documented)	Yes
Mechanism	Constrained decoding	Constrained decoding (grammar-based)	Constrained decoding	GBNF grammar-based token masking (llama.cpp)
Response location	choices[0].message.content (string)	content[0].text (string)	candidates[0].content.parts[0].text (string)	message.content (string)

Provider-Specific Request Formats

OpenAI

{
  "model": "gpt-4o",
  "messages": [...],
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "my_schema",
      "strict": true,
      "schema": {
        "type": "object",
        "properties": {
          "action": { "type": "string" },
          "confidence": { "type": "number" }
        },
        "required": ["action", "confidence"],
        "additionalProperties": false
      }
    }
  }
}

Key constraints: additionalProperties: false mandatory on all objects. All fields must be in required (use "type": ["string", "null"] for optional). Max 100 properties, 5 nesting levels, 500 enum values. First request with new schema has extra latency (~10s) for schema caching.

Supported models: GPT-4o (2024-08-06+), GPT-4.1/4.1-mini/4.1-nano, o1, o3-mini, o3, o4-mini, o3-pro, GPT-5 family.

Anthropic (Claude)

Anthropic now has native structured output support (GA) via output_config.format — no longer requires the tool-use workaround:

{
  "model": "claude-sonnet-4-5-20250514",
  "max_tokens": 1024,
  "messages": [...],
  "output_config": {
    "format": {
      "type": "json_schema",
      "schema": {
        "type": "object",
        "properties": {
          "action": { "type": "string" },
          "confidence": { "type": "number" }
        },
        "required": ["action", "confidence"],
        "additionalProperties": false
      }
    }
  }
}

Key constraints: additionalProperties: false mandatory. No $ref/$defs support. Max 20 strict tools, 24 optional parameters. pattern (regex) supported. Incompatible with citations and message prefilling. Compatible with streaming and batch processing.

Thinking mode interaction: Native output_config.format IS compatible with Extended Thinking — grammar applies only to final output, not thinking blocks. However, the legacy tool_choice forcing approach is NOT compatible with thinking mode.

Supported models: Claude Opus 4.6, Sonnet 4.6, Sonnet 4.5, Opus 4.5, Haiku 4.5 (GA). Older Claude 3.x models only support the legacy tool-use workaround.

Legacy fallback (for Claude 3.x models): Use tool-use-as-JSON pattern with tool_choice: {"type": "tool", "name": "..."} and extract from content[].input.

Gemini

{
  "contents": [...],
  "generationConfig": {
    "responseMimeType": "application/json",
    "responseJsonSchema": {
      "type": "object",
      "properties": {
        "action": { "type": "string" },
        "confidence": { "type": "number" }
      },
      "required": ["action", "confidence"]
    }
  }
}

Two mutually exclusive schema fields:

• responseJsonSchema (newer, standard JSON Schema, recommended) — uses lowercase types ("string", "object")
• responseSchema (older, OpenAPI-style) — uses UPPERCASE types ("STRING", "OBJECT")

Key constraints: responseMimeType must be set to "application/json". Schema counts toward input token limit. Supports anyOf but not $ref, allOf, oneOf, not. For enum-only output, use responseMimeType: "text/x.enum" with responseSchema.

Supported models: Gemini 3.x, 2.5 Pro/Flash/Flash-Lite, 2.0 Flash/Flash-Lite. Gemini 2.0 models require explicit propertyOrdering.

Ollama

{
  "model": "llama3.1",
  "messages": [...],
  "format": {
    "type": "object",
    "properties": {
      "action": { "type": "string" },
      "confidence": { "type": "number" }
    },
    "required": ["action", "confidence"]
  }
}

Simplest format — the raw JSON Schema goes directly into format. For simple JSON mode: "format": "json".

Key details: Works with ALL models (grammar-based enforcement at llama.cpp sampling layer). No server-side schema validation. Performance impact for complex schemas. Best practice: also describe the desired JSON format in the prompt text.

Also supports OpenAI-compatible endpoint at /v1/chat/completions with response_format: {type: "json_schema", json_schema: {schema: {...}}}.

---

Critical Design Constraint: NetLogo Has No Map/Dictionary Type

NetLogo's type system is limited to: strings, numbers, booleans, lists, and agents. There is no native Map, Dict, or Object type. This means a JSON response like {"name": "Alice", "age": 30} cannot be returned as a key-value structure natively.

Proposed Solution: JSON → Nested Lists + Helper Primitive

Auto-convert JSON to nested NetLogo lists using this mapping:

JSON Type	NetLogo Type	Example
Object {}	List of [key value] pairs	[["name" "Alice"] ["age" 30]]
Array []	NetLogo list	["a" "b" "c"]
String	String	"hello"
Number	Number	42, 3.14
Boolean	Boolean	true, false
Null	Empty string	""

Add llm:get helper primitive for key-based access:

let result llm:chat-with-schema "Extract info" schema
let name llm:get result "name"           ;; "Alice"
let age llm:get result "age"             ;; 30
let skills llm:get result "skills"       ;; ["coding" "design"]
let city llm:get (llm:get result "address") "city"  ;; nested lookup

The llm:get primitive searches a [[key value] ...] list for a matching key and returns the value. This is ~10 lines of Scala and makes structured output consumption natural in NetLogo.

Implementation: A recursive ujson.Value → AnyRef converter (ujson is already a dependency):

def ujsonToNetLogo(value: ujson.Value): AnyRef = value match {
  case obj: ujson.Obj =>
    LogoList.fromIterator(obj.value.map { case (k, v) =>
      LogoList(k, ujsonToNetLogo(v))
    }.iterator)
  case arr: ujson.Arr =>
    LogoList.fromIterator(arr.value.map(ujsonToNetLogo).iterator)
  case ujson.Str(s)  => s
  case ujson.Num(n)  => Double.box(n)
  case ujson.Bool(b) => Boolean.box(b)
  case ujson.Null    => ""
}

---

Proposed Implementation

Phase 1: Data Model — Extend ChatRequest with responseFormat

// New: src/main/models/ResponseFormat.scala
sealed trait ResponseFormat
case class JsonSchemaFormat(schema: ujson.Value) extends ResponseFormat
case class EnumFormat(choices: List[String]) extends ResponseFormat
case object JsonObjectFormat extends ResponseFormat

// Modified: src/main/models/ChatRequest.scala
case class ChatRequest(
  model: String,
  messages: Seq[ChatMessage],
  maxTokens: Option[Int] = None,
  temperature: Option[Double] = None,
  thinkingConfig: Option[ThinkingConfig] = None,
  responseFormat: Option[ResponseFormat] = None  // NEW
)

Phase 2: Provider-specific request construction

Each provider's createProviderRequest adds the schema in the correct format:

Provider	Where schema goes	Notes
OpenAI	response_format.json_schema.schema	Auto-inject name, strict: true, enforce additionalProperties: false
Anthropic	output_config.format.schema	Use native format for Claude 4.5+; fall back to tool-use for Claude 3.x
Gemini	generationConfig.responseJsonSchema	Set responseMimeType: "application/json"
Ollama	format (raw schema)	Simplest — raw schema object

Schema normalization: Since OpenAI and Anthropic require additionalProperties: false and all fields in required, the extension should auto-inject these into user-provided schemas for consistency across providers.

Phase 3: Response parsing + JSON → NetLogo conversion

Response content comes back as a JSON string in the same field all providers already use — no changes to response parsing. The new step is converting the JSON string to a nested NetLogo list structure using the ujsonToNetLogo converter above.

Phase 4: New NetLogo primitives

Primitive	Purpose	Input	Output
llm:chat-with-schema	Schema-constrained chat	prompt-string schema-string	Parsed nested list
llm:chat-json	Freeform JSON mode	prompt-string	Raw JSON string
llm:get	Key lookup in parsed JSON	list key-string	Value at key
llm:set-response-format	Set persistent schema	schema-string	(command)
llm:clear-response-format	Remove persistent schema	—	(command)

Usage examples:

;; One-shot schema-constrained chat (returns parsed nested list)
let schema "{\"type\":\"object\",\"properties\":{\"action\":{\"type\":\"string\"},\"confidence\":{\"type\":\"number\"}},\"required\":[\"action\",\"confidence\"]}"
let result llm:chat-with-schema "What should the agent do?" schema
let action llm:get result "action"       ;; "explore"
let conf llm:get result "confidence"     ;; 0.85

;; Persistent schema (applies to regular llm:chat calls)
llm:set-response-format schema
let r1 llm:chat "What should agent 1 do?"
let r2 llm:chat "What should agent 2 do?"
llm:clear-response-format

;; Freeform JSON mode (no schema, just valid JSON)
let json-str llm:chat-json "List top 3 actions as JSON"

;; Improved llm:choose (uses EnumFormat internally, no API change)
let choice llm:choose "Pick a direction" ["north" "south" "east" "west"]

Phase 5: Update llm:choose internally

ChooseReporter switches to EnumFormat — no API change for NetLogo users:

val request = ChatRequest(
  model = model,
  messages = history.toSeq,
  responseFormat = Some(EnumFormat(choices))
)

With fallback to current fuzzy matching for providers/models that don't support structured output.

Phase 6: Graceful degradation

Not all models support structured output. The implementation must:

1. Detect capability: Each provider reports whether it supports structured output
2. Fallback: If unsupported, fall back to prompt engineering (current behavior)
3. Warning: Print a one-time warning: "Note: [provider] does not support structured output. Using prompt-based constraints."
4. No error: Never fail just because structured output is unavailable

---

Thinking Mode Compatibility

Provider	Structured Output + Thinking?	Notes
OpenAI	Yes	o-series works with response_format
Anthropic	Yes (native only)	output_config.format works with thinking. Legacy tool_choice forcing does NOT.
Gemini	Likely yes	Both live under generationConfig, no documented incompatibility
Ollama	Yes	Grammar-level enforcement is independent of thinking

---

Files to Modify

File	Change
src/main/models/ChatRequest.scala	Add responseFormat field
NEW src/main/models/ResponseFormat.scala	ResponseFormat sealed trait + case classes
src/main/providers/LLMProvider.scala	Add supportsStructuredOutput to trait
src/main/providers/BaseHttpProvider.scala	Pass responseFormat through ChatRequest
src/main/providers/OpenAIProvider.scala	Add response_format to request body
src/main/providers/ClaudeProvider.scala	Add output_config.format (native) with tool-use fallback
src/main/providers/GeminiProvider.scala	Add responseMimeType + responseJsonSchema
src/main/providers/OllamaProvider.scala	Add format to request body
src/main/LLMExtension.scala	Add ChatWithSchemaReporter, ChatJsonReporter, GetReporter, SetResponseFormatCommand, ClearResponseFormatCommand; update ChooseReporter
src/main/config/ConfigStore.scala	Add RESPONSE_FORMAT config key
NEW src/main/utils/JsonToNetLogo.scala	Recursive ujson.Value → LogoList converter

---

New Primitives Summary

Primitive	Type	Purpose	Response Format Used
llm:chat-with-schema	Reporter	Schema-constrained chat, returns parsed list	JsonSchemaFormat
llm:chat-json	Reporter	Force valid JSON output (raw string)	JsonObjectFormat
llm:get	Reporter	Key lookup in [[key val]...] list	N/A (utility)
llm:set-response-format	Command	Set persistent response schema	Config storage
llm:clear-response-format	Command	Remove persistent schema	Config storage
llm:choose (updated)	Reporter	Pick from enum list (internal upgrade)	EnumFormat

---

Backward Compatibility

• All existing primitives (llm:chat, llm:chat-async, llm:chat-with-template, llm:choose) continue to work unchanged
• llm:choose gets more reliable silently (structured output when available, same fuzzy fallback otherwise)
• New primitives are purely additive
• Providers that don't support structured output gracefully degrade

---

Testing Plan

Deterministic tests (CI, no API keys)

• Verify each provider generates correct request JSON for EnumFormat, JsonSchemaFormat, JsonObjectFormat
• Verify ujsonToNetLogo converter handles all JSON types (object, array, string, number, boolean, null, nested)
• Verify llm:get finds keys, returns correct values, handles missing keys
• Verify schema normalization (auto-inject additionalProperties: false, required)
• Verify llm:choose constructs EnumFormat request correctly

Ollama integration tests (local)

• llm:choose with structured output confirms exact match
• llm:chat-json returns parseable JSON
• llm:chat-with-schema returns schema-conforming JSON parsed as nested list
• llm:get works on live structured output responses

Manual live tests (all providers)

• Schema-constrained output with OpenAI, Claude, Gemini, Ollama
• Thinking mode + structured output combined
• Graceful degradation on unsupported models