diff --git a/demos/baba-is-ai/README.md b/demos/baba-is-ai/README.md new file mode 100644 index 0000000..6f43ff4 --- /dev/null +++ b/demos/baba-is-ai/README.md @@ -0,0 +1,125 @@ +# Baba Is AI + +An LLM agent plays a simplified "Baba Is You" puzzle game where the rules of the world are physical word-block objects on a grid. The agent must push word-blocks to change or break rules in order to reach the goal. + +## The Experiment + +### Background + +Based on findings from the [BALROG benchmark](https://arxiv.org/abs/2411.13543) (Benchmarking Agentic LLM and VLM Reasoning On Games), which found that **all tested LLMs score near 0% on Baba Is You** despite understanding the rules perfectly when quizzed in isolation. + +### The Knowing-Doing Gap + +LLMs can explain that "rules are pushable objects" and "you need to rearrange word-blocks to change rules." But when given a spatial grid and asked to actually execute moves to push specific blocks in specific directions, they fail consistently. They *know* what to do but *cannot do* it. + +### What This Demo Tests + +Can an LLM agent, given a text observation of a 9x9 grid, figure out that it needs to: +1. Navigate to the right word-block +2. Push it in the right direction +3. Verify the rule changed +4. Navigate through the now-passable area to win + +This is **meta-rule discovery** -- the agent doesn't just follow rules, it manipulates them. + +### Why It Matters for ABM + +Agent-based models traditionally have fixed rules. This demo shows agents operating in a world where **the rules themselves are agents** (word-blocks) that can be moved. The rule engine is pure NetLogo ABM; the decision-making is LLM. Together they create a system where rules are emergent from the spatial arrangement of objects. + +## Prerequisites + +1. **Ollama** running locally with `qwen2.5:7b` pulled: + ```bash + ollama pull qwen2.5:7b + ``` +2. **NetLogo 7.0.3** with the LLM extension JAR installed +3. Built extension: run `./build.sh` from the extension root if needed + +## How to Run + +1. Open `baba-is-ai.nlogox` in NetLogo 7.0.3 +2. Select a level from the chooser +3. Click **Setup** to initialize +4. Click **Go** to let the LLM agent play autonomously, or **Step** for one move at a time +5. Toggle **show-observations?** to see the text observation sent to the LLM each turn +6. Watch the **Active Rules** monitor update as word-blocks are pushed + +## Levels + +### Level 1: Navigate (baseline) +``` +Rules: BABA IS YOU, FLAG IS WIN +Layout: baba on the left, flag on the right, clear path +Solution: Walk right to the flag (~7 moves) +Expected: Most LLMs solve this consistently +``` + +### Level 2: Break the Wall (medium) +``` +Rules: BABA IS YOU, WALL IS STOP, FLAG IS WIN +Layout: Wall column blocking the path to the flag +Solution: Navigate to the STOP word-block, push it away to break + "WALL IS STOP", then walk through walls to the flag +Expected: Some LLMs occasionally solve this; most get stuck +``` + +### Level 3: Push and Rearrange (hard) +``` +Rules: BABA IS YOU, ROCK IS PUSH, WALL IS STOP, FLAG IS WIN +Layout: Rocks near the WALL IS STOP rule, wall column, flag behind walls +Solution: Navigate around rocks, push STOP away, walk through walls +Expected: Very rarely solved -- demonstrates the knowing-doing gap +``` + +## Swapping Providers + +Edit `config` to try different LLM providers: + +``` +# OpenAI +provider=openai +model=gpt-4o-mini +api_key=YOUR_KEY_HERE + +# Anthropic +provider=anthropic +model=claude-sonnet-4-20250514 +api_key=YOUR_KEY_HERE + +# Gemini +provider=gemini +model=gemini-2.0-flash +api_key=YOUR_KEY_HERE +``` + +## Files + +| File | Purpose | +|------|---------| +| `baba-is-ai.nlogox` | Main NetLogo model with rule engine, levels, and LLM integration | +| `config` | LLM provider configuration (defaults to local Ollama) | +| `action-template.yaml` | Documents the prompt format (not used at runtime -- `llm:choose` builds its own prompt) | +| `README.md` | This file | + +## How It Works + +### Rule Engine +Three consecutive word-blocks in a horizontal or vertical line form a rule: `[NOUN] IS [PROPERTY]`. The engine scans all possible 3-tuples every tick and rebuilds the active rule set. + +### Push System +Word-blocks are **always pushable**. Other entities (rocks, walls) are pushable only if they have the PUSH property. Push chains propagate: pushing a block into another pushable block pushes both. + +### LLM Integration +Each tick, the agent receives a text observation including the grid layout, active rules, adjacent cell descriptions, and feedback from the last action. It chooses from `["up", "down", "left", "right"]` via `llm:choose`. History accumulates across ticks so the agent can learn from failed moves. + +## Extension Points + +- **New properties**: Add DEFEAT (touching kills baba), SINK (object + baba both destroyed), MELT/HOT combos +- **Multiple YOU entities**: What if two things are YOU and must coordinate? +- **Procedural level generation**: Random placement of word-blocks and obstacles +- **Rule construction**: Levels where the agent must *build* a rule, not just break one + +## Reference + +- BALROG paper: [arXiv 2411.13543](https://arxiv.org/abs/2411.13543) +- Baba Is You (original game): [hempuli.com/baba](https://hempuli.com/baba/) diff --git a/demos/baba-is-ai/action-template.yaml b/demos/baba-is-ai/action-template.yaml new file mode 100644 index 0000000..ed40e18 --- /dev/null +++ b/demos/baba-is-ai/action-template.yaml @@ -0,0 +1,20 @@ +system: | + You are playing Baba Is AI, a puzzle game where the rules of the world are + physical word-block objects on a grid. Three consecutive word-blocks in a row + or column form a rule: [NOUN] IS [PROPERTY]. For example, "BABA IS YOU" means + you control the baba character, "WALL IS STOP" means walls block movement, + and "FLAG IS WIN" means reaching a flag wins the level. + + CRITICAL INSIGHT: You can PUSH word-blocks to change or break rules. If you + push the STOP block away from "WALL IS STOP", walls become passable. Rules + are not fixed -- they are objects you can physically manipulate. + + Your goal: reach an entity that has the WIN property. + + Respond with exactly one of: up, down, left, right + +template: | + {observation} + + Think about which rules you need to change to reach the goal. + Choose your move. diff --git a/demos/baba-is-ai/baba-is-ai.nlogox b/demos/baba-is-ai/baba-is-ai.nlogox new file mode 100644 index 0000000..f368501 --- /dev/null +++ b/demos/baba-is-ai/baba-is-ai.nlogox @@ -0,0 +1,819 @@ + + + = 0] [ + let p item chain-idx chain-patches + let pushables-here get-pushables-on p + ask pushables-here [ + setxy (xcor + move-dx) (ycor + move-dy) + ] + set chain-idx chain-idx - 1 + ] + + report true +end + +;; ============================================================ +;; LLM INTEGRATION +;; ============================================================ + +to go + if game-state != "playing" [ stop ] + if not llm-config-loaded? [ + output-print "LLM config not loaded. Check config file." + set game-state "error" + stop + ] + + ask babas [ + if has-property? "BABA" "YOU" [ + ;; Cap history to system message + last 10 exchanges to avoid token bloat + let hist llm:history + if length hist > 21 [ + llm:set-history (sentence (list item 0 hist) (sublist hist (length hist - 20) (length hist))) + ] + + let obs build-observation + if show-observations? [ output-print obs ] + + set total-llm-calls total-llm-calls + 1 + carefully [ + let action llm:choose obs ["up" "down" "left" "right"] + set last-action action + set total-moves total-moves + 1 + set move-history lput action move-history + try-move action + ] [ + ;; Rate limit: wait and retry next tick + if member? "rate_limit" error-message [ + output-print "Rate limited — waiting 2s..." + wait 2 + stop + ] + output-print (word "LLM error: " error-message) + ] + ] + ] + + evaluate-rules + check-win-defeat + tick +end + +to step + go +end + +;; ============================================================ +;; OBSERVATION BUILDER +;; ============================================================ + +to-report build-observation + let obs "" + let baba-x [xcor] of one-of babas + let baba-y [ycor] of one-of babas + + ;; Find flag position + let flag-x -1 + let flag-y -1 + if any? flags [ + set flag-x [xcor] of one-of flags + set flag-y [ycor] of one-of flags + ] + + ;; Game context + set obs (word obs "BABA IS AI PUZZLE GAME\n") + set obs (word obs "You control BABA. Move up/down/left/right on a 9x9 grid.\n") + set obs (word obs "Word-blocks like [BABA] [IS] [YOU] form rules when aligned.\n") + set obs (word obs "You can push word-blocks by walking into them.\n\n") + + ;; Position and goal with explicit direction + set obs (word obs "Your position: (" baba-x ", " baba-y ")\n") + if flag-x >= 0 [ + set obs (word obs "FLAG position: (" flag-x ", " flag-y ")\n") + ;; Explicit directional hint + let dir-hint "" + if flag-x > baba-x [ set dir-hint (word dir-hint "right") ] + if flag-x < baba-x [ set dir-hint (word dir-hint "left") ] + if flag-y > baba-y [ + if dir-hint != "" [ set dir-hint (word dir-hint " and ") ] + set dir-hint (word dir-hint "up") + ] + if flag-y < baba-y [ + if dir-hint != "" [ set dir-hint (word dir-hint " and ") ] + set dir-hint (word dir-hint "down") + ] + if dir-hint != "" [ + set obs (word obs "FLAG is " dir-hint " from you.\n") + ] + ] + set obs (word obs "\n") + + ;; Active rules + set obs (word obs "Active Rules:\n") + foreach active-rules [ r -> + set obs (word obs "- " r "\n") + ] + if has-property? "WALL" "STOP" [ + ;; Find the STOP block and tell the agent exactly where it is + let stop-block one-of word-blocks with [word-text = "STOP"] + ifelse stop-block != nobody [ + let sx [xcor] of stop-block + let sy [ycor] of stop-block + set obs (word obs "!! WALLS BLOCK YOU. The [STOP] block is at (" sx ", " sy "). " + "Go to it and push it away to break WALL IS STOP, then walls become passable.\n") + ] [ + set obs (word obs "(Walls block movement. Push the STOP word-block away to break this rule.)\n") + ] + ] + set obs (word obs "\n") + + ;; Grid map — fixed-width for readability + set obs (word obs "Map (row 8=top, row 0=bottom, col 0=left, col 8=right):\n") + set obs (word obs " 0 1 2 3 4 5 6 7 8\n") + let grid-y max-pycor + while [grid-y >= min-pycor] [ + let row (word grid-y " ") + if grid-y < 10 [ set row (word " " row) ] + let grid-x min-pxcor + while [grid-x <= max-pxcor] [ + let p patch grid-x grid-y + let cell " . " + if any? (flags-on p) [ set cell " F " ] + if any? (rocks-on p) [ set cell " R " ] + if any? (walls-on p) [ set cell " W " ] + if any? (word-blocks-on p) [ + let wb one-of (word-blocks-on p) + let txt [word-text] of wb + ;; Pad/trim to 4 chars + while [length txt < 4] [ set txt (word txt " ") ] + set cell (word "[" substring txt 0 4 "]") + ] + if any? (babas-on p) [ set cell " YOU " ] + set row (word row cell) + set grid-x grid-x + 1 + ] + set obs (word obs row "\n") + set grid-y grid-y - 1 + ] + set obs (word obs "\n") + + ;; Adjacent info + set obs (word obs "What is next to you:\n") + set obs (word obs " up: " (describe-adjacent baba-x (baba-y + 1)) "\n") + set obs (word obs " down: " (describe-adjacent baba-x (baba-y - 1)) "\n") + set obs (word obs " left: " (describe-adjacent (baba-x - 1) baba-y) "\n") + set obs (word obs " right: " (describe-adjacent (baba-x + 1) baba-y) "\n") + set obs (word obs "\n") + + ;; Recent move history — helps prevent looping + if not empty? move-history [ + let recent-count min (list 5 (length move-history)) + let recent sublist move-history (length move-history - recent-count) (length move-history) + set obs (word obs "Your last " recent-count " moves: " reduce [ [a b] -> (word a ", " b) ] recent "\n") + ;; Detect looping + if recent-count >= 4 [ + let unique-moves remove-duplicates recent + if length unique-moves <= 2 [ + set obs (word obs "WARNING: You are stuck in a loop! Try a completely different direction.\n") + ] + ] + set obs (word obs "\n") + ] + + ;; Last action feedback + if [last-action] of one-of babas != "" [ + let b one-of babas + let result ifelse-value [last-action-succeeded?] of b [ "succeeded" ] [ "FAILED - blocked" ] + set obs (word obs "Last action: " [last-action] of b " (" result ")\n") + if not [last-action-succeeded?] of b [ + set obs (word obs "That direction is blocked. Try a different direction.\n") + ] + set obs (word obs "\n") + ] + + set obs (word obs "Which direction should you move?") + report obs +end + +to-report describe-adjacent [ adj-x adj-y ] + let p patch adj-x adj-y + if p = nobody [ report "boundary (impassable)" ] + + let descriptions [] + + if any? (babas-on p) [ set descriptions lput "you" descriptions ] + if any? (word-blocks-on p) [ + let wb one-of (word-blocks-on p) + set descriptions lput (word "word-block [" [word-text] of wb "] (pushable)") descriptions + ] + if any? (walls-on p) [ + ifelse has-property? "WALL" "STOP" + [ set descriptions lput "wall (blocking)" descriptions ] + [ set descriptions lput "wall (passable)" descriptions ] + ] + if any? (rocks-on p) [ + ifelse has-property? "ROCK" "PUSH" + [ set descriptions lput "rock (pushable)" descriptions ] + [ set descriptions lput "rock" descriptions ] + ] + if any? (flags-on p) [ + ifelse has-property? "FLAG" "WIN" + [ set descriptions lput "flag (WIN!)" descriptions ] + [ set descriptions lput "flag" descriptions ] + ] + + ifelse empty? descriptions + [ report "empty" ] + [ report reduce [ [a b] -> (word a ", " b) ] descriptions ] +end + +;; ============================================================ +;; WIN/DEFEAT CHECK +;; ============================================================ + +to check-win-defeat + ;; Check if BABA IS YOU rule is broken + if not has-property? "BABA" "YOU" [ + set game-state "lost - BABA IS YOU rule broken" + output-print "DEFEAT: The BABA IS YOU rule was broken!" + stop + ] + + ;; Check if baba is on a flag with WIN property + ask babas [ + if any? (flags-on patch-here) and has-property? "FLAG" "WIN" [ + set game-state "won" + output-print (word "VICTORY! Solved in " total-moves " moves and " total-llm-calls " LLM calls.") + ] + ] + + ;; Check move limit + if total-moves >= max-moves and game-state = "playing" [ + set game-state "out of moves" + output-print (word "OUT OF MOVES after " total-moves " moves.") + ] +end + +;; ============================================================ +;; REPORTERS FOR MONITORS +;; ============================================================ + +to-report level-report + report (word "Level " (substring level-choice 0 1)) +end + +to-report moves-report + report total-moves +end + +to-report llm-calls-report + report total-llm-calls +end + +to-report status-report + report game-state +end + +to-report rules-report + ifelse empty? active-rules + [ report "none" ] + [ report reduce [ [a b] -> (word a ", " b) ] active-rules ] +end + ]]> + + + + + + + + + + + + + level-report + moves-report + llm-calls-report + status-report + rules-report + + + + + + plot total-moves + + + + + + + + plot total-llm-calls + + + + + ## Baba Is AI + +### What Is It? + +A simplified recreation of "Baba Is You" where an LLM agent must solve puzzles by pushing word-blocks that define the rules of the world. Three consecutive word-blocks in a row or column form a rule like "BABA IS YOU" or "WALL IS STOP". The agent can push these word-blocks apart to change or break rules. + +### How It Works + +**Rules are physical objects.** Word-blocks sit on a 9x9 grid. When three blocks align as [NOUN] IS [PROPERTY], that rule takes effect. Push a block out of alignment and the rule breaks. + +**The LLM decides each move.** Every tick, the agent receives a text observation of the grid and chooses up/down/left/right via `llm:choose`. History accumulates so the agent can learn from failed moves. + +**The knowing-doing gap.** Based on the BALROG benchmark, LLMs score near 0% on Baba Is You despite understanding the rules perfectly when quizzed. They *know* rules are pushable but *cannot execute* the spatial reasoning to rearrange them. + +### Levels + +1. **Navigate** — Walk right to the flag. No rule manipulation needed. +2. **Break the Wall** — A wall blocks the path. Push the STOP word-block away from "WALL IS STOP" to make walls passable. +3. **Push and Rearrange** — Rocks and walls block the path. Navigate around rocks, break the wall rule, reach the flag. + +### How to Use + +1. Select a level from the chooser +2. Click **Setup** to initialize the level +3. Click **Go** to let the LLM agent play, or **Step** for one move at a time +4. Toggle **show-observations?** to see what the LLM receives each turn +5. Watch the Active Rules monitor — it updates when word-blocks are pushed + +### Things to Notice + +- Level 1 is usually solved quickly (simple navigation) +- Level 2 requires the agent to realize it must push STOP away — most LLMs struggle here +- Level 3 adds rocks and more complex spatial reasoning — rarely solved +- The Active Rules monitor shows rules changing in real-time as blocks move + +### Extending + +- Edit `config` to try different LLM providers (OpenAI, Claude, Gemini) +- Add new word types (DEFEAT, SINK) for more complex puzzles +- Create levels where the agent must *construct* rules, not just break them + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + setup repeat 75 [ go ] + diff --git a/demos/baba-is-ai/config b/demos/baba-is-ai/config new file mode 100644 index 0000000..4fcd404 --- /dev/null +++ b/demos/baba-is-ai/config @@ -0,0 +1,6 @@ +provider=ollama +model=qwen2.5:7b +base_url=http://localhost:11434 +temperature=0.3 +max_tokens=100 +timeout_seconds=45