diff --git a/.gitignore b/.gitignore index b1cf135..e5f71bd 100644 --- a/.gitignore +++ b/.gitignore @@ -50,6 +50,7 @@ commands/ config.txt config-*.txt !demos/provider-sensitivity/config-multi-provider.txt +!demos/social-deduction-game/config.txt *.env *.env.* *.secrets* diff --git a/demos/social-deduction-game/README.md b/demos/social-deduction-game/README.md new file mode 100644 index 0000000..89ea3b7 --- /dev/null +++ b/demos/social-deduction-game/README.md @@ -0,0 +1,57 @@ +# Social Deduction Game (Undercover / Spy Words) + +## Hypothesis + +LLM agents will correctly identify spies at below-chance accuracy because they struggle to integrate cross-round description patterns into coherent suspicion, while LLM spies will be caught faster than chance because they produce descriptions that are subtly misaligned with the majority word. + +## Source + +Inspired by LLMArena (ACL 2024) — social deduction games as an evaluation framework for LLM reasoning and theory of mind. + +## Game Mechanics + +1. **Setup**: N players arranged in a circle. One is randomly assigned the "spy word"; the rest share the "regular word". Words are similar but distinct (e.g., apple vs pear). +2. **Description Phase**: Each alive player describes their word in 1-2 sentences without saying it. +3. **Voting Phase**: Each alive player votes for who they think is the spy based on descriptions. +4. **Elimination**: The player with the most votes is eliminated. +5. **Win Condition**: Regulars win if the spy is eliminated. The spy wins if only 2 players remain or max rounds are reached. + +## Word Pairs (3 difficulty tiers) + +| Difficulty | Regular Word | Spy Word | +|-----------|-------------|----------| +| High (hard to detect) | apple/guitar/coffee/pillow/ocean | pear/ukulele/tea/cushion/lake | +| Medium | bicycle/painting/castle/penguin/candle | motorcycle/photograph/mansion/duck/flashlight | +| Low (easy to detect) | airplane/forest/piano/book/snowflake | submarine/desert/drums/television/raindrop | + +## Files + +| File | Purpose | +|------|---------| +| `social-deduction-game.nlogox` | Main NetLogo model | +| `config.txt` | LLM provider configuration (default: Ollama) | +| `description-template.yaml` | Prompt template for description phase | +| `README.md` | This file | + +## Setup + +1. Install the LLM extension (see main repo README) +2. Configure `config.txt` with your preferred provider +3. Open `social-deduction-game.nlogox` in NetLogo 7.0.3+ +4. Click **Setup** to initialize players +5. Click **Go** to run the full game, or **Step** for one round at a time + +## Interface Controls + +- **num-players**: Number of players (4-8, default 6) +- **max-rounds**: Maximum rounds before spy wins (3-10, default 6) +- **difficulty-level**: Word pair similarity ("low"/"medium"/"high") +- **show-descriptions?**: Display description speech bubbles +- **show-votes?**: Display vote targets +- **auto-reveal?**: Reveal spy identity when game ends + +## Expected Results + +- **Detection rate**: Expected below 1/N chance (where N = number of players) +- **Spy survival**: Spy descriptions tend to be subtly "off", leading to faster-than-chance detection in some rounds, but overall detection accuracy remains low because regulars struggle to synthesize cross-round patterns +- **Difficulty effect**: Higher difficulty (more similar words) should decrease detection rate diff --git a/demos/social-deduction-game/config.txt b/demos/social-deduction-game/config.txt new file mode 100644 index 0000000..19611c7 --- /dev/null +++ b/demos/social-deduction-game/config.txt @@ -0,0 +1,5 @@ +provider=ollama +model=llama3.2:3b +temperature=0.4 +max_tokens=80 +timeout_seconds=30 diff --git a/demos/social-deduction-game/description-template.yaml b/demos/social-deduction-game/description-template.yaml new file mode 100644 index 0000000..263271a --- /dev/null +++ b/demos/social-deduction-game/description-template.yaml @@ -0,0 +1,29 @@ +system: | + You are {player_name} in a social deduction game called Undercover. + Several players each have a secret word. Most share the SAME word, but one player (the spy) has a DIFFERENT but similar word. + Each round you give a clue about your word. After clues, players vote to eliminate the suspected spy. + + CRITICAL RULES: + - Your clue must be DIFFERENT from every clue already given. Read the previous clues carefully and say something NEW. + - Be INDIRECT — describe a memory, association, or situation involving your word, NOT its obvious properties + - Do NOT describe what it looks/tastes/smells like directly + - Do NOT say the word itself + - Write exactly ONE sentence, under 20 words + - Do NOT start with "Best enjoyed" or "Often enjoyed" or "My word" + + GOOD CLUE EXAMPLES (indirect, subtle): + - "Some people collect fancy versions of these from around the world." + - "You might see one of these at a birthday party or during a power outage." + + BAD CLUE EXAMPLES (too obvious, reveals the word): + - "It is brewed in a pot every morning" — too obvious + - "It has a warm glow and a wick" — basically says the word + +template: | + Your secret word is: {my_word} + Round {round_number}. + + Previous clues from all players (DO NOT repeat any of these): + {transcript} + + Think of a unique angle or personal association with your word that nobody has mentioned yet. Write ONE indirect, subtle clue under 20 words. diff --git a/demos/social-deduction-game/social-deduction-game.nlogox b/demos/social-deduction-game/social-deduction-game.nlogox new file mode 100644 index 0000000..a6be13a --- /dev/null +++ b/demos/social-deduction-game/social-deduction-game.nlogox @@ -0,0 +1,661 @@ + + + + ask p [ + llm:clear-history + let desc "" + carefully [ + set desc llm:chat-with-template "description-template.yaml" (list + (list "player_name" player-name) + (list "my_word" my-word) + (list "round_number" (word current-round)) + (list "transcript" transcript-text)) + set total-llm-calls total-llm-calls + 1 + ] [ + set desc "My word relates to something common and familiar." + print (word player-name " description error: " error-message) + ] + + ;; Word leak check: if description contains the actual word, replace + if member? my-word desc [ + set desc "My word relates to something common and familiar." + ] + + set my-description desc + + ;; Update transcript + set full-transcript (word full-transcript "Round " current-round " - " player-name ": " desc "\n") + + ;; Update label + if show-descriptions? [ + set label (word player-name ": " truncate-string desc 40) + ] + + if show-descriptions? [ + print (word player-name ": " desc) + ] + ] + ] +end + +;;; ========== VOTING PHASE ========== + +to run-voting-phase + set current-phase "voting" + + ;; Update center indicator + ask indicators [ + set label (word "ROUND " current-round " — VOTING...") + set label-color white + ] + + ;; Reset vote counts + ask players with [alive?] [ + set vote-count 0 + ] + + let alive-players sort-on [who] players with [alive?] + foreach alive-players [ p -> + ask p [ + ;; Build list of other alive player names + let others-names map [q -> [player-name] of q] filter [q -> q != self] alive-players + let others-str reduce [[ a b ] -> (word a ", " b)] others-names + + ;; Vote via template — LLM picks from constrained list + llm:clear-history + let vote-name "" + + ;; Step 1: Try template-based vote + carefully [ + let raw-vote llm:chat-with-template "vote-template.yaml" (list + (list "player_name" player-name) + (list "my_word" my-word) + (list "transcript" full-transcript) + (list "other_players" others-str)) + set total-llm-calls total-llm-calls + 1 + ;; Match response against valid player names (others only) + foreach others-names [ n -> + if member? n raw-vote and vote-name = "" [ + set vote-name n + ] + ] + ] [ + print (word player-name " template vote error: " error-message) + ] + + ;; Step 2: Fallback to llm:choose if template didn't yield a valid name + if vote-name = "" [ + carefully [ + llm:clear-history + set vote-name llm:choose + (word "You are " player-name ". Pick the most suspicious player from: " others-str ". You CANNOT pick yourself.") + others-names + set total-llm-calls total-llm-calls + 1 + ] [ + set vote-name one-of others-names + print (word player-name " vote fallback error: " error-message) + ] + ] + + set my-vote vote-name + + ;; Increment the target's vote count + let target one-of players with [player-name = vote-name and alive?] + if target != nobody [ + ask target [ + set vote-count vote-count + 1 + set total-votes-received total-votes-received + 1 + ] + ] + + ;; Draw vote arrow from voter to target + if target != nobody [ + create-vote-arrow-to target [ + set color [base-color] of myself + set thickness 0.3 + ] + ] + + if show-votes? [ + set label (word player-name " -> " vote-name) + print (word player-name " votes for " vote-name) + ] + ] + ] +end + +;;; ========== ELIMINATION ========== + +to run-elimination + set current-phase "elimination" + + let alive-players players with [alive?] + let max-votes max [vote-count] of alive-players + + ;; Find all players with max votes + let candidates alive-players with [vote-count = max-votes] + + ;; Tie-breaking: highest total historical votes, then random + let eliminated-player max-one-of candidates [total-votes-received] + + ask eliminated-player [ + set alive? false + set elimination-round current-round + set color base-color - 4 + set size 2.0 + set label (word player-name " [OUT R" current-round "]") + set label-color base-color - 4 + + print (word ">>> " player-name " eliminated in round " current-round + " with " vote-count " votes" + (ifelse-value is-spy? [" (SPY!)"] [" (regular)"])) + ] + + ;; Update center indicator + ask indicators [ + set label (word "ELIMINATED: " [player-name] of eliminated-player) + set label-color 15 + ] +end + +;;; ========== WIN CONDITION ========== + +to check-win-condition + let alive-count count players with [alive?] + let spy-alive? any? players with [alive? and is-spy?] + + ;; Spy eliminated -> regulars win + if not spy-alive? [ + set game-over? true + set winner "regulars" + set spy-detected? true + set rounds-to-detection current-round + set total-games-played total-games-played + 1 + set total-regular-wins total-regular-wins + 1 + print "*** REGULARS WIN! The spy has been eliminated. ***" + ask indicators [ set label "REGULARS WIN!" set label-color 55 ] + if auto-reveal? [ reveal-spy ] + stop + ] + + ;; 2 or fewer alive -> spy wins + if alive-count <= 2 [ + set game-over? true + set winner "spy" + set spy-detected? false + set total-games-played total-games-played + 1 + set total-spy-wins total-spy-wins + 1 + print "*** SPY WINS! Too few players remain. ***" + ask indicators [ set label "SPY WINS!" set label-color 15 ] + if auto-reveal? [ reveal-spy ] + stop + ] + + ;; Max rounds reached -> spy wins + if current-round >= max-rounds [ + set game-over? true + set winner "spy" + set spy-detected? false + set total-games-played total-games-played + 1 + set total-spy-wins total-spy-wins + 1 + print "*** SPY WINS! Max rounds reached. ***" + ask indicators [ set label "SPY WINS!" set label-color 15 ] + if auto-reveal? [ reveal-spy ] + stop + ] +end + +;;; ========== REVEAL & UTILITIES ========== + +to reveal-spy + let the-spy one-of players with [is-spy?] + let spy-name [player-name] of the-spy + + ask the-spy [ + set color 15 + set size 5.0 + set label (word "SPY: " my-word) + set label-color 15 + ] + + ask players with [not is-spy?] [ + ifelse alive? [ + set label (word player-name " [" my-word "]") + set label-color base-color + 2 + ] [ + set label (word "[OUT R" elimination-round "] " my-word) + set label-color base-color - 4 + ] + ] + + ;; Update center indicator + ask indicators [ + set label (word "SPY WAS " spy-name "!") + set label-color 15 + ] + + print (word "Regular word: " regular-word " | Spy word: " spy-word) +end + +to draw-background + ;; Dark navy/charcoal gradient — lighter near center + ask patches [ + let dist distancexy 0 0 + set pcolor scale-color 3 dist 25 0 + ] + + ;; Circular table: ring at distance 5-6 from center + ask patches with [distancexy 0 0 >= 5 and distancexy 0 0 <= 6] [ + set pcolor 13 + ] + ;; Inner table fill + ask patches with [distancexy 0 0 < 5] [ + set pcolor 12 + ] +end + +to new-game + setup +end + +to-report truncate-string [str max-len] + ifelse length str > max-len + [ report (word substring str 0 max-len "...") ] + [ report str ] +end + +to-report detection-rate + ifelse total-games-played > 0 + [ report precision (total-regular-wins / total-games-played * 100) 1 ] + [ report 0 ] +end + ]]> + + + + + + + + + + + + + + + + + regular-word + total-regular-wins + total-spy-wins + total-games-played + winner + total-llm-calls + detection-rate + current-phase + current-round + ifelse-value game-over? [spy-word] ["(hidden)"] + + + + let alive-players players with [alive? or elimination-round = current-round] + ask alive-players [ + create-temporary-plot-pen player-name + set-current-plot-pen player-name + set-plot-pen-color base-color + plotxy current-round vote-count + ] + + + + + + + + 0 and game-over? [ plotxy total-games-played detection-rate ]]]> + + + + ## WHAT IS IT? + +A social deduction game (Undercover / Spy Words) where LLM agents try to identify a hidden spy among them. Each player receives a secret word — most players share the same word (regulars), but one player has a different but similar word (the spy). Players describe their words and vote to eliminate the suspected spy. + +## HOW IT WORKS + +1. **Setup**: Players are arranged in a circle. One random player is assigned the spy word; the rest get the regular word. +2. **Description Phase**: Each alive player uses the LLM to generate a 1-2 sentence description of their word without saying it. +3. **Voting Phase**: Each alive player uses `llm:chat-with-template` to vote for the most suspicious player based on all descriptions so far. +4. **Elimination**: The player with the most votes is eliminated. Ties broken by historical vote totals. +5. **Win Check**: Regulars win if the spy is eliminated. The spy wins if 2 or fewer players remain or max rounds are reached. + +## HOW TO USE IT + +1. Set **num-players** (4-8), **max-rounds**, and **difficulty-level** +2. Click **Setup** to initialize the game +3. Click **Go** for continuous play or **Step** for one round at a time +4. Watch descriptions and votes in the Command Center and on player labels +5. Click **New Game** to start fresh (cumulative stats are preserved) + +## THINGS TO NOTICE + +- Do spy descriptions seem subtly different from regular players? +- Do regulars converge on voting for the spy, or do they scatter votes? +- How does difficulty level affect detection rate? +- Does the spy get caught faster with more obviously different words (low difficulty)? + +## THINGS TO TRY + +- Compare detection rates across difficulty levels +- Try with 4 players (harder for regulars) vs 8 players (more data points) +- Change the LLM provider/model in config.txt and compare performance +- Watch how vote patterns evolve across rounds + +## EXTENDING THE MODEL + +- Add a "guess the word" phase where the spy tries to guess the regular word before being eliminated +- Implement multiple spies +- Add inter-player discussion/messaging between description and voting phases +- Track which specific descriptions led to correct/incorrect identifications + +## CREDITS AND REFERENCES + +Inspired by the Undercover (Who Is Spy?) party game and LLMArena (ACL 2024) — social deduction games as an evaluation framework for LLM reasoning. + +Built using the NetLogo LLM Extension. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + setup repeat 75 [ go ] + diff --git a/demos/social-deduction-game/vote-template.yaml b/demos/social-deduction-game/vote-template.yaml new file mode 100644 index 0000000..58c84d5 --- /dev/null +++ b/demos/social-deduction-game/vote-template.yaml @@ -0,0 +1,19 @@ +system: | + You are {player_name} in a social deduction game called Undercover. + Most players share the same secret word, but ONE player (the spy) has a different but similar word. + You must vote for the player whose descriptions suggest they have a DIFFERENT word than yours. + IMPORTANT: You CANNOT vote for yourself. You must pick someone else. + Respond with ONLY a player name (e.g. "Player 3"). Nothing else — no explanation. + +template: | + Your secret word is: {my_word} + You are: {player_name} + + All descriptions so far: + {transcript} + + Players you can vote for (NOT yourself): {other_players} + + Think about which player's descriptions don't quite match your word "{my_word}". + Look for concrete details that fit a SIMILAR but DIFFERENT object. + Respond with ONLY the player name you vote for. You CANNOT vote for {player_name}.