Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ commands/
config.txt
config-*.txt
!demos/provider-sensitivity/config-multi-provider.txt
!demos/social-deduction-game/config.txt
*.env
*.env.*
*.secrets*
57 changes: 57 additions & 0 deletions demos/social-deduction-game/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Social Deduction Game (Undercover / Spy Words)

## Hypothesis

LLM agents will correctly identify spies at below-chance accuracy because they struggle to integrate cross-round description patterns into coherent suspicion, while LLM spies will be caught faster than chance because they produce descriptions that are subtly misaligned with the majority word.

## Source

Inspired by LLMArena (ACL 2024) — social deduction games as an evaluation framework for LLM reasoning and theory of mind.

## Game Mechanics

1. **Setup**: N players arranged in a circle. One is randomly assigned the "spy word"; the rest share the "regular word". Words are similar but distinct (e.g., apple vs pear).
2. **Description Phase**: Each alive player describes their word in 1-2 sentences without saying it.
3. **Voting Phase**: Each alive player votes for who they think is the spy based on descriptions.
4. **Elimination**: The player with the most votes is eliminated.
5. **Win Condition**: Regulars win if the spy is eliminated. The spy wins if only 2 players remain or max rounds are reached.

## Word Pairs (3 difficulty tiers)

| Difficulty | Regular Word | Spy Word |
|-----------|-------------|----------|
| High (hard to detect) | apple/guitar/coffee/pillow/ocean | pear/ukulele/tea/cushion/lake |
| Medium | bicycle/painting/castle/penguin/candle | motorcycle/photograph/mansion/duck/flashlight |
| Low (easy to detect) | airplane/forest/piano/book/snowflake | submarine/desert/drums/television/raindrop |

## Files

| File | Purpose |
|------|---------|
| `social-deduction-game.nlogox` | Main NetLogo model |
| `config.txt` | LLM provider configuration (default: Ollama) |
| `description-template.yaml` | Prompt template for description phase |
| `README.md` | This file |

## Setup

1. Install the LLM extension (see main repo README)
2. Configure `config.txt` with your preferred provider
3. Open `social-deduction-game.nlogox` in NetLogo 7.0.3+
4. Click **Setup** to initialize players
5. Click **Go** to run the full game, or **Step** for one round at a time

## Interface Controls

- **num-players**: Number of players (4-8, default 6)
- **max-rounds**: Maximum rounds before spy wins (3-10, default 6)
- **difficulty-level**: Word pair similarity ("low"/"medium"/"high")
- **show-descriptions?**: Display description speech bubbles
- **show-votes?**: Display vote targets
- **auto-reveal?**: Reveal spy identity when game ends

## Expected Results

- **Detection rate**: Expected below 1/N chance (where N = number of players)
- **Spy survival**: Spy descriptions tend to be subtly "off", leading to faster-than-chance detection in some rounds, but overall detection accuracy remains low because regulars struggle to synthesize cross-round patterns
- **Difficulty effect**: Higher difficulty (more similar words) should decrease detection rate
5 changes: 5 additions & 0 deletions demos/social-deduction-game/config.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
provider=ollama
model=llama3.2:3b
temperature=0.4
max_tokens=80
timeout_seconds=30
29 changes: 29 additions & 0 deletions demos/social-deduction-game/description-template.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
system: |
You are {player_name} in a social deduction game called Undercover.
Several players each have a secret word. Most share the SAME word, but one player (the spy) has a DIFFERENT but similar word.
Each round you give a clue about your word. After clues, players vote to eliminate the suspected spy.

CRITICAL RULES:
- Your clue must be DIFFERENT from every clue already given. Read the previous clues carefully and say something NEW.
- Be INDIRECT — describe a memory, association, or situation involving your word, NOT its obvious properties
- Do NOT describe what it looks/tastes/smells like directly
- Do NOT say the word itself
- Write exactly ONE sentence, under 20 words
- Do NOT start with "Best enjoyed" or "Often enjoyed" or "My word"

GOOD CLUE EXAMPLES (indirect, subtle):
- "Some people collect fancy versions of these from around the world."
- "You might see one of these at a birthday party or during a power outage."

BAD CLUE EXAMPLES (too obvious, reveals the word):
- "It is brewed in a pot every morning" — too obvious
- "It has a warm glow and a wick" — basically says the word

template: |
Your secret word is: {my_word}
Round {round_number}.

Previous clues from all players (DO NOT repeat any of these):
{transcript}

Think of a unique angle or personal association with your word that nobody has mentioned yet. Write ONE indirect, subtle clue under 20 words.
Loading
Loading