🦞 x-tweet-fetcher

Fetch tweets, lists, articles, and WeChat content — with smart backend routing.

Three backends · Auto fallback · Works everywhere (VPS / Mac / Windows / CI / Claude Code / OpenClaw)

Quick Start · Backends · Capabilities · Agent Waystation · Self-hosted Nitter · Claude Code & CC

---

😤 Problem

You: fetch that tweet / list / article for me
AI:  I can't access X/Twitter. Please copy-paste the content manually.

You: ...seriously?

X has no free API. Scraping gets you blocked. Browser automation is fragile and won't work in headless environments.

x-tweet-fetcher solves this with smart backend routing: Nitter for zero-dependency speed, Playwright for full-feature coverage, auto fallback between them.

🔀 Three Backends

# Auto mode (default) — Nitter first, browser fallback
python3 scripts/fetch_tweet.py --user elonmusk

# Nitter only — zero dependency, no browser
python3 scripts/fetch_tweet.py --user elonmusk --backend nitter

# Browser only — full features (lists, articles)
python3 scripts/fetch_tweet.py --list 1455045069516357634 --backend browser

Backend	Deps	Speed	Features
nitter	None (stdlib only)	⚡ Fast	Timeline, search, replies, profile, mentions
browser	Playwright/Chromium	🐢 Slower	Everything above + Lists + Articles + fetch_china
auto (default)	Best available	⚡→🐢	Tries nitter first, falls back to browser

OpenClaw users: Playwright + Chromium are built-in. --backend auto just works — no extra install needed.

📊 Capabilities

Feature	Backend	Output
Single tweet	FxTwitter (always)	text, stats, media, quotes
Reply comments	nitter / browser	threaded comment list
User timeline	nitter / browser	paginated tweet list
@mentions monitor	nitter / browser	incremental new mentions
Keyword search	nitter / browser	real-time tweet stream
X Lists	browser only	list member tweets
X Articles	browser only	full long-form content
User profile analysis	nitter + LLM	MBTI, Big Five, topic graph
WeChat article search	Sogou (direct HTTP)	title, url, author, date
WeChat/Weibo/Bilibili	browser only	via fetch_china.py
Tweet growth tracker	FxTwitter API	growth curves, burst detection

For AI Agents: All output is structured JSON. Import as Python modules for direct integration. Exit codes are cron-friendly (0=nothing new, 1=new content).

🧭 Agent Waystation

x-tweet-fetcher is one of the field tools maintained from Agent Waystation — a stopover for AI agents, builders, tools, and field reports.

If you are building an agent that reads X/Twitter, start from the #22 Teahouse / 茶座. It is the active front desk for real usage stories, questions, failures, and agent field notes.

Bring one concrete scene:

• What does your agent observe?
• What does it remember?
• Where does it fail?
• What field tool does it still need?
• Which runtime, backend, shipped output, and evidence links would help another builder reproduce it?

From there, tool-specific bugs can move to x-tweet-fetcher issues, and broader agent workflow notes can become Waystation field reports.

Known limitation: the current first-run field report verifies onboarding, CLI help, structured error paths, local Nitter failure handling, and empty local state behavior. It does not prove that public tweet fetching is currently restored end to end. Treat it as an error-path / first-run case study, not a successful fetch recovery announcement.

中文：如果你也在养一个会看 X/Twitter 的 Agent，先来 #22 茶座 坐坐。贴链接、贴推文、贴使用场景、贴翻车现场都可以；茶座会把它分流成工具问题、field report，或者新的 agent 任务。

→ Join #22 Teahouse / 进入 #22 茶座

🚀 Quick Start

Single tweet (zero setup)

# Works immediately — no Nitter, no browser needed
python3 scripts/fetch_tweet.py --url https://x.com/elonmusk/status/123456789

Timeline, search, replies

# Set your Nitter instance URL (for nitter/auto mode)
export NITTER_URL=http://127.0.0.1:8788

# User timeline
python3 scripts/fetch_tweet.py --user elonmusk --limit 20

# Keyword search — real-time tweets
python3 scripts/nitter_client.py --search "AI agent"

# Tweet replies
python3 scripts/fetch_tweet.py --url https://x.com/elonmusk/status/123456789 --replies

# @mentions monitoring (cron-friendly)
python3 scripts/fetch_tweet.py --monitor @yourusername

# User profile analysis
python3 scripts/x-profile-analyzer.py --user elonmusk --count 100

Lists & Articles (browser backend)

# X List — requires Playwright
python3 scripts/fetch_tweet.py --list 1455045069516357634 --backend browser

# X Article
python3 scripts/fetch_tweet.py --article https://x.com/user/article/123 --backend browser

# WeChat / Weibo / Bilibili
python3 scripts/fetch_china.py --url "https://mp.weixin.qq.com/s/..."

WeChat search (always zero-dep)

python3 scripts/sogou_wechat.py --keyword "AI Agent" --limit 5 --json

🖥️ Works with Claude Code / CC

Since x-tweet-fetcher has zero mandatory dependencies, it works perfectly in constrained environments:

Environment	nitter mode	browser mode	Notes
Claude Code (CC)	✅	❌	No browser runtime
OpenClaw	✅	✅	Playwright built-in
VPS (headless Linux)	✅	✅*	*needs pip install playwright
Mac / Windows	✅	✅*	*needs pip install playwright
CI/CD pipelines	✅	⚠️	Possible but heavy
Docker containers	✅	⚠️	Needs Chromium in image
Termux (Android)	✅	❌	No Chromium

# In Claude Code (nitter mode, zero deps):
export NITTER_URL=http://your-vps:8788
python3 scripts/fetch_tweet.py --user YuLin807 --limit 10

# In OpenClaw (auto mode, full features):
python3 scripts/fetch_tweet.py --user YuLin807 --limit 10
# → auto-detects Nitter, falls back to Playwright if needed

🔧 Self-hosted Nitter Setup

⚠️ Public Nitter instances are dead or unreliable (as of March 2026). Self-hosting is the only reliable option.

Why you need this

Twitter removed guest API access in 2023. Public Nitter instances get rate-limited because thousands of users share a few accounts. Your own instance = your own rate limits.

5-minute setup guide

1. Install dependencies

# Ubuntu/Debian
sudo apt install -y redis-server libpcre3-dev libsass-dev

# Install Nim
curl https://nim-lang.org/choosenim/init.sh -sSf | sh
export PATH=$HOME/.nimble/bin:$PATH

2. Build Nitter

git clone https://github.com/zedeus/nitter
cd nitter
nimble build -d:release
nimble scss
cp nitter.example.conf nitter.conf

3. Get X session cookies

Use a secondary account (not your main).

1. Log into X in browser → DevTools → Application → Cookies → x.com
2. Copy auth_token and ct0
3. Create sessions.jsonl:

{"kind":"cookie","username":"myaccount","authToken":"YOUR_AUTH_TOKEN","ct0":"YOUR_CT0"}

4. Configure

[Server]
address = "127.0.0.1"  # Local only!
port = 8788

[Config]
hmacKey = "$(openssl rand -hex 32)"

[Tokens]
tokenFile = "sessions.jsonl"

5. Run & test

sudo systemctl start redis-server
./nitter

# Test
curl http://127.0.0.1:8788/YuLin807
export NITTER_URL=http://127.0.0.1:8788
python3 scripts/nitter_client.py --search "test"

Security

• Bind to 127.0.0.1 only — never expose to public internet
• Use a secondary X account — session token gives full access
• Session tokens last ~1 year

📐 How It Works

                    ┌─────────────┐
 --url              │  FxTwitter  │  ← Public API, no auth needed
                    │  (free)     │
                    └──────┬──────┘
                           │ JSON
              ┌────────────┴────────────┐
              │    --backend auto       │
              │  ┌───────┐  ┌────────┐  │       ┌──────────┐
 --user       │  │Nitter │→→│Browser │  │       │  Agent   │
 --replies    │  │(fast) │  │(full)  │  │──────▶│  (JSON)  │
 --monitor    │  │ 0 dep │  │Playwrt │  │       │          │
 --search     │  └───────┘  └────────┘  │       └──────────┘
 --list       └─────────────────────────┘
 --article
              ┌─────────────┐
 sogou_wechat │   Sogou     │  ← Direct HTTP, no API key
 fetch_china  │  (search)   │
              └─────────────┘

• Single tweets: FxTwitter — always works, zero auth
• Timeline / Replies / Search / Mentions: Self-hosted Nitter or Playwright browser
• Lists / Articles: Playwright browser (Nitter doesn't support these)
• WeChat / China platforms: Sogou search + fetch_china.py

📦 Requirements

Python 3.7+     (that's it for nitter mode)

Mode	Extra requirement
--backend nitter	Nothing (Python stdlib only)
--backend browser	pip install playwright + playwright install chromium
--backend auto	Uses whatever is available

⏰ Cron Integration

Exit codes for automation: 0=nothing new, 1=new content, 2=error.

# Check mentions every 30 min
*/30 * * * * NITTER_URL=http://127.0.0.1:8788 python3 fetch_tweet.py --monitor @username

# Discover tweets daily
0 9 * * * python3 nitter_client.py --search "AI Agent" >> ~/discoveries.jsonl

🤝 Contributing

Issues and PRs welcome! Core platforms:

• X/Twitter — Nitter + Playwright backends
• WeChat articles — Sogou search

Other platforms welcome as community PRs.

🙏 Acknowledgments

• Nitter by zedeus (12.6k ⭐) — self-hosted Twitter frontend
• FxTwitter — public API for single tweet data
• Playwright — browser automation for full-feature coverage
• OpenClaw — AI agent framework

📄 License

MIT

---

Three backends. Auto fallback. Works everywhere. 🦞

GitHub · Issues · #22 Teahouse · Agent Waystation