Sport Events Parser

🇷🇺 Русская версия · this is the main README

Automates an operator's weekly search for privately organized sports events — competitions, tournaments, training camps — filtering out municipal ones, extracting the event name and date with an LLM, and appending everything to a running Excel sheet ready for final human verification.

Every week, a manager used to run a set of search queries by hand, open each result, skip anything municipal/free, follow links to the private organizer's page (sometimes digging through a PDF regulation document), and copy the name and date into a spreadsheet. This project automates that whole loop end to end, leaving the manager with exactly one task: reviewing the collected candidates before they're used.

Demo

Key features

• Dual-channel search — merges results from Google News RSS (free, no key required) and Tavily /search (freshness defaults to the number of days since the previous run — so a weekly run doesn't re-scan the same date range — falling back to 7 days on the very first run; --news-days overrides this with a fixed day count, e.g. for testing on a larger sample or catching events whose news is older than a week), deduplicated by URL.
• Search log cache — links already processed in a past run are skipped without re-fetching the page or re-running the LLM on it (no TTL — an accepted tradeoff).
• LLM classification with a reason attached — a structured Anthropic Claude tool-use call decides private vs. municipal organizer, writes a short justification for the call (explicitly hedging when uncertain), extracts the organizer's name if mentioned, and flags pages that aren't actually a specific organized event (match schedules, transfer news, league tables) so they're discarded before extraction ever runs.
• Structured event extraction — event name and free-text date via a second structured tool-use call; incomplete results are kept and flagged rather than silently dropped. An event is saved regardless of how far in the future (or past) its date is — nothing is held back for a later run.
• Privacy by design — the extractor's tool schema has no fields for personal data at all. Organizer names, phone numbers, and emails are never collected — this was an explicit requirement from the outset, enforced structurally rather than by convention.
• Accumulating Excel storage — append-only .xlsx via openpyxl.
• Exact-duplicate pre-check — a new row with the same name + date as an already-stored row (from this run or a past one) is skipped before the LLM deduplication step is even invoked.
• LLM-based fuzzy deduplication — after each weekly run, every accumulated row is re-grouped by "same real event, different wording" and merged, combining all of that event's sources.
• Resilient to per-link and per-request failures — a failed fetch, classification, or extraction for one link is logged and skipped rather than aborting the whole run; transient network errors (timeouts, TLS hiccups) are retried with exponential backoff.
• Bilingual output — --lang ru|en controls the language of the search query phrasing, LLM-generated justification/organizer-name text, and the spreadsheet headers. The event name and date are never translated, to avoid distorting facts taken from the source.

How it works

Sport list + current week
        │
        ▼
   queries       — build this week's search queries per sport
        │
        ▼
   search        — Google News RSS + Tavily, merged & deduplicated
        │            (skips links already seen in past runs)
        ▼
   classify      — private vs. municipal, "is this a specific event?" (LLM)
        │            (municipal / non-events discarded here)
        ▼
   extract       — event name + date + normalized ISO date (LLM)
        │
        ▼
   save          — append to results.xlsx
        │            (date-window guard + exact-duplicate guard)
        ▼
   deduplicate   — collapse same-event rows across the whole
                    accumulated file (LLM)

The final step — reviewing the resulting spreadsheet and confirming or rejecting each row — stays a manual, human step by design.

Quick start

Requires Python ≥3.11 and uv.

uv sync
cp .env.example .env   # then fill in the values below
uv run python -m sport_events_parser

Configuration (.env)

Variable	Purpose
ANTHROPIC_API_KEY	Anthropic API key (required) — used for classification, extraction, and deduplication
TAVILY_API_KEY	Tavily API key (required) — one of the two search channels, 1,000 free searches/month

.env is git-ignored and is never committed — see .gitignore.

Running

uv run python -m sport_events_parser --limit 5 --sport football --sport volleyball --lang en
uv run python -m sport_events_parser --limit 5 --sport футболу --sport волейболу --lang ru

Flag	Purpose
--limit N	Cap the number of candidate links processed — for a fast manual check without a full run. Doesn't affect the search step itself, only trims the list before processing.
--sport <word>	Sport to search for, in the form matching --lang: English name for en (e.g. football, volleyball) or Russian dative case for ru (e.g. футболу, волейболу) — the search query templates and month/season names are translated per --lang, but the sport word itself is not translated by the code, so it must already match. Repeatable; defaults to football + volleyball (in the language of --lang) if omitted.
--lang ru|en	Output language: the search query phrasing, results.xlsx headers, and the LLM-generated justification/organizer-name text. Defaults to en. The event name and date are never translated — they're kept exactly as extracted from the source.
--news-days N	Override how many days back Tavily search results may be (via start_date), counted from today regardless of the previous run. Without this flag, freshness defaults to the number of days since the previous run (from search_log.json), falling back to 7 on the first run ever. Doesn't affect Google News RSS (no freshness parameter there) or the event date-window filter used when saving results.

Collects this week's events into results.xlsx in the current directory, alongside a local cache of processed links, search_log.json.

Running the tests

uv sync --extra dev
uv run pytest

99 unit tests cover query generation, both search channels, aggregation, classification, extraction, the date-window filter, the search log, the results table (including the header-schema guard), deduplication, and the end-to-end pipeline — all against fake LLM/network dependencies via dependency injection, so no API key or network access is needed to run the suite.

Tech stack

• Python 3.11+, uv for dependency management
• requests — HTTP client for Google News RSS, Tavily, and Jina Reader
• Anthropic API (anthropic SDK) — classification, extraction, and deduplication
• openpyxl — accumulating Excel report
• python-dotenv — .env configuration loading
• pytest — test suite

Limitations

• Only the event name and date are collected — no personal data (names, phone numbers, emails) of organizers or participants, by design.
• Only private events are in scope; municipal/free events (run by local administrations) are filtered out.
• Classification and extraction accuracy isn't guaranteed — hence the mandatory final human verification step.
• Following in-page links to PDF regulation documents isn't implemented yet — extraction currently sees only the content of the page that was found, not documents it links to.
• Google News RSS is an unofficial format with no SLA guarantee.
• Concurrent writes to results.xlsx from multiple processes aren't supported.
• The search-log cache has no TTL — if a page's content changes later, it won't be re-checked.
• results.xlsx / results.xlsx.*.bak / search_log.json aren't version-controlled (.gitignore) — they're local run artifacts, not release data.

Project structure

sport_events_parser/
  query_generator.py       — build this week's search queries
  http_retry.py            — retry-with-backoff for transient network failures
  google_news_rss.py       — search channel: Google News RSS (no key)
  tavily_client.py         — search channel: Tavily /search (freshness in days, computed by main.py)
  search_aggregator.py     — merge & deduplicate links from both channels
  search_log.py            — local JSON cache of already-processed links + last run date
  organizer_classifier.py  — private/municipal + specific-event classification (Anthropic, --lang)
  jina_reader.py           — read page/PDF content by URL
  event_data_extractor.py  — extract event name and date (Anthropic)
  results_table.py         — accumulating .xlsx writer (--lang, schema guard)
  event_deduplicator.py    — LLM-based fuzzy deduplication of events (Anthropic)
  pipeline.py              — orchestrates every step into one run
  main.py                  — entry point: .env → keys → CLI (--limit/--sport/--lang/--news-days) → pipeline
  __main__.py              — python -m sport_events_parser
tests/                     — unit tests (pytest)

License

MIT