ChessVision

A computer vision application that locates a chessboard in an image, video, or live webcam feed, identifies every piece using a fine-tuned YOLO11n model, and displays the board state as a text grid and FEN string.

FEN is the abbreviation of Forsyth-Edwards Notation, and it is the standard notation to describe positions of a chess game in a single line e.g. 1B1B2K1-1B6-5N2-6k1-8-8-8-4nq2

---

Requirements

Python 3.10 or higher. Install dependencies with:

pip install -r requirements.txt

The training datasets used are available on Kaggle:

• Chess Detection (81 real photos, Pascal VOC XML annotations) — kaggle.com/datasets/tannergi/chess-piece-detection
• Chess Positions (100k digital renders, FEN-encoded filenames) — kaggle.com/datasets/koryakinp/chess-positions

---

Running the App

cd ChessVision/src
streamlit run main.py

Open the browser tab that appears (usually http://localhost:8501).

To stop the server:

python stop.py

---

Input Modes

Mode	Description
Image	Upload a JPEG / PNG / WebP chess photo.
Video	Upload an MP4 / AVI / MOV file — every 5th frame is processed.
Webcam (live)	Live stream with real-time piece detection, ByteTrack tracking, and MOG2 motion highlighting.
Webcam snapshot	Single-frame capture from your webcam.

---

Fine-tuning the Model

To re-train on additional data:

cd ChessVision/src
python train.py

The script builds a YOLO-format dataset from the raw data folders, runs transfer learning from best.pt, and saves the result to models/best_finetuned.pt.

---

Project Structure

ChessVision/
├── src/
│   ├── main.py          — Streamlit UI
│   ├── chess_vision.py  — Core vision pipeline (7 steps)
│   ├── utils.py         — Drawing and FEN helpers
│   ├── config.py        — Detection thresholds and model paths
│   ├── train.py         — YOLO fine-tuning script
│   └── stop.py          — Stop the running server
├── models/
│   ├── best.pt           — Pre-trained chess YOLO model (Apache-2.0)
│   └── best_finetuned.pt — Fine-tuned model (generated by train.py)
├── data/
│   ├── Chess Detection/  — 81 real photos with Pascal VOC annotations
│   └── Chess Positions/  — 300 digital renders with FEN filenames
├── samples/              — Example images and video for testing
├── runs/                 — Training logs and weight checkpoints (generated)
├── training_data/        — YOLO-format dataset (generated by train.py)
├── .gitignore
├── requirements.txt
├── README.md
└── report.md             — Technical design report

---

Pipeline

load_input()         ← file bytes, path, or numpy array
      ↓
detect_board()       ← CLAHE → Canny edges → contours → checkerboard validation
      ↓
rectify_board()      ← perspective warp to 640 × 640
      ↓
detect_pieces()      ← YOLO11n inference + colour verification
      ↓
create_board_state() ← map each bounding box to an 8 × 8 grid cell
      ↓
draw_results()       ← annotate frame with board outline, grid, and piece labels

---

Model

Architecture: YOLO11n (Nano, ~2.6 M parameters)
Base weights source: yamero999/chess-piece-detection-yolo11n — Apache-2.0
Classes (12): white/black × pawn, knight, bishop, rook, queen, king

---

Known Limitations

• Board detection relies on a clear rectangular outline. Very angled or heavily occluded boards may not be found.
• Piece colour heuristics work best with normal lighting. Extreme contrast (e.g. fluorescent backlit screens) may cause misclassifications.
• The FEN output covers piece placement only — castling rights and en-passant are not tracked.