Real-time face tracking

Real-time face tracking is a computer-vision service that detects and tracks faces in a webcam stream, in image or video files, or over an HTTP API. It runs YOLOv8-nano as the primary detector with a Haar cascade last-resort fallback, smooths detections across frames, and keeps running when a camera or GPU backend fails instead of crashing. It returns face boxes with center points and confidence scores as JSON, or draws them on a live or annotated frame. I built it with Parshav.

Demo

Real-time face detection demo — live webcam feed with bounding boxes, confidence scores, and frame-by-frame detection status:

How it works

Each frame runs YOLOv8-nano (primary); if YOLO returns no detections or is unavailable, the Haar cascade runs as a last-resort fallback. Non-maximum suppression de-duplicates overlapping boxes. A motion gate and a temporal filter follow: the temporal filter holds a candidate across a 5-frame window and only reports it once it shows up consistently, which clears out most one-frame false positives before they reach the output.

Detection picks a hardware backend at startup and falls back on failure. It tries CUDA first; if there is no NVIDIA GPU or the CUDA backend fails to initialize, it drops to OpenCL through OpenCV's T-API, which runs on Intel, AMD, or Apple GPUs; if that is unavailable it runs on CPU. A circuit-breaker and retry layer wraps the camera and detectors, so a transient failure degrades (backend fallback, frame skipping) instead of stopping the stream.

Writing tests for the camera-failure path turned up a real bug: the recovery code called ErrorHandler.handle_camera_error without an instance bound to it, so every camera failure raised an AttributeError instead of resetting the camera. The fix gave the capture loop its own ErrorHandler instance, and the test that caught it forces a camera failure and checks that recovery runs.

Features

• YOLOv8-nano primary detector with automatic fallback to Haar cascade when YOLO is unavailable or returns no detections
• 5-frame temporal filter for stable tracking and fewer one-frame false positives
• Optical-flow motion gate (optional)
• Hardware backend auto-selection: CUDA → OpenCL (T-API) → CPU
• Circuit-breaker and exponential-backoff retry around the camera and detectors
• FastAPI service: POST /detect (image upload → JSON faces) and GET /health
• Headless CLI over image and video files, plus a live webcam tracker
• Adaptive 1–5 frame skipping under load
• Bounded concurrency — slot counter capped by MAX_CONCURRENT_DETECTIONS; overflow requests receive HTTP 503 with Retry-After
• Kubernetes HPA — deploy/k8s/hpa.yaml scales on CPU and on the custom face_detection_backend_per_second metric (via prometheus-adapter)
• GCP deployment — deploy/gcp/ (Cloud Run service YAML + Cloud Build pipeline) and terraform/gcp/ provision the full stack
• 226 pytest tests at 96% line / 93% branch coverage on Python 3.10–3.12

Skills Demonstrated

Skill	Implementation
Hybrid Detection	YOLOv8-nano primary + Haar cascade fallback; NMS de-duplication
Rate Limiting & Abuse Protection	slowapi per-IP limiter, configurable RATE_LIMIT_PER_MINUTE, 429 with retry_after
Observability & Metrics	Prometheus counters via prometheus-fastapi-instrumentator, structlog JSON per request
Backpressure / Graceful Degradation	Bounded slot counter limited by MAX_CONCURRENT_DETECTIONS, 503 on overflow
Liveness Detection	mediapipe EAR blink analysis across frame sequences, opt-in via LIVENESS_CHECK_ENABLED
AI-Assisted Triage	Claude Haiku advisory notes on low-confidence detections, opt-in via ?triage=true

Rate Limiting & Abuse Protection

POST /detect enforces a per-IP rate limit using slowapi. The limit is configurable via the RATE_LIMIT_PER_MINUTE environment variable (default: 30/minute).

When the limit is exceeded, the API returns HTTP 429:

{"error": "rate_limit_exceeded", "retry_after": 60}

The Retry-After header is also set. Note that 413 (oversized upload) is a separate check unaffected by the rate limiter.

Observability & Metrics

GET /metrics exposes a Prometheus-format endpoint auto-mounted by prometheus-fastapi-instrumentator.

Two custom counters are emitted per request:

Metric	Labels	Description
face_detection_backend_total	backend	Detections grouped by compute backend (CUDA/OpenCL/CPU)
face_detection_errors_total	error_type	Errors grouped by type (oversized_upload, decode_failure, server_busy, …)

Each request also emits a structlog JSON record with request_id, backend, latency_ms, and face_count — never face coordinates or image bytes.

Note: Protect /metrics at the network or ingress layer — the application does not rate-limit this endpoint.

Backpressure / Graceful Degradation

A plain integer slot counter caps simultaneous detections at MAX_CONCURRENT_DETECTIONS (default 10). The maximum accepted upload size is MAX_UPLOAD_BYTES (default 10 MB); uploads over this limit receive HTTP 413 immediately. Requests that arrive when all slots are occupied receive HTTP 503 immediately:

{"error": "server_busy", "retry_after": 5}

The slot counter is always restored in a finally block, so a detector exception never leaks a permit.

Liveness Detection

Set LIVENESS_CHECK_ENABLED=true to enable the liveness detector. The REST /detect endpoint always returns {"checked": false, "reason": "single_frame_input"} — blink detection is inherently multi-frame and cannot be performed on a single uploaded image.

The full Eye Aspect Ratio (EAR) blink analysis runs on the webcam path (main.py) where a sequence of frames is available. EAR variance across the sequence distinguishes live blinking from a printed photograph.

Why mediapipe? It bundles its TFLite models with no native compilation step (unlike dlib, which requires CMake and Boost) and no separate model downloads (unlike OpenCV LBF). It is Apache 2.0 licensed. Install it separately to keep the core API lightweight:

pip install -r requirements-liveness.txt

AI-Assisted Triage

Add ?triage=true to a /detect request to enable Claude Haiku advisory notes on low-confidence detections. Faces below TRIAGE_CONFIDENCE_THRESHOLD (default 0.6) receive a triage_note field inside their face dict:

{"rect": [...], "center": [...], "confidence": 0.42, "triage_note": "Low light or partial occlusion may explain the reduced confidence — review the source image."}

Requires ANTHROPIC_API_KEY. If the key is unset, triage is silently disabled and all existing response fields are unaffected. Each API call uses a 30-second timeout with up to 3 retries (exponential backoff: 0.5 s, 1.0 s); worst-case blocking per face is ~91 s before triage_note is omitted. If the API is unreachable, the /detect endpoint still returns normally — triage failure is never surfaced as an HTTP error.

Running the Project

Prerequisites: Python 3.10–3.12 and OpenCV 4.x (requirements.txt pins opencv below 5 because 5.x breaks the bundled detector initialization). A CUDA or OpenCL GPU is optional; without one it runs on CPU.

Option A — HTTP API with Docker (recommended)

docker build -t face-detection-api .
docker run -p 8000:8000 face-detection-api

Test the API:

curl http://localhost:8000/health
# {"status":"ok"}

curl -X POST http://localhost:8000/detect -F "file=@your_photo.jpg"
# {"count": 1, "faces": [{"rect": [142, 88, 95, 95], "center": [189, 135], "confidence": 0.99}]}

Interactive API docs are auto-generated at http://localhost:8000/docs.

Option B — HTTP API locally

python3 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -r requirements-api.txt
uvicorn src.api:app --port 8000

Option C — Live webcam tracker

Requires a connected camera and full dependencies:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python src/main.py --width 1280 --height 720
# press Q to quit

The tracker auto-selects the fastest available backend at startup (CUDA → OpenCL → CPU) and prints which one it chose.

Option D — Headless CLI (image or video file, no server)

python src/cli_detect.py --image portrait.jpg --out result.jpg

API reference

Method	Path	Description
GET	/health	Liveness probe → {"status": "ok"}
POST	/detect	Multipart image upload → {"count": N, "faces": [{rect, center, confidence}]}
GET	/metrics	Prometheus metrics (protect at ingress — not rate-limited)

POST /detect query parameters:

Parameter	Default	Description
max_faces	10	Maximum faces to return (1–100)
triage	false	Attach Claude advisory notes to low-confidence faces (requires ANTHROPIC_API_KEY)

curl -X POST http://localhost:8000/detect -F "file=@face.jpg"
# {"count": 1, "faces": [{"rect": [120, 80, 90, 90], "center": [165, 125], "confidence": 0.99}]}

Interactive API docs are generated at http://localhost:8000/docs.

Architecture diagram

Each frame runs YOLOv8-nano first; if YOLO fails or finds nothing, the Haar cascade runs as a last-resort fallback. NMS de-duplicates boxes, an optional motion gate filters low-movement candidates, and the 5-frame temporal filter smooths detections before drawing. A circuit-breaker/retry layer wraps the camera and detectors so a transient failure degrades (CUDA → OpenCL → CPU, frame skipping) instead of crashing.

flowchart LR
    CAM[Camera / image / video] --> ACQ[VideoCapture<br/>pacing + recovery]
    ACQ --> YOLO[YOLOv8-nano<br/>primary]
    YOLO -->|faces found| NMS[NMS de-dup]
    YOLO -->|no detections / failed| HAAR[Haar cascade<br/>fallback]
    HAAR --> NMS
    NMS --> MOT[Motion gate<br/>optional]
    MOT --> TMP[Temporal filter<br/>5-frame consistency]
    TMP --> VIS[Visualizer / JSON]
    ACCEL[Acceleration<br/>CUDA→OpenCL→CPU] -.-> YOLO
    ACCEL -.-> HAAR

Loading

Tests

226 pytest tests cover the detection logic, NMS, temporal filtering, optical-flow motion analysis, the circuit breaker and recovery paths (including the camera-failure bug above), backend selection, the REST API, rate limiting, metrics, backpressure, liveness detection, AI triage (including explicit-timeout regression and graceful-retry-on-timeout), and the CLI, at 96% line and 93% branch coverage. They are hermetic: no camera or display is needed, and cv2's GUI and capture calls are mocked. CI runs them on Python 3.10, 3.11, and 3.12. Run them locally with:

python -m pytest tests/ -v

Privacy, Biometrics & Responsible Use

This is a free and open-source demonstration / trial project, not a certified commercial biometric platform.

Biometric data disclosure

Face detection processes biometric-related data (images of human faces). Depending on where you and your data subjects are located, this may be regulated by laws including:

• Illinois Biometric Information Privacy Act (BIPA)
• Texas Capture or Use of Biometric Identifier Act (CUBI)
• Washington biometric privacy law (HB 1493)
• GDPR Article 9 (special-category data) in the EU/EEA
• PIPEDA in Canada, and other equivalent regimes

You are responsible for determining which laws apply and for obtaining any legally required notice and consent before processing images of identifiable people.

Data retention

The REST API processes images entirely in memory — uploaded images are decoded, analyzed, and discarded as the response is built. No image data is stored. Every API response includes the header:

X-Data-Retention: no image data stored; processed in-memory only

The webcam tracker renders to a display window only. The headless CLI writes an annotated image only when you explicitly pass --out; otherwise it persists nothing.

Authorized & responsible use

• Use this software only with images or video you own or are authorized to process.
• Do not use it to monitor, track, identify, or surveil people unlawfully, or without legally required notice/consent.
• The /detect endpoint must only be used where the operator holds all required rights.
• Provided as-is, with no warranty and no liability for misuse.

⚖️ Legal Notice & Responsible Use

This project is free and open-source software, released under the MIT License as a demonstration / learning / trial project. It is provided "as is", without warranty of any kind, and is not an audited or certified commercial biometric product.

• Authorized use only. Use it solely with images, video, and devices that you own or are explicitly authorized to process.
• Do no harm. Do not use it to surveil, stalk, harass, invade the privacy of, or conduct unauthorized monitoring or identification of any person.
• Consent & notice. Facial detection processes biometric-related data; obtaining any legally required notice and consent is the operator's responsibility.
• Compliance is the operator's responsibility. Compliance with BIPA, CUBI, Washington HB 1493, GDPR (incl. Article 9), PIPEDA, CCPA, and equivalent laws — where applicable — rests with the operator.
• Misuse may be illegal. Unauthorized monitoring or biometric processing may violate privacy, biometric, and computer-misuse laws in your jurisdiction.

By using this software you accept responsibility for operating it lawfully. See SECURITY.md to report a vulnerability.

License

MIT — see LICENSE.