AeroTwin AI is a local-first terminal operations monitoring platform for airport operations teams. It provides real-time visibility into passenger flow, queue congestion, waiting time, check-in counter utilization, abnormal crowding, and operational bottlenecks without sending camera data to cloud services.
The project combines edge computer vision, real-time WebSockets, a local LLM, and an operational dependency graph in one terminal operations dashboard.
Airport operations teams currently depend on manual observation and fragmented systems. AeroTwin provides a selected-zone operational pilot that can:
- Detect passenger queues, congestion, and abnormal crowding.
- Estimate queue waiting time and check-in counter utilization.
- Stream stable real-time alerts to a terminal operations dashboard.
- Inspect downstream flight, aircraft, crew, gate, and airport relationships.
- Produce grounded, auditable bottleneck recommendations.
- Run without cloud LLMs, hosted databases, external telemetry, or managed inference APIs.
- Core logic must run offline and on-premise.
- No OpenAI API, AWS, GCP, Azure, Vercel, hosted vector DB, hosted telemetry, or external decision API is allowed for the core workflow.
- FastAPI calls a separate local
llama-servercontainer runningllama.cpp. - LLM tool selection must use JSON Schema constrained output and local schema validation before tool execution.
- Operational recommendations must cite local evidence from graph queries, mock middleware, CV events, or local fixtures.
React Ops Center
<-> FastAPI backend and WebSockets
<-> local llama-server running llama.cpp
<-> in-memory CV state store
<-> Neo4j downstream flight graph
<-> mock Sabre and AMOS downstream context
<-> Ultralytics YOLO QueueManager
<-> OBS Virtual Camera or local video source
The current accepted LLM runtime shape is:
FastAPI backend -> http://llama-server:8080/v1/chat/completions -> local GGUF
FastAPI does not load the GGUF directly. The separate llama-server container
owns model loading, GPU layers, and context-size tuning.
| Component | Role |
|---|---|
backend/ |
FastAPI orchestrator, agentic loop, WebSockets, local tool execution |
llm_engine/ |
Local llama-server runtime and GGUF model mount point |
neo4j/ |
Flight graph schema and seed data |
cv_pipeline/ |
Ultralytics YOLO QueueManager with hold-time hysteresis |
frontend/ |
React, Vite, TailwindCSS dark-mode Ops Center |
docs/ |
Harness docs, architecture, stories, decisions, and validation matrix |
The agentic core follows a deterministic loop:
airport-operations prompt
-> local llama-server schema-constrained tool selection
-> local schema validation
-> allowed local tool execution
-> tool result injection
-> agent.trace WebSocket broadcast
-> structured reflection
-> airport-operations recommendation
Initial tools:
get_operational_area_status(area_id)get_flight_context(flight_id)get_gate_congestion(gate_id)check_crew_legality(flight_id, crew_id, aircraft_family, projected_leg_minutes)check_aircraft_mel(tail_number)
get_operational_area_status is the primary P2 tool. It reads the same
aggregated queue, wait-time, counter, crowding, and risk state used by realtime
alerts. Flight, crew, and MEL tools provide downstream operational context.
The CV pipeline uses Ultralytics YOLO and ByteTrack to count and order people inside a configured check-in queue corridor. It applies hold-time hysteresis so alerts do not flicker when the count oscillates around the threshold.
Event contract:
{
"type": "cv.zone_count",
"zone_id": "JFK-T4-CHECKIN-A",
"area_id": "JFK-T4-CHECKIN-A",
"count": 31,
"alert_state": true,
"threshold": 28,
"timestamp": "2026-07-06T12:00:00Z"
}The current CV pipeline also emits cv.crowding_alert payloads for un-zoned
DBSCAN crowding observation changes. The backend/frontend handoff contract is
documented in CV Backend Handoff.
For demos, OBS Virtual Camera is the preferred input because it allows a repeatable local video scenario instead of relying on venue lighting, crowd movement, or webcam permissions.
For direct local replay, place the approved demo video at
cv_pipeline/data/demo_sources/jfk_a12_queue_demo.mp4 and set CV_SOURCE to
that path.
The CV pipeline, FastAPI contracts/state aggregation, realtime P2 risk, operational-area agent tool, deterministic scenario generator, Neo4j seed, and frontend fallback dashboard are implemented and covered by local tests. Live GGUF schema reliability and airport CV accuracy remain separate validation gates and must not be inferred from unit or mock-scenario results.
When the frontend uses VITE_OPS_WS_URL=ws://localhost:8000/ws/ops and
VITE_BACKEND_HTTP_URL=http://localhost:8000, the Investigate button calls
POST /api/demo/replay. The backend broadcasts the dependency graph, six queue
and crowding observations, counter states, deterministic risk
0 -> 5 -> 5 -> 77 -> 0 -> 0, and six agent-trace steps over the same
/ws/ops connection used by live CV. This replay does not require llama-server
and remains explicitly labelled as a deterministic demo fixture.
Current CV demo assets:
- Local YOLO weights:
cv_pipeline/models/yolo11n.pt - Ready zone config:
cv_pipeline/data/zone_configs/zones.yaml - User-provided video target:
cv_pipeline/data/demo_sources/jfk_a12_queue_demo.mp4
Multi-camera tracking remains out of scope for the current CV pilot unless the judging requirement explicitly changes.
Start with:
- Technical Implementation Guide
- Architecture
- Test Matrix
- CV Backend Handoff
- P2 Scope Alignment
- Research Evidence Summary
- STORY-001 Agentic Loop
- STORY-002 YOLO Queue Integration
- ADR 001: llama.cpp over Ollama
- ADR 002: OBS Virtual Camera for CV
- ADR 003: Deterministic Risk, LLM Explanation Only
The target deployment is a five-service Docker Compose stack:
llama-server
neo4j
fastapi-backend
yolo-cv
react-frontend
GPU passthrough is configured for llama-server and yolo-cv. The validated
4 GB demo profile reserves CUDA for YOLO with LLAMA_N_GPU_LAYERS=0,
LLAMA_CTX_SIZE=4096, and YOLO_DEVICE=cuda:0. Increase LLM offload only after
moving YOLO to CPU or confirming free VRAM with nvidia-smi.
Place the approved GGUF shards in llm_engine/models/, then run:
./setup.shFor the AI/backend demo without the optional CV service:
AEROTWIN_SKIP_CV=1 ./setup.shThe script validates Docker Compose v2, the NVIDIA driver and container
runtime, .env, and every GGUF shard. GPUs below 6 GB automatically use CPU
LLM inference and CPU YOLO unless AEROTWIN_FORCE_GPU=1 is explicitly set.
After Neo4j is seeded, setup verifies these endpoints:
| Service | URL |
|---|---|
| Ops Center | http://localhost:5173 |
| FastAPI | http://localhost:8000/health |
| Neo4j Browser | http://localhost:7474 |
| llama-server | http://localhost:8080/health |
This repo uses Harness to keep AI-agent work reviewable and evidence-driven. Before implementation work, read:
Useful Harness commands:
scripts/bin/harness-cli query matrix
scripts/bin/harness-cli query decisions
scripts/bin/harness-cli query backlog --open
scripts/bin/harness-cli query intakesAeroTwin AI is not considered complete until proof exists for:
- Deterministic JSON-schema tool calls from the local LLM path.
- Rejection of invalid or unknown tool calls before execution.
- Neo4j flight graph queries for flight context and gate metadata.
- Mock Sabre and AMOS responses with stable aviation-shaped JSON.
- YOLO queue alerts that do not flicker around threshold boundaries.
- Ops Center WebSocket playback for CV, agent trace, graph, and risk events.
- Offline execution with outbound network egress blocked.
AeroTwin AI addresses Aviation Track P2, real-time airport operations monitoring, as a practical demonstration of enterprise edge AI:
- Passenger queue, wait-time, counter-utilization, and crowding monitoring.
- Real-time terminal alerts without cloud dependency.
- Multi-modal bottleneck context from graph, middleware, and CV.
- Deterministic local tool-calling rather than free-form generation.
- A terminal-operations dashboard for selected check-in and operational zones.
The aim is not to replace certified airline systems. The aim is to show how a local AI layer can safely fuse operational signals and help dispatchers reason faster with auditable evidence.
See P2 Airport Operations Alignment for requirement traceability, pilot boundaries, and pitch-safe claims.