Prototype. This is a research prototype built to explore the detect-reason-act loop in autonomous cloud operations. Remediation coverage is intentionally narrow (graceful reset + container restart). See Limitations before drawing production conclusions.
ARGUS is an AIOps platform that monitors cloud microservices, detects infrastructure anomalies using ML, generates AI-powered root cause analysis, and executes autonomous remediation - without human intervention.
[microservice] -> [prometheus] -> [argus-engine] -> [remediation]
|
[dashboard / SSE]
- Architecture
- Components
- Detection Pipeline
- Root Cause Analysis
- Remediation Engine
- Dashboard & Real-Time Streaming
- Running Locally
- Fault Injection
- Configuration
- Limitations
- Future Scope
Three Docker containers, one network:
+---------------------------------------------------------+
| argus-net (bridge) |
| |
| +------------------+ +------------------------+ |
| | payment-service |---> | prometheus | |
| | FastAPI :8001 | | time-series DB | |
| | /metrics (prom) | | scrape: 5s :9090 | |
| +------------------+ +------------+-----------+ |
| | |
| +------------v-----------+ |
| | argus-engine | |
| | FastAPI :8000 | |
| | detector.py (ML) | |
| | rca.py (LLM) | |
| | remediator.py | |
| | SSE dashboard | |
| +------------------------+ |
+---------------------------------------------------------+
payment-service is a stand-in for any Prometheus-instrumented microservice. Replace it with a real service by pointing Prometheus at its /metrics endpoint.
| File | Role |
|---|---|
main.py |
Monitoring loop, incident lifecycle, SSE broadcast, REST API |
detector.py |
Isolation Forest + z-score anomaly detection per metric stream |
rca.py |
Groq LLM call -> structured JSON root cause analysis |
remediator.py |
Remediation strategies: graceful API reset, Docker socket fallback |
static/ |
Single-page dashboard (vanilla JS, Chart.js, SSE) |
FastAPI service that:
- Exposes
payment_memory_mb,payment_cpu_percent,payment_error_rateas Prometheus gauges - Simulates realistic gradual fault ramp-up via
/stress/*endpoints - Generates continuous background traffic so metrics are never flat zero
Single prometheus.yml config. Scrape interval: 5s. Evaluation interval: 5s.
On startup, ARGUS enters LEARNING state. For each watched metric, it collects 24 samples (one per 5s poll). During this window no incidents are fired regardless of values - the system has no baseline to deviate from.
Once all metric streams have 24 samples, training runs:
# Small jitter added to prevent degenerate all-zeros training data
data = np.array(history) + np.random.normal(0, max(std * 0.01, 0.01), len(history))
model = IsolationForest(n_estimators=100, contamination=0.05, random_state=42)
model.fit(data.reshape(-1, 1))Baseline mean and std are recorded at this point as the reference for z-score calculations.
Each new metric reading is scored by two independent detectors:
Isolation Forest
Scores each point by how many random splits it takes to isolate it from the training cluster. Points that require few splits are anomalous - they're already on the fringe. Returns True if predict() returns -1.
Z-Score Fallback
z = abs(value - baseline_mean) / max(baseline_std, 1e-6)
z_anomaly = z > 2.8Z-score runs because Isolation Forest is unreliable when training variance is near-zero (e.g., error_rate constant at 0.0 for the full learning window - all training points collapse to a single cluster, making isolation meaningless). The 2.8 threshold corresponds to the 99.5th percentile of a normal distribution.
A metric is flagged anomalous if either detector fires:
is_anomaly = bool(iso_anomaly or z_anomaly)Deviation calculation:
# When baseline is near-zero, % deviation is misleading (0->75 = 7500%).
# Use absolute change instead.
if abs(baseline_mean) < 1.0:
deviation = value - baseline_mean # absolute units
else:
deviation = ((value - baseline_mean) / abs(baseline_mean)) * 100 # %A single anomalous reading does not fire an incident. The anomaly must appear in 2 consecutive polls. This is a debounce mechanism - momentary metric blips (GC pauses, network jitter) reset the streak and never reach the incident threshold.
CONFIRM_CYCLES = 2
if is_anomaly:
streak[metric] += 1
else:
streak[metric] = 0
if streak[metric] >= CONFIRM_CYCLES:
# -> fire incidentBefore firing, the monitoring loop checks:
can_fire = (
anomalies # at least one confirmed
and not in_cooldown # HEAL_COOLDOWN_SEC not elapsed
and not _healing_active # not mid-heal from prior incident
and system_status not in ("HEALING", "AWAITING_APPROVAL", "LEARNING")
)_healing_active is set to True synchronously - before the first await in the anomaly handling block - to prevent re-entry during the async RCA call (which takes 2-5s). This was the hardest correctness problem: asyncio.to_thread is an await point, meaning the event loop can run another monitoring cycle during RCA generation. Setting the gate after the await was too late.
The worst anomaly by deviation magnitude is selected as the primary incident metric. All co-occurring anomalies are passed to the RCA engine as secondary context.
ARGUS calls llama-3.3-70b-versatile via the Groq API with a structured prompt:
ANOMALY DETECTED
Service: payment-service
Metric: Error Rate (%)
Current: 74.20
Baseline: 0.00
Deviation: +74.20 units
Trend: rising (+12.4/cycle)
Last 10: [0.0, 0.0, 1.2, 3.8, 9.1, 18.4, 31.2, 51.0, 66.3, 74.2]
CO-OCCURRING ANOMALIES:
- Memory (MB): 89.4 (+31.2%)
- CPU (%): 67.1 (+58.3%)
Response schema (enforced via prompt):
{
"severity": "CRITICAL | HIGH | MEDIUM | LOW",
"root_cause": "string",
"technical_detail": "string",
"recommended_action": "string",
"time_to_failure": "string",
"confidence": 0-100
}- Timeout: 12 seconds
- Fallback: template-based response from raw metric values if Groq fails or returns unparseable JSON
- The RCA call runs in
asyncio.to_thread- it does not block the monitoring loop
Two strategies, tried in order:
Strategy 1 - Graceful API Reset (primary)
async with httpx.AsyncClient(timeout=6.0) as client:
resp = await client.post(f"{PAYMENT_SERVICE_URL}/stress/reset")Calls the service's own recovery endpoint. Zero downtime. Equivalent to triggering a service's built-in fault-clearing logic. Preferred for demo and for services that implement a health reset API.
Strategy 2 - Docker Socket Restart (fallback)
import docker
dc = docker.from_env()
container = dc.containers.get(f"argus-{service}")
container.restart(timeout=10)Used when the service is completely unresponsive to HTTP. Requires the Docker socket to be mounted into the argus-engine container (/var/run/docker.sock).
Post-remediation:
_last_heal_atis set immediately after remediation returns (same event loop iteration)HEAL_COOLDOWN_SEC = 40- prevents re-triggering on residual metric values before the service stabilizes- A background coroutine flips status to
HEALTHYafterHEALTHY_DELAY_SEC = 20 - MTTR is calculated as
now - incident["timestamp"]and stored on the incident record
Operating modes:
- Auto-heal ON: full loop runs unattended. Remediation executes immediately after RCA.
- Auto-heal OFF (human-in-the-loop): ARGUS pauses at
AWAITING_APPROVAL. Engineer reviews the RCA card on the dashboard and clicks Execute. Same remediation code path runs.
Transport: Server-Sent Events (SSE)
GET /events -> text/event-stream -> persistent connection
The server pushes JSON events on every state change:
event types:
status - system badge, KPI values, metrics_history
incident - new incident created
rca - RCA result attached to incident
remediated - incident resolved, MTTR recorded
heartbeat - keepalive every 15s
Polling fallback: syncStatus() polls /api/status every 4 seconds. If the SSE connection drops silently (common with proxies and some browsers), the dashboard stays current via polling. Chart history is re-populated from metrics_history in the status response on SSE reconnect.
Chart: Chart.js, 40-point rolling window, one dataset per metric. Data pre-populated from metrics_history on page load so the chart isn't empty when you open the tab mid-session.
Incident deduplication: Frontend tracks _lastSeenIncidentId. Incidents already rendered are not re-rendered on subsequent status polls.
Prerequisites: Docker, Docker Compose, a Groq API key (free tier at console.groq.com).
# 1. Clone and enter
git clone <repo>
cd argus
# 2. Set your Groq API key
echo "GROQ_API_KEY=gsk_..." > .env
# 3. Build and start
docker compose up -d --build
# 4. Open dashboard
# http://localhost:8000
# Wait ~2 minutes for baseline learning to complete.
# Dashboard transitions from LEARNING BASELINE -> SYSTEM HEALTHY automatically.# From the project root (requires Python 3, no dependencies)
python fault_injector.py memory # memory leak: +1 MB/s, ramps until reset
python fault_injector.py cpu # CPU ramp: 0->65-85% over ~40s
python fault_injector.py errors # error rate ramp: 0->70-80% over ~40s
python fault_injector.py all # all three simultaneously
python fault_injector.py reset # clear all faults, restore baselineFrom another machine on the same network:
# Edit fault_injector.py line 19:
BASE = "http://<host-machine-ip>:8001"
python fault_injector.py errorsPort 8001 (payment-service) is exposed to the host. Port 8000 (dashboard) is also accessible remotely.
Fault ramp behavior: Faults do not spike instantly. Error rate and CPU load increase in random steps over ~40 seconds, simulating realistic incident escalation. Memory leaks at a constant +1 MB/s by design (that's how real leaks behave).
All tuneable constants are at the top of argus_engine/main.py:
| Constant | Default | Effect |
|---|---|---|
POLL_INTERVAL |
5 |
Seconds between Prometheus queries |
CONFIRM_CYCLES |
2 |
Consecutive anomalous cycles before incident fires |
HEAL_COOLDOWN_SEC |
40 |
Minimum seconds between incidents on same service |
HEALTHY_DELAY_SEC |
20 |
Seconds after remediation before status -> HEALTHY |
WINDOW_SIZE |
120 |
Max metric history retained per stream |
TRAINING_SAMPLES |
24 |
Samples required before baseline is established |
In argus_engine/detector.py:
| Constant | Default | Effect |
|---|---|---|
Z_SCORE_THRESHOLD |
2.8 |
Standard deviations above mean to flag via z-score |
This is a prototype. The following are known constraints, not oversights:
Remediation coverage is narrow. ARGUS remediates via service reset or container restart. Real production incidents require database query optimization, cache invalidation, circuit breaker tuning, CDN purge, auto-scaling, dependency rerouting, and more. Container restart covers a small fraction of real failure modes.
Cold-start baseline problem. If the monitored service is already misbehaving when ARGUS starts, the anomalous state becomes the learned baseline. Mitigation in production: seed the baseline from historical metric data rather than live observation.
In-memory state. Incidents, metrics history, and trained models live in process memory. An argus-engine restart loses all state. Production deployment requires persistent storage (PostgreSQL for incidents, model serialization or retraining from stored metric history).
Single-service scope. The prototype monitors one service. The detector architecture is per-stream and scales horizontally, but cross-service correlation (service A's latency spike causing service B's error rate to rise) is not implemented.
LLM dependency. RCA quality depends on Groq API availability. The fallback is mechanical and less useful. Air-gapped environments require a locally-hosted model (Ollama, vLLM).
No authentication. The dashboard and API have no auth. The approve endpoint accepts any request. Do not expose port 8000 to untrusted networks.
The detect-reason-act loop is the architecturally hard part - it exists and works. Everything below is engineering time given a stable foundation:
Short term
- Persistent storage (PostgreSQL) for incidents and metric history
- Multi-service monitoring - add services to
prometheus.yml, ARGUS picks them up automatically - Slack/PagerDuty webhook on incident creation (even in auto-heal mode, notify the team)
- Model persistence - serialize trained Isolation Forest models to disk, skip cold-start on restart
Medium term
- Kubernetes integration - replace Docker socket with
kubectl rollout restart, pod autoscaling, deployment rollbacks - OpenTelemetry distributed tracing - correlate anomalies across service call chains, not just individual metric streams
- Feedback loop - when an engineer overrides a remediation recommendation, log the outcome and adjust the detector's contamination threshold
- RBAC - team-scoped approval permissions
Long term
- Full remediation playbook library (database, cache, CDN, circuit breaker operations)
- Multi-cloud metric ingestion via OTLP (AWS CloudWatch, Azure Monitor, GCP Cloud Monitoring)
- Anomaly-based security detection - the same pipeline that catches memory leaks can flag unusual authentication patterns and API abuse
- Carbon-aware scheduling - route remediation restarts to low-carbon-intensity regions when possible (Green AIOps)
| Layer | Technology |
|---|---|
| Microservice framework | FastAPI (Python) |
| Metrics collection | Prometheus |
| Anomaly detection | scikit-learn Isolation Forest + z-score |
| LLM / RCA | Groq API - llama-3.3-70b-versatile |
| Remediation fallback | Docker SDK (socket) |
| Real-time transport | Server-Sent Events (SSE) |
| Dashboard | Vanilla JS, Chart.js |
| Orchestration | Docker Compose |