Media Processor

A video processing platform built with Go microservices. Upload a video, extract one frame per second, and download a ZIP with all the images.

API documentation & collection: media-processor-docs

Architecture

For a detailed architecture overview with C4 diagrams and sequence diagrams, see docs/architecture.md.

The project is organized as a Go workspace with three microservices under services/ and shared packages under pkg/.

media-processor/
├── services/
│   ├── core/           # REST API — auth, video upload/management
│   ├── processor/      # Worker — consumes queue, extracts frames
│   └── notification/   # Worker — delivers processing result notifications
├── pkg/                # Shared libraries (logger, metrics, tracer, queue, …)
├── docker/             # Config files for the observability stack
├── k6/                 # Load and smoke tests
└── docker-compose.yml  # Full local environment

Services

Service	Port	Responsibility
core	8080	Authenticates users, receives video uploads, stores metadata in Postgres, publishes jobs to SQS
processor	8081 (health)	Polls SQS, downloads video from S3, runs ffmpeg to extract frames, uploads the ZIP back to S3
notification	8082 (health)	Polls SQS, sends processing-result notifications to users

Infrastructure

Component	Port	Role
Postgres	5432	User and video metadata
Redis	6379	Rate limiting
Ministack	4566	Local AWS emulation (S3 + SQS)
Grafana	3000	Dashboards
Prometheus	9090	Metrics
Loki	3100	Logs
Tempo	3200	Traces (OTLP on 4317/4318)

Running locally

Prerequisites: Docker and Docker Compose.

# Clone and enter the repo
git clone git@github.com:FIAP-13SOAT/media-processor.git
cd media-processor

# Start everything (builds the three services on every run)
docker compose up --build

All services and infrastructure come up automatically in the correct order via depends_on health checks. The first run takes a bit longer because Docker builds the Go binaries.

Useful endpoints once running

URL	What it is
http://localhost:8080	API
http://localhost:3000	Grafana (anonymous access, no login required)
http://localhost:9090	Prometheus

Stopping

docker compose down

To also remove all persistent volumes (database, logs, traces):

docker compose down -v

Load testing (k6)

Prerequisites: k6 installed and the stack running locally.

Smoke test

Exercita todos os endpoints uma vez com um único VU. Falha se qualquer assertion não passar.

k6 run k6/smoke.js

Load test

Dois cenários em paralelo:

Cenário	VUs	Duração	O que faz
read_heavy	ramp 0→20	~2min	list + get por ID
upload_flow	5 constantes	~2min	upload + list + get

k6 run k6/load.js

Contra um ambiente remoto:

BASE_URL=https://api.staging.example.com k6 run k6/load.js

Thresholds configurados:

• http_req_duration p(95) < 500ms
• http_req_failed rate < 1%
• checks rate ≥ 98%
• upload_duration p(95) < 2s

O setup() do load test cria 30 usuários antes dos VUs iniciarem, usando round-robin para distribuí-los entre os VUs. O cenário upload_flow trata 429 (rate limit e limite diário FREE de 2 uploads/dia) como resultado aceitável durante carga.

Shared packages (pkg/)

Package	Purpose
pkg/logger	Structured JSON logger (Zap-based)
pkg/metrics	Prometheus metric helpers and standard metric names
pkg/tracer	OpenTelemetry tracer setup
pkg/queue	SQS client wrapper
pkg/bucket	S3 client wrapper
pkg/topic	SNS topic helpers
pkg/events	Shared event type definitions
pkg/env	Environment variable loading helpers