Formation Web Analytics Collector

Minimal self-hosted analytics collector in Go with a separate TypeScript client library.

Introduction

Formation Web Analytics Collector is currently an early access release. It is already useful for teams that want to own their analytics pipeline, but it should still be treated as infrastructure for technical adopters rather than a finished product. Expect rough edges, limited ergonomics, and breaking changes while the project matures.

The main way to send data to this collector is the Formation Web Analytics Client. That client library is the intended integration path for browser-based event collection and should be considered part of the recommended setup for this server. The client repository was renamed, but the published npm package is currently still @tryformation/formation-web-analytics-client.

If you decide to use this today, assess the operational and compliance risks carefully. There is no polished UI yet, so setup, validation, and analysis require hands-on work with your own infrastructure and Elasticsearch. You should review your deployment, retention settings, access controls, backups, incident handling, cookie and consent requirements, and privacy notices before collecting production traffic. The mitigation is straightforward: start with a limited rollout, keep your configuration conservative, test your data handling end to end, and make sure your legal and security review matches your specific use case.

The main reason to adopt this approach is control. This project is intended for self-hosting web analytics in the EU in a way that can reduce third-country transfer exposure and support a GDPR-aligned deployment when configured and operated correctly. Instead of sending visitor data to foreign advertising or tracking companies, you keep collection, storage, access, and governance under your own control. That reduces third-party exposure, simplifies data residency choices, and lets you decide exactly what is collected, retained, and shared.

There is no user interface yet. Deployers currently use Elasticsearch tooling or their own reporting layer to inspect the collected data.

The published container setup is designed around compliant GeoIP distribution. The Docker image does not need to ship a MaxMind database. Instead, the Compose stack expects you to provide your own MaxMind account ID and license key, downloads GeoLite2-City.mmdb into a mounted volume at runtime, and periodically refreshes that database in place.

Backend

The collector exposes:

• GET /openapi.json
• POST /collect
• POST /batch
• GET /healthz
• GET /readyz

Events are validated, enriched with request metadata, parsed user-agent metadata, parsed language metadata, optional client timezone metadata, and local GeoIP metadata, queued in memory, and flushed to an Elasticsearch data stream via the Bulk API. Validation and abuse guard rails include request body limits, JSON-only ingest, bounded batch sizes, field-length limits, and payload depth/entry limits. The default in-memory queue size is 10_000 events and the default maximum bulk batch size is 500 events. Metrics are disabled by default; when enabled, GET /metrics is served on a separate listener configured with METRICS_LISTEN_ADDR.

See the API guide for request examples, event fields, admission semantics, error behavior, CORS, health checks, and metrics. The canonical machine-readable contract is api/openapi.json, and each running collector serves the same OpenAPI 3.1 document from GET /openapi.json.

Local development

1. Copy .env.example to .env and set Elasticsearch credentials.
2. Get your own MaxMind account ID and license key, then set MAXMIND_ACCOUNT_ID and MAXMIND_LICENSE_KEY.
3. For non-Docker runs, download/update GeoLite2-City.mmdb locally with geoipupdate and point GEOIP_DB_PATH at it.
4. Start local Elasticsearch with docker compose -f docker-compose.elasticsearch.yml up -d.
5. Create the default web-analytics data stream, ILM policy, and templates with ./scripts/create-data-stream-and-templates.sh. Use --data-stream-name your-name if you want a different data stream.
6. Run make test-backend.
7. Run go run ./cmd/collector.
8. Or start the deployment stack with docker compose up --build.
9. Run make smoke-test for a local collector-to-Elasticsearch verification.
10. Run make smoke-test-browser-client for an end-to-end check using the Formation Web Analytics Client in a Docker-managed test container.

Sending analytics requests

Use the Formation Web Analytics Client for browser applications when possible. For direct HTTP and server-side integrations:

• Send JSON with Content-Type: application/json.
• Use the configured site_id and send an Origin allowed for that site. Browsers set Origin automatically. Server-side clients must set it explicitly.
• Set a stable, descriptive User-Agent, such as Formation-Web-Analytics-Python/0.2. Avoid runtime defaults such as Python-urllib/3.x; CDNs and proxies may reject them before the request reaches the collector.
• Keep the event url host equal to the Origin host when REQUIRE_URL_HOST_MATCH=true.
• Keep batches within MAX_EVENTS_PER_REQUEST. Retry 429 and 503 responses with backoff. Fix the request or deployment configuration before retrying 400 and 403 responses.
• If a CDN or web application firewall fronts the collector, exempt /collect and /batch from browser-integrity checks designed for HTML pages. Keep rate limits, request-size limits, and payload validation enabled.

The collector accepts requests with missing or configured automated user agents. It records those events with traffic_quality=suspect and adds a user_agent:* reason to suspicion_reasons.

GeoIP updates

Docker Compose now includes a geoipupdate service based on MaxMind's official container image. It downloads GeoLite2-City.mmdb into a shared Docker volume, and the collector waits for that database before starting.

Use your own MaxMind credentials here. The values in .env.example are placeholders only, and you should never commit real MAXMIND_LICENSE_KEY values or any populated local env files to your repository.

This is also the recommended distribution model for public container releases: publish the collector image without bundling the MaxMind database, mount a persistent volume for GeoIP data, and let geoipupdate fetch and periodically refresh the database on the user's deployment using their own credentials.

Attribution: This product includes GeoLite Data created by MaxMind, available from maxmind.com.

Relevant deployment variables:

• ALLOWED_DOMAINS must include the collector request host and every browser Origin host that may send events
• SITE_ORIGIN_MAP optional per-site origin allowlist in the form marketing:tryformation.com|www.tryformation.com;docs:docs.tryformation.com
• CADDY_DOMAINS lists the public collector hostnames served by Caddy; default example analytics.tryformation.com
• CADDY_RATE_LIMIT_EVENTS default 120
• CADDY_RATE_LIMIT_WINDOW default 1m
• MAXMIND_ACCOUNT_ID
• MAXMIND_LICENSE_KEY
• GEOIPUPDATE_EDITION_IDS default GeoLite2-City
• GEOIPUPDATE_FREQUENCY in hours; 0 means run once and exit
• GEOIP_DB_PATH default /data/GeoLite2-City.mmdb
• GEOIP_WAIT_TIMEOUT collector startup wait timeout in seconds
• STORE_IP_METADATA default false
• SANITIZE_URLS default true
• REQUIRE_ORIGIN default true
• REQUIRE_URL_HOST_MATCH default true
• RATE_LIMIT_PER_MINUTE default 300
• RATE_LIMIT_MAX_CLIENTS default 100000; new client identities are rejected when the in-memory limiter reaches this bound
• BLOCKED_USER_AGENTS legacy-compatible list of user-agent substrings marked as suspect; default bot,crawler,spider,curl,wget,python-requests,python-urllib,go-http-client
• SUSPECT_USER_AGENTS additional user-agent substrings marked as suspect; default headless,playwright,puppeteer,selenium,phantomjs

If your environment uses egress controls, allow HTTPS redirects to:

• mm-prod-geoip-databases.a2649acb697e2c09b632799562c076f2.r2.cloudflarestorage.com

Docker Hub publishing

This repository includes a GitHub Actions workflow at .github/workflows/docker-publish.yml that publishes the server container to Docker Hub when a new Git tag is pushed.

The workflow currently pushes:

• tryformation/formation-web-analytics-server:<tag>

Required GitHub repository secrets:

• DOCKERHUB_USERNAME: the Docker Hub user or service account name
• DOCKERHUB_PASSWORD: the Docker Hub password or access token for that account

The Docker Hub account behind that token must have permission to push to the tryformation organization repository. Branch CI runs the Go tests, checks the Elasticsearch bootstrap, builds the server and Caddy images, and validates the generated Caddy configuration. Build the versioned server image locally as a final check before creating a tag.

Example release flow:

VERSION=vX.Y.Z
git status --short --branch
make test-backend
docker build --build-arg "VERSION=$VERSION" --file server/Dockerfile .
git tag --annotate "$VERSION" --message "$VERSION"
git push origin "$VERSION"

The tag push embeds $VERSION in the collector binary and publishes tryformation/formation-web-analytics-server:$VERSION. Watch the publish-server-image workflow through completion and verify that Docker Hub exposes the new tag before announcing the release.

The image workflow does not create a GitHub Release. Create one separately after the image has published if the version should appear on the repository's Releases page:

gh release create "$VERSION" --verify-tag --generate-notes

The published image does not bundle a MaxMind database. You should keep geoipupdate as a separate runtime sidecar or companion job that downloads GeoLite2-City.mmdb with your own MaxMind credentials and mounts it into the collector container.

Pulling the published image

Pull a tagged release from Docker Hub. Replace <tag> with an explicit version such as v0.2.1:

docker pull tryformation/formation-web-analytics-server:<tag>

The workflow also updates the floating latest tag. Pin an explicit version in production so a later release cannot change the deployed image unexpectedly.

The collector expects a GeoIP database file to be mounted at /data/GeoLite2-City.mmdb by default. A minimal direct run looks like this:

docker run --rm -p 8080:8080 \
  --env-file .env \
  -v /path/to/geoip:/data:ro \
  tryformation/formation-web-analytics-server:<tag>

For production deployments, prefer running the collector together with MaxMind's ghcr.io/maxmind/geoipupdate:latest container, as shown in docker-compose.yml, so the database is fetched and refreshed separately from the collector image.

Test Elasticsearch

Start a local Elasticsearch 9 node with:

docker compose -f docker-compose.elasticsearch.yml up -d

The local test cluster is exposed on http://localhost:19920.

Provision the default web-analytics data stream, ILM policy, and templates with:

./scripts/create-data-stream-and-templates.sh

Or specify a different data stream name:

./scripts/create-data-stream-and-templates.sh --data-stream-name your-data-stream

The script creates:

• data stream <name> where the default is web-analytics
• ILM policy <name>-ilm-policy
• component templates <name>-template-settings and <name>-template-mappings
• index template <name>-template

The mappings are tuned for this collector's analytics event shape: fixed top-level dimensions as keywords/dates/IP or wildcard fields, and a payload field stored as flattened for arbitrary event properties without unbounded mapping growth.

Guard rails

• Contributor workflow and test/run expectations: CONTRIBUTING.md
• Done checklist and current reassessment: docs/definition-of-done.md
• Privacy, consent, and compliance notes for deployers: docs/privacy-and-compliance.md
• Third-party licenses and notices: THIRD_PARTY_NOTICES.md

Validation limits

• Content-Type must be application/json when present.
• Requests larger than MAX_PAYLOAD_BYTES are rejected.
• Batches larger than MAX_EVENTS_PER_REQUEST are rejected.
• Core string fields are bounded by MAX_FIELD_LENGTH (default 10240, about 10 KB).
• payload is bounded by MAX_PAYLOAD_ENTRIES and MAX_PAYLOAD_DEPTH.
• Unknown top-level JSON fields are rejected.
• GEOIP_DB_PATH is required; ingest startup fails without a local database.

Abuse controls

• Requests without an Origin header are rejected by default.
• Requests with missing, automated, or browser-automation user agents are accepted but marked as suspect.
• Requests are rate limited per client IP in-memory with RATE_LIMIT_PER_MINUTE.
• Caddy applies edge rate limiting before the request reaches the collector.
• SITE_ORIGIN_MAP can bind each site_id to an explicit set of allowed origins.
• Event url hosts must match the request Origin host by default.
• Query strings and fragments are stripped from url, referrer, referer_header, and path by default.

Production Example

For your production setup, keep analytics.tryformation.com as the collector host and bind each site to a canonical site_id, for example:

ALLOWED_DOMAINS=analytics.tryformation.com,open-rtls.com,formation-xyz.com,tryformation.com
SITE_ORIGIN_MAP=open-rtls:open-rtls.com;formation-xyz:formation-xyz.com;tryformation:tryformation.com

This keeps site_id stable per property and prevents one allowed origin from writing traffic into another site's bucket.

Notes

• The collector is intentionally lossy under pressure or prolonged Elasticsearch outages.
• Graceful shutdown stops HTTP admission and attempts to drain the in-memory queue for up to 10 seconds.
• The collector checks for a replaced GeoIP database once per day at midnight UTC and reloads it without stopping ingest.
• The backend enforces CORS. Caddy proxies preflight and ingest requests to it.
• /metrics is intentionally not exposed through Caddy.
• Raw IP storage is disabled unless CAPTURE_CLIENT_IP=true.
• GeoIP enrichment still works without storing raw IP metadata because the lookup happens before indexing.