Features/UI restructured

.claude/skills/datapipe-examples/SKILL.md

@@ -18,7 +18,7 @@ Each `examples/*` pipeline has a dedicated setup skill — pick by what it does:
 | `e2e_template/image_detection` | YOLO detection + Label Studio human-in-the-loop → train → FiftyOne | **setup-e2e-template** |
 | `e2e_template/image_keypoints` | YOLO-pose keypoints + Label Studio → train → FiftyOne | **setup-e2e-template** |
 | `sam_cvat` | SAM3 text-prompt boxes+masks → CVAT pre-annotations | **setup-sam-cvat** |
-| `detection_tags` | YOLO detection + **tags** (per-scenario metrics), no Label Studio / FiftyOne, GT injected | **setup-detection-tags** |
+| `detection_tags` | YOLO detection + **tags** (per-scenario metrics), injected GT, FiftyOne A/B view | **setup-detection-tags** |
 | `ocr` | Multi-LLM structured OCR (OpenAI/Gemini/Qwen) on passport/id-doc images → FiftyOne | **setup-ocr** |
 
 ## Ask first — don't assume (only the unresolved)
@@ -38,8 +38,8 @@ useful results. The per-example skill lists the exact knobs.
 
 ## Universal prerequisites (every example)
 1. **PostgreSQL** at `DB_URL`. embedder + sam_cvat + ocr need an EXTERNAL Postgres (they don't start one);
-   e2e_template / detection_tags bundle Postgres in `docker compose`. Empty DB is fine; tables auto-create via
-   `datapipe db create-all`. `.env.example` default `...postgres:postgres@localhost:5432/postgres`.
+   e2e_template / detection_tags bundle Postgres (+ ClickHouse for ops run logs) in `docker compose`. Empty DB is fine; tables auto-create via
+   `datapipe db create-all`. `.env.example` sets `DB_URL` and `CLICKHOUSE_RUN_LOGS_URL` (required for `datapipe --pipeline app api`).
    External quick start: `docker run -d -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres -e POSTGRES_DB=postgres -p 5432:5432 postgres:16`
    (k8s pod / no docker → conda postgres, `initdb` as non-root with `--locale=C`.)
 2. **A GPU big enough** (`nvidia-smi`): DINOv2/YOLO fit ~8 GB; SAM3 wants >8 GB (OOM'd on an 8 GB Pascal in our tests).
@@ -48,6 +48,11 @@ useful results. The per-example skill lists the exact knobs.
    needs an external CVAT you provide (URL + creds + a project whose labels match its config).
 4. **`uv` + Python ≥3.10,<3.13.** Each example has a `pyproject.toml` → `uv sync` (cu124 torch pinned,
    CUDA OOTB; no manual venv/pip). Read-only/small `$HOME`: `export UV_CACHE_DIR=/tmp/uvcache HF_HOME=/tmp/hf`.
+   **ML UI / ops specs:** examples with `datapipe-app` + metrics/training panels also pull
+   **`datapipe-app-ml-ops`** (ops specs, image records, observability routes — not `datapipe-ml`).
+   Pipeline code imports `datapipe_app_ml_ops.ops_specs`; the observability plugin entry point lives there too.
+   **Legacy CPU (pre-AVX2, e.g. epoch8 gpu5):** on `e2e_template` / `detection_tags` only —
+   `uv sync --extra old-cpu` then force `polars-lts-cpu` (see those skills' Troubleshooting).
 5. **Human-in-the-loop** (e2e_template, sam_cvat): a human reviews and marks tasks completed before
    the pipeline advances.
 

.claude/skills/setup-e2e-template/SKILL.md

@@ -1,75 +1,110 @@
 ---
 name: setup-e2e-template
 description: >
-  Use when setting up or running examples/e2e_template (image_detection or image_keypoints) —
-  datapipe YOLO/Label Studio pipelines — or debugging their install, stages, training, or FiftyOne output.
+  Use when setting up or running examples/e2e_template (image_detection, image_keypoints, or
+  image_classification) — datapipe YOLO/TF + Label Studio pipelines — or debugging their install,
+  stages, training, or FiftyOne output.
 ---
 
-# e2e_template (detection + keypoints YOLO/Label-Studio pipeline)
+# e2e_template (detection + keypoints + classification)
 
-This skill = run the YOLO/Label-Studio detection/keypoints pipeline on YOUR images. The COCO seed sample is just a smoke-test; the real goal is your own images — set the knobs below first.
+This skill = run the Label-Studio e2e pipelines on YOUR images. The COCO seed sample is just a
+smoke-test; the real goal is your own images — set the knobs below first.
 
-**Ask first — don't assume (only the unresolved):** demo or your own data? bring up the bundled `docker compose` services or reuse existing? **which Postgres + which database** for `DB_URL` — never point it at an existing DB without confirming; reuse an existing venv / `uv` env or create a fresh one? GPU? **annotate for real in Label Studio, or inject ready-made ground truth** (reuse existing labels / a labelled dataset) to skip the human `annotation` step? surface stage logs or run quiet? (per-tag scenario metrics live in a separate example → **setup-detection-tags**)
+Templates:
+- `image_detection` — YOLO bbox (Cat/Dog)
+- `image_keypoints` — YOLO pose (Person)
+- `image_classification` — TF classification (`Has Animal` / `No Animals`)
 
-**How to work:** read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with its logs shown and, after each, say what you did and what changed — don't run the pipeline silently. If a stage fails and the cause isn't clear from the normal logs, re-run it with `datapipe --debug … run` (or `--debug-sql` for SQL errors); debug is very verbose, so send it to a file and `grep` it (e.g. `datapipe --debug run > /tmp/dp_debug.log 2>&1; grep -nEi "error|traceback" /tmp/dp_debug.log`) rather than dumping it inline.
+**Ask first — don't assume (only the unresolved):** demo or your own data? which template?
+bring up the bundled `docker compose` services or reuse existing? **which Postgres + which
+database** for `DB_URL` — never point it at an existing DB without confirming; reuse an existing
+venv / `uv` env or create a fresh one? GPU? **annotate for real in Label Studio, or inject
+ready-made ground truth** (reuse existing labels / a labelled dataset) to skip the human
+`annotation` step? surface stage logs or run quiet? (per-tag scenario metrics live in a
+separate example → **setup-detection-tags**)
+
+**How to work:** read the setup, then propose a short plan and get a go-ahead before touching
+anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage
+with its logs shown and, after each, say what you did and what changed — don't run the pipeline
+silently. If a stage fails and the cause isn't clear from the normal logs, re-run it with
+`datapipe --debug … run` (or `--debug-sql` for SQL errors); debug is very verbose, so send it to
+a file and `grep` it (e.g. `datapipe --debug run > /tmp/dp_debug.log 2>&1; grep -nEi
+"error|traceback" /tmp/dp_debug.log`) rather than dumping it inline.
 
 ## Run on YOUR data
-- **Align your class everywhere** (mismatch → 0 useful results): `LABEL_CONFIG` names == `CLASSES_TO_KEEP`. Detection also `COCO_CLASSES`/`DETECTION_MODEL_CONFIG`; keypoints also `KEYPOINTS_LABELS` (order matters), `COCO_PERSON_KEYPOINT_FLIP_IDX`, `KEYPOINTS_MODEL_CONFIG`.
-- **Where YOUR images go:** put them under `$DATAPIPE_E2E_DIR/images` (`DATAPIPE_E2E_DIR` defaults to `s3://datapipe-e2e`; the pipeline writes its own artifacts under `$DATAPIPE_E2E_DIR/datapipe`, a sibling — input and working dir don't overlap). Set `AWS_*`, `S3_ENDPOINT_URL`, and `S3_PUBLIC_URL` (browser-reachable — Label Studio loads images from it). Label Studio reaches S3 via its own `LABEL_STUDIO_S3_ENDPOINT_URL` (`minio:9000`), not `S3_ENDPOINT_URL`.
+- **Align your class everywhere** (mismatch → 0 useful results): `LABEL_CONFIG` names == class
+  constants. Detection: `CLASSES_TO_KEEP` + `COCO_CLASSES`/`DETECTION_MODEL_CONFIG`. Keypoints:
+  `KEYPOINTS_LABELS` (order matters), `COCO_PERSON_KEYPOINT_FLIP_IDX`, `KEYPOINTS_MODEL_CONFIG`.
+  Classification: `CLASSIFICATION_CLASSES` = `Has Animal` / `No Animals` (Choices, not rectangles).
+- **Where YOUR images go:** put them under `$DATAPIPE_E2E_DIR/images` (`DATAPIPE_E2E_DIR` defaults
+  to `s3://datapipe-e2e`; the pipeline writes its own artifacts under `$DATAPIPE_E2E_DIR/datapipe`,
+  a sibling — input and working dir don't overlap). Set `AWS_*`, `S3_ENDPOINT_URL`, and
+  `S3_PUBLIC_URL` (browser-reachable — Label Studio loads images from it). Label Studio reaches S3
+  via its own `LABEL_STUDIO_S3_ENDPOINT_URL` (`minio:9000`), not `S3_ENDPOINT_URL`.
 
 ## Prerequisites
-- **Python 3.10–3.12** (hard pin `>=3.10,<3.13`). **Install:** `cd examples/e2e_template && uv sync`
-  (`pyproject.toml` declares editable workspace libs + cu124 `torch==2.6.0`).
+- **Python 3.10–3.12** (hard pin `>=3.10,<3.13`). **Install:** `cd examples/e2e_template && uv sync --extra ray`
+  (`pyproject.toml`: editable `datapipe-app`, **`datapipe-app-ml-ops`**, `datapipe-ml[torch,fiftyone,tensorflow]`,
+  cu124 `torch==2.6.0`).
+  Modern hosts: run `uv sync --extra ray` unmodified — do NOT edit/re-lock deps (drifts training across machines).
+  **Legacy host (pre-AVX2 CPU, e.g. epoch8 gpu5):** `uv sync --extra ray --extra old-cpu`, then force lts polars
+  (`uv pip uninstall polars polars-lts-cpu && uv pip install polars-lts-cpu==1.33.1`) — see detection_tags Troubleshooting.
+- **Ops specs / app UI:** `image_*/app.py` registers specs via `datapipe_app_ml_ops.ops.ops_specs` +
+  `app.add_specs([...])`. Run the front with `uv run datapipe --executor RayExecutor --pipeline app api`
+  after `uv sync --extra ray` (ports: detection 8001, keypoints 8002, classification 8003).
 - **Services:** `docker compose up` → Postgres 5432, MinIO (9000/9001, bucket `datapipe-e2e`,
-  anon-download for browser images), MongoDB 27017, Label Studio :8080.
+  anon-download for browser images), MongoDB 27017, Label Studio :8080, ClickHouse :8123 (ops run logs).
 - **Env:** `cp .env.example .env` then `set -a && source .env && set +a` before any `datapipe` command
-  (`config.py` raises if `DB_URL` unset). Detection/keypoints use separate schemas.
+  (`config.py` raises if `DB_URL` or `CLICKHOUSE_RUN_LOGS_URL` unset). Separate schemas:
+  `DB_SCHEMA_DETECTION`, `DB_SCHEMA_KEYPOINTS`, `DB_SCHEMA_CLASSIFICATION`.
 - **LS API token:** :8080 → Account & Settings → Access Token → `LABEL_STUDIO_API_KEY=...` in `.env`
   (or `scripts/label_studio_token.py`).
-- **GPU for training** (no CPU knob). **≥10 annotated images** to freeze a dataset (`min_delta=10`).
+- **GPU for training.** YOLO has no CPU knob. TF classification needs GPU **or**
+  `CUDA_VISIBLE_DEVICES=` (empty) so `cpu_training_allowed` passes. **≥10 annotated images** to
+  freeze (`min_delta=10`).
 - Read-only/small `/home`: `export UV_CACHE_DIR=/tmp/uvcache HF_HOME=/tmp/hf` before `uv sync`.
-- Stages: `annotation`, `ls-sync`, `train`, `fiftyone`.
+- Stages: `annotation`, `train`, `fiftyone`.
 
 ## Quick demo to verify setup
-Skip if you have data: `uv run python scripts/seed_sample_data.py` downloads ~20 COCO images (10 cat/dog + 10 person keypoints; `--detection-limit`/`--keypoints-limit` to change) and uploads them to MinIO; then run §Run as-is.
+Skip if you have data: `uv run python scripts/seed_sample_data.py` downloads COCO images (cat/dog,
+person keypoints; limit flags to change) and uploads them to MinIO; then run §Run as-is.
+For classification add `--classification-animal-limit 12 --classification-no-animal-limit 12`
+to seed mixed Has Animal / No Animals images for Label Studio.
 
 ## Run (from the project subdir)
 ```bash
-cd image_detection                            # or image_keypoints
-source ../.venv/bin/activate                  # else prefix every command with `uv run`
+cd image_detection                            # or image_keypoints / image_classification
 set -a && source ../.env && set +a            # config raises w/o DB_URL
-datapipe db create-all
-datapipe step --labels=stage=annotation run   # LS tasks + pre-annotations
+uv run datapipe db create-all
+uv run datapipe --executor RayExecutor step --labels=stage=annotation run   # LS tasks (+ YOLO pre-anns for det/kp)
 # → annotate ≥10 images in LS (:8080), mark completed →
-datapipe step --labels=stage=ls-sync run      # → image__ground_truth
-datapipe step --labels=stage=train run        # freeze + train YOLO + metrics + best
-datapipe step --labels=stage=fiftyone run
-fiftyone app launch datapipe_detection_e2e    # or datapipe_keypoints_e2e
+uv run datapipe --executor RayExecutor step --labels=stage=annotation run   # sync annotations → image__ground_truth
+uv run datapipe --executor RayExecutor step --labels=stage=train run
+uv run datapipe --executor RayExecutor step --labels=stage=fiftyone run
+uv run fiftyone app launch datapipe_detection_e2e    # or datapipe_keypoints_e2e / datapipe_classification_e2e
 ```
-Train uses `yolov8n*.pt` (imgsz 320, 30 ep); pre-annotation fallback `yolo11n*.pt`; best on `subset_id=val`.
 
 ## Skip annotation — inject ground truth (unattended / demo)
-No human in Label Studio? You can bypass `annotation`/`ls-sync` by writing `image__ground_truth` +
-`image__subset` directly, then run `stage=train`. **The injected rows MUST follow the pipeline's own
-conventions, or the freeze join silently yields nothing / classes don't match:**
-- **`image_name` must equal what `list_s3_images` emits** — the key **relative to `INPUT_IMAGES_DIR`
-  (`$DATAPIPE_E2E_DIR/images`)**, i.e. the plain object name (`000000008458.jpg`). Don't invent a
-  prefixed form — if `s3_images` has `X` and your GT has `pfx___X`, the `GT ⋈ subset ⋈ s3_images` join
-  is empty → `freeze_dataset` fails with `No ground truth`.
-- **`labels` must match `DETECTION_CLASSES` casing** (e.g. `Cat`/`Dog`, not COCO lowercase `cat`/`dog`)
-  — otherwise predictions and GT land in *different* classes and every metric is 0.
-- **`bboxes`** = pixel `[x1,y1,x2,y2]`; assign `image__subset.subset_id` (`train`/`val`) yourself.
-- Write via datapipe `DataStore`/`UpdateExternalTable` (keeps `*_meta` in sync) — not raw SQL `UPDATE`
-  on PK columns. Then `datapipe step --labels=stage=train run` (skip `annotation`/`ls-sync`).
+No human in Label Studio? Bypass `annotation` by writing `image__ground_truth` + `image__subset`,
+then run `stage=train`. **Rows MUST follow the pipeline's conventions:**
+- **`image_name` must equal what `list_s3_images` emits** — key relative to `INPUT_IMAGES_DIR`
+  (`$DATAPIPE_E2E_DIR/images`), plain object name (`000000008458.jpg`).
+- **Detection/keypoints:** `labels` match class casing; `bboxes` = pixel `[x1,y1,x2,y2]`.
+- **Classification:** scalar `label` column (`Has Animal` / `No Animals`) — not a JSON list.
+- Write via datapipe `DataStore`/`UpdateExternalTable` (keeps `*_meta` in sync) — not raw SQL
+  `UPDATE` on PK columns.
 
 ## Per-scenario tag metrics → see the dedicated example
 Want to tag a scenario (e.g. dark-room pallets), add it to training, and measure the model on that
 scenario separately (baseline vs retrained)? That lives as its own self-contained example —
-`examples/detection_tags` (`tag`/`image__tag`/`tag_metrics` built into the pipeline, **no Label Studio
-or FiftyOne**, ground truth injected). Use the **setup-detection-tags** skill.
+`examples/detection_tags`. Use the **setup-detection-tags** skill.
 
 ## Troubleshooting (may already be fixed — verify against current files)
-- **No model after `train`, exit 0** → datapipe swallows step errors; check `detection_training_status`, not the exit code.
-- **Demo pre-annotations are empty** → the fallback `DETECTION_MODEL_CONFIG` is a smoke model (`yolo11n`, `input_size:[16,16]`, `score_threshold:0.01`) used until a trained model exists, so it detects almost nothing. Expected for the demo; for useful pre-annotations set a real model + `input_size`/`score_threshold`.
-- **Training metrics ~0** → not a config bug: the trained model is `yolov8n` (imgsz 320, 30 ep), and the seed sample (~20 images) is simply too small to learn from. Real metrics need enough annotated data.
+- **No model after `train`, exit 0** → datapipe swallows step errors; check `*_training_status`, not the exit code.
+- **TF train fails with "GPU not found"** → set `CUDA_VISIBLE_DEVICES=` (empty) for CPU, or use a real GPU.
+- **Demo detection pre-annotations are empty** → fallback `DETECTION_MODEL_CONFIG` is a smoke model until a trained model exists.
+- **Training metrics ~0** → seed sample is small; real metrics need enough annotated data.
+- **`SIGILL` / pre-AVX2 CPU** → same as **setup-detection-tags**: `uv sync --extra ray --extra old-cpu`, reinstall
+  `polars-lts-cpu==1.33.1`, and `pi-heif` if ultralytics image verification fails.
 - **`cv_pipeliner` keypoint pre-annotation is broken** (pinned rev): inferencer drops keypoints, LS parser doesn't apply them → keypoint `train` needs real keypoint GT injected.

.github/workflows/e2e-template.yml

@@ -11,6 +11,10 @@ on:
       - "tests/e2e_template/**"
       - "libs/datapipe-core/**"
       - "libs/datapipe-label-studio/**"
+      - "libs/datapipe-app/**"
+      - "libs/datapipe-app-ml-ops/**"
+      - "libs/datapipe-ui/**"
+      - "libs/datapipe-ui-ml/**"
       - "libs/datapipe-ml/**"
       - "pyproject.toml"
   pull_request:
@@ -22,6 +26,10 @@ on:
       - "tests/e2e_template/**"
       - "libs/datapipe-core/**"
       - "libs/datapipe-label-studio/**"
+      - "libs/datapipe-app/**"
+      - "libs/datapipe-app-ml-ops/**"
+      - "libs/datapipe-ui/**"
+      - "libs/datapipe-ui-ml/**"
       - "libs/datapipe-ml/**"
       - "pyproject.toml"
 
@@ -40,6 +48,7 @@ env:
   DB_URL: postgresql+psycopg://postgres:password@localhost:5432/postgres
   DB_SCHEMA_DETECTION: datapipe_e2e_detection
   DB_SCHEMA_KEYPOINTS: datapipe_e2e_keypoints
+  CLICKHOUSE_RUN_LOGS_URL: clickhouse://default:@localhost:8123/default
   FIFTYONE_DATABASE_URI: mongodb://localhost:27017
 
 jobs:

.github/workflows/lib-datapipe-app-ml-ops.yml

@@ -0,0 +1,76 @@
+name: datapipe-app-ml-ops
+
+on:
+  push:
+    branches:
+      - "master"
+      - "v**"
+    paths:
+      - ".github/workflows/lib-datapipe-app-ml-ops.yml"
+      - "libs/datapipe-app-ml-ops/**"
+      - "libs/datapipe-app/**"
+      - "libs/datapipe-core/**"
+      - "libs/datapipe-ml/**"
+      - "pyproject.toml"
+  pull_request:
+    branches:
+      - "**"
+    paths:
+      - ".github/workflows/lib-datapipe-app-ml-ops.yml"
+      - "libs/datapipe-app-ml-ops/**"
+      - "libs/datapipe-app/**"
+      - "libs/datapipe-core/**"
+      - "libs/datapipe-ml/**"
+      - "pyproject.toml"
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v5
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v6
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Install dependencies with uv
+        run: uv sync --package datapipe-app-ml-ops --group dev
+
+      - name: Lint with mypy
+        run: |
+          uv run --package datapipe-app-ml-ops mypy -p datapipe_app_ml_ops --ignore-missing-imports --follow-imports=silent --namespace-packages --check-untyped-defs
+
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v5
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v6
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Install dependencies with uv
+        run: uv sync --package datapipe-app-ml-ops --group dev
+
+      - name: Test with pytest
+        run: uv run --package datapipe-app-ml-ops pytest libs/datapipe-app-ml-ops/tests -q