From ed6349da05694317a80e62645643432d61535860 Mon Sep 17 00:00:00 2001 From: Jan Eitzinger Date: Fri, 19 Jun 2026 06:23:39 +0200 Subject: [PATCH] Sync documentation with cc-specifications - Expand lineProtocol.md to cover all message categories (Events, Controls) and NATS subject hierarchy; add mandatory tags section, core metric list, and per-category examples - Fix stale cc-backend GitHub link in jobarchive/_index.md; add explanation of the timestamp directory level - Add missing interconnect metrics (ic_rcv_packets, ic_send_packets, ic_read_bw, ic_write_bw) to job-data-schema.md; fix source URL Co-Authored-By: Claude Sonnet 4.6 --- content/en/docs/explanation/lineProtocol.md | 174 +++++++++++--- .../reference/cc-backend/jobarchive/_index.md | 4 +- .../cc-backend/schemas/job-data-schema.md | 220 +++++++++++++----- 3 files changed, 312 insertions(+), 86 deletions(-) diff --git a/content/en/docs/explanation/lineProtocol.md b/content/en/docs/explanation/lineProtocol.md index 5654d79..b9826d8 100644 --- a/content/en/docs/explanation/lineProtocol.md +++ b/content/en/docs/explanation/lineProtocol.md @@ -1,85 +1,197 @@ --- title: InfluxDB Line Protocol description: > - Detailed specification of the InfluxDB Line Protocol format used for metric ingestion, covering Node and Hardware level metrics. -tags: ["influxdb", "protocol", "metrics"] + Specification of the InfluxDB line-protocol flavor used for messaging between ClusterCockpit components, covering metrics, events, and control messages. +tags: ["influxdb", "protocol", "metrics", "events", "nats"] categories: [cc-backend] --- -## Overview - -All metrics ingested into the cc-metric-store—whether via REST API or NATS—must strictly adhere to the **InfluxDB Line Protocol**. This text-based format allows us to tag high-frequency telemetry data with the necessary dimensions (cluster, host, hardware type) for efficient querying. -## Line Protocol Syntax +## Overview -The general format for a single data point is: +ClusterCockpit uses an [InfluxData line-protocol](https://docs.influxdata.com/influxdb/v2.1/reference/syntax/line-protocol/) flavor for transferring messages between its components. All messages share the same text-based format: ```text , ``` +Where `` and `` are comma-separated lists of `key=value` entries. The timestamp is Unix epoch time in **seconds**. + +{{< alert title="Backward Compatibility" >}} +Initially only metrics (number values) were sent. The specification was extended to support messages with different purposes (events, controls). This extension is backward-compatible — metric messages are unchanged. +{{< /alert >}} + +## Message Categories + +Three message categories are distinguished by their field key: + +| Category | Field Key | Field Type | Purpose | +| :---------- | :------------------- | :-------------- | :--------------------------------------- | +| **Metric** | `value=` | float/integer | Performance metric time series | +| **Event** | `event=""` | string (JSON) | Actionable job and cluster events | +| **Control** | `control=""` | string | Component configuration requests | -In our specific cc-metric-store implementation, the structure translates to: +## NATS Subject Hierarchy + +ClusterCockpit uses NATS for messaging. The subject hierarchy lets components subscribe only to the message types they need: ```text -metric_name,cluster=,hostname=,type=,type-id= value= +. | + --- metrics + | + --- events.[job, slurm] + | + --- control.[get, put] ``` -| Component | Description | Example | -| :-------------- | :--------------------------------------------------------- | :----------------------------- | -| **Measurement** | The specific metric name being recorded. | `cpu_load` | -| **Tags** | Key-value pairs providing context (metadata). | `cluster=alex,hostname=node01` | -| **Fields** | The actual data value. We use a single field key: `value`. | `value=45.2` | -| **Timestamp** | Unix timestamp in seconds. | `1725827464` | +## Tags + +### Mandatory Tags + +Every message — regardless of category — must include: + +| Tag | Description | Values | +| :-------- | :----------------------------------- | :------------------------------------------------------------------------------ | +| `hostname` | Source node hostname | e.g., `node01` | +| `type` | Hardware scope | `node`, `socket`, `die`, `memoryDomain`, `llc`, `core`, `hwthread`, `accelerator` | +| `type-id` | Component index within the type | e.g., `0`, `1`, `2` | + +Although `type-id` is not strictly required when `type=node`, sending `type=node,type-id=0` is recommended for consistency. + +### Optional Tags + +Some message types require additional tags: + +- `function` — for Event messages: the event purpose, e.g., `start_job`, `stop_job` +- `method` — for Control messages: `GET` or `PUT` + +For sub-typing (e.g., filesystem name or device path), use `stype` and `stype-id` rather than free-form tag names: + +```text +# Preferred +stype=filesystem,stype-id=/homes + +# Avoid +filesystem=/homes +``` --- -## Metric Modes +## Metric Messages + +**Identification:** `value=` field where the value is a float or integer. -We distinguishes between two primary scopes of metrics: **Hardware Level** and **Node Level**. +The measurement name is the metric name. While metric names can be chosen freely, the following core metrics should be present in any ClusterCockpit-compatible system: -### 1. Hardware Level Metrics -These metrics track the performance of specific sub-components *within* a node (e.g., a specific CPU core, a GPU, or a memory domain). +| Metric | Description | Unit | +| :----------- | :-------------------------------------------- | :------ | +| `flops_sp` | Single-precision floating point rate | Flops/s | +| `flops_dp` | Double-precision floating point rate | Flops/s | +| `flops_any` | Combined floating point rate | Flops/s | +| `cpu_load` | 1-minute load average (`/proc/loadavg`) | — | +| `mem_used` | Memory used by applications (`/proc/meminfo`) | Bytes | +| `ipc` | Instructions per cycle | — | +| `mem_bw` | Main memory bandwidth (read + write) | MB/s | +| `cpu_power` | CPU package power consumption | W | +| `mem_power` | Memory subsystem power consumption | W | +| `clock` | CPU clock frequency | MHz | -**Requirement:** You must include the `type-id` tag to distinguish between multiple components of the same type on the same host. +For the complete metric list see the [job-data schema reference]({{< ref "job-data-schema" >}}). + +**Example:** + +```text +flops_any,hostname=e1208,type=core,type-id=23 value=1203.3 1740027951 +``` + +For metrics ingested into **cc-metric-store** (via REST API or NATS), the `cluster` tag is additionally required: + +```text +flops_any,cluster=alex,hostname=e1208,type=core,type-id=23 value=1203.3 1740027951 +``` + +### Metric Scopes + +We distinguish two primary scopes: **Hardware Level** and **Node Level**. + +#### Hardware Level Metrics + +These metrics track performance of specific sub-components within a node (e.g., a CPU core, GPU, or memory domain). The `type-id` tag identifies which component instance. **Schema:** ```text ,cluster=,hostname=,type=,type-id= value=