Skip to content

Sync documentation with cc-specifications #154

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
moebiusband73 merged 1 commit into from
Jun 30, 2026
Merged

Sync documentation with cc-specifications #154

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
174 changes: 143 additions & 31 deletions content/en/docs/explanation/lineProtocol.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,85 +1,197 @@
---
title: InfluxDB Line Protocol
description: >
Detailed specification of the InfluxDB Line Protocol format used for metric ingestion, covering Node and Hardware level metrics.
tags: ["influxdb", "protocol", "metrics"]
Specification of the InfluxDB line-protocol flavor used for messaging between ClusterCockpit components, covering metrics, events, and control messages.
tags: ["influxdb", "protocol", "metrics", "events", "nats"]
categories: [cc-backend]
---
## Overview

All metrics ingested into the cc-metric-store—whether via REST API or NATS—must strictly adhere to the **InfluxDB Line Protocol**. This text-based format allows us to tag high-frequency telemetry data with the necessary dimensions (cluster, host, hardware type) for efficient querying.

## Line Protocol Syntax
## Overview

The general format for a single data point is:
ClusterCockpit uses an [InfluxData line-protocol](https://docs.influxdata.com/influxdb/v2.1/reference/syntax/line-protocol/) flavor for transferring messages between its components. All messages share the same text-based format:

```text
<measurement>,<tag_set> <field_set> <timestamp>
```

Where `<tag_set>` and `<field_set>` are comma-separated lists of `key=value` entries. The timestamp is Unix epoch time in **seconds**.

{{< alert title="Backward Compatibility" >}}
Initially only metrics (number values) were sent. The specification was extended to support messages with different purposes (events, controls). This extension is backward-compatible — metric messages are unchanged.
{{< /alert >}}

## Message Categories

Three message categories are distinguished by their field key:

| Category | Field Key | Field Type | Purpose |
| :---------- | :------------------- | :-------------- | :--------------------------------------- |
| **Metric** | `value=<number>` | float/integer | Performance metric time series |
| **Event** | `event="<json>"` | string (JSON) | Actionable job and cluster events |
| **Control** | `control="<string>"` | string | Component configuration requests |

In our specific cc-metric-store implementation, the structure translates to:
## NATS Subject Hierarchy

ClusterCockpit uses NATS for messaging. The subject hierarchy lets components subscribe only to the message types they need:

```text
metric_name,cluster=<name>,hostname=<host>,type=<hw_type>,type-id=<id> value=<float> <unix_epoch>
<cluster name>. |
--- metrics
|
--- events.[job, slurm]
|
--- control.[get, put]
```

| Component | Description | Example |
| :-------------- | :--------------------------------------------------------- | :----------------------------- |
| **Measurement** | The specific metric name being recorded. | `cpu_load` |
| **Tags** | Key-value pairs providing context (metadata). | `cluster=alex,hostname=node01` |
| **Fields** | The actual data value. We use a single field key: `value`. | `value=45.2` |
| **Timestamp** | Unix timestamp in seconds. | `1725827464` |
## Tags

### Mandatory Tags

Every message — regardless of category — must include:

| Tag | Description | Values |
| :-------- | :----------------------------------- | :------------------------------------------------------------------------------ |
| `hostname` | Source node hostname | e.g., `node01` |
| `type` | Hardware scope | `node`, `socket`, `die`, `memoryDomain`, `llc`, `core`, `hwthread`, `accelerator` |
| `type-id` | Component index within the type | e.g., `0`, `1`, `2` |

Although `type-id` is not strictly required when `type=node`, sending `type=node,type-id=0` is recommended for consistency.

### Optional Tags

Some message types require additional tags:

- `function` — for Event messages: the event purpose, e.g., `start_job`, `stop_job`
- `method` — for Control messages: `GET` or `PUT`

For sub-typing (e.g., filesystem name or device path), use `stype` and `stype-id` rather than free-form tag names:

```text
# Preferred
stype=filesystem,stype-id=/homes

# Avoid
filesystem=/homes
```

---

## Metric Modes
## Metric Messages

**Identification:** `value=<number>` field where the value is a float or integer.

We distinguishes between two primary scopes of metrics: **Hardware Level** and **Node Level**.
The measurement name is the metric name. While metric names can be chosen freely, the following core metrics should be present in any ClusterCockpit-compatible system:

### 1. Hardware Level Metrics
These metrics track the performance of specific sub-components *within* a node (e.g., a specific CPU core, a GPU, or a memory domain).
| Metric | Description | Unit |
| :----------- | :-------------------------------------------- | :------ |
| `flops_sp` | Single-precision floating point rate | Flops/s |
| `flops_dp` | Double-precision floating point rate | Flops/s |
| `flops_any` | Combined floating point rate | Flops/s |
| `cpu_load` | 1-minute load average (`/proc/loadavg`) | — |
| `mem_used` | Memory used by applications (`/proc/meminfo`) | Bytes |
| `ipc` | Instructions per cycle | — |
| `mem_bw` | Main memory bandwidth (read + write) | MB/s |
| `cpu_power` | CPU package power consumption | W |
| `mem_power` | Memory subsystem power consumption | W |
| `clock` | CPU clock frequency | MHz |

**Requirement:** You must include the `type-id` tag to distinguish between multiple components of the same type on the same host.
For the complete metric list see the [job-data schema reference]({{< ref "job-data-schema" >}}).

**Example:**

```text
flops_any,hostname=e1208,type=core,type-id=23 value=1203.3 1740027951
```

For metrics ingested into **cc-metric-store** (via REST API or NATS), the `cluster` tag is additionally required:

```text
flops_any,cluster=alex,hostname=e1208,type=core,type-id=23 value=1203.3 1740027951
```

### Metric Scopes

We distinguish two primary scopes: **Hardware Level** and **Node Level**.

#### Hardware Level Metrics

These metrics track performance of specific sub-components within a node (e.g., a CPU core, GPU, or memory domain). The `type-id` tag identifies which component instance.

**Schema:**
```text
<metric>,cluster=<c>,hostname=<h>,type=<component>,type-id=<index> value=<v> <time>
```

**Example Hardware Types:**
**Example hardware types:**
* **`hwthread`**: Logical CPU threads. (IDs: `0..127` for Cluster1, `0..71` for Cluster2)
* **`socket`**: Physical CPU sockets. (IDs: `0..1`)
* **`accelerator`**: GPUs or FPGA cards. (IDs: PCI Bus Address, e.g., `00000000:49:00.0`)
* **`memoryDomain`**: NUMA nodes. (IDs: `0..7`)

**Example Payload:**
**Examples:**
```text
cpu_user,cluster=alex,hostname=a0603,type=hwthread,type-id=12 value=88.5 1725827464
core_power,cluster=fritz,hostname=f0201,type=socket,type-id=0 value=120.0 1725827464
```

### 2. Node Level Metrics
These metrics represent the aggregate state of the entire node.
#### Node Level Metrics

**Requirement:** The `type` tag is set to `node`. The `type-id` tag is usually omitted or ignored for these metrics.
These metrics represent the aggregate state of the entire node. Set `type=node`; the `type-id` tag can be omitted or set to `0`.

**Schema:**
```text
<metric>,cluster=<c>,hostname=<h>,type=node value=<v> <time>
```

**Example Payload:**
**Example:**
```text
mem_used,cluster=alex,hostname=a0603,type=node value=64000.0 1725827464
ib_xmit,cluster=fritz,hostname=f0201,type=node value=1024500.0 1725827464
```

---

## Event Messages

**Identification:** `event="<json>"` field where the value is a JSON string.

The measurement name indicates the event class. The `function` tag specifies the purpose (similar to a REST endpoint path).

| Event Class | `function` values |
| :---------- | :---------------------------- |
| `job` | `start_job`, `stop_job` |
| `slurm` | slurm-specific event types |

**Example:**

```text
job,hostname=mngmt02,type=node,type-id=0,function=stop_job event={"jobId": 69, "cluster": "ccfront", "stopTime": 1738842306, "jobState": "completed"} 1740027951
```

---

## Control Messages

**Identification:** `control="<string>"` field where the value is the control request payload.

The measurement name is the control class. The `method` tag is either `GET` or `PUT`.

| Control Class | Description |
| :------------ | :----------------------------------- |
| `rapl` | CPU power capping (RAPL interface) |
| `freq` | CPU frequency control |
| `prefetcher` | Hardware prefetcher control |
| `topology` | Topology configuration |
| `config` | Component configuration |

**Example:**

```text
rapl,hostname=e1208,type=socket,type-id=2,method=GET control=intel.pkg.energy_status 1740027951
```

---

## Related Tools

To test this protocol with synthetic data, you can use the **Metric Generator**.
See the documentation here:
To test metric ingestion with synthetic data, use the **Metric Generator Script**:
[Metric Generator Script]({{< ref "/docs/reference/cc-backend/tools/dataGenerator.md" >}})

---
4 changes: 3 additions & 1 deletion content/en/docs/reference/cc-backend/jobarchive/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ weight: 5
The job archive specifies an exchange format for job meta and performance metric
data. It consists of two parts:

- a [Json file format](https://github.com/ClusterCockpit/cc-backend/tree/main/pkg/schema/schemas)
- a [Json file format](https://github.com/ClusterCockpit/cc-lib/tree/main/schema/schemas)
- a Directory hierarchy / Key specification

By using an open, portable and simple specification based on JSON objects it is
Expand Down Expand Up @@ -56,6 +56,8 @@ and column value there.
For the job ID 1034871 on cluster `large` with start time `1768978339` the key
is `./large/1034/871/1768978339`.

The final path component is the Unix epoch start timestamp (in seconds). This disambiguates jobs whose ID suffix collides — for example after a job-ID counter reset or a cluster migration where IDs are reused.

## Create a Job archive from scratch

In case you place the job-archive in the `./var` folder create the folder with:
Expand Down
Loading
Loading