Monitor every device in the field.
Fix your firmware in hours, not weeks.

Documentation · Website · Portal · Samples

Spotflow Observability Device SDK

Device SDK for Spotflow embedded observability platform.

This SDK provides a set of tools and libraries for Zephyr RTOS to send your logs to the Spotflow observability platform.

Device SDK is integrated with Zephyr as a module that contains the Spotflow logging backend that seamlessly integrates with the Zephyr logging subsystem.

Our solution was tested on the following Zephyr boards (more are coming soon):

• ESP32-C3-DevKitC
• ESP32-C6-DevKitC
• ESP32-S3-DevKitC
• Infineon CY8CProto-062-4343W
• Nordic nRF7002DK
• NXP FRDM-RW612
• Raspberry Pi Pico 2 W

We currently support:

• Zephyr 3.7.x, 4.1.x, 4.2.x, 4.3.x, 4.4.x
• nRF Connect SDK 3.0.x and 3.1.x

For more information, check Spotflow or Spotflow documentation.

Getting Started

Register and get your Ingest key at Spotflow.

Follow the Quickstart guide that is available in our portal after registration.

Alternatively, you can check sample applications in the samples. The device SDK is meant to be used as a Zephyr module. You can add it to your Zephyr project by adding the following line to your west.yml:

manifest:
    projects:
    - name: spotflow
      path: modules/lib/spotflow
      revision: main
      url: https://github.com/spotflow-io/device-sdk

Documentation

Architecture

Logging

---
title: Main log flow
---
flowchart LR
    A[User Code] --> B[Zephyr RTOS]
    B --> C[Spotflow Logging Backend]
    C --> D[Encode CBOR]
    D --> E[Spotflow Backend Processor]
    E --> F[Spotflow Transport]
    F --> G[MQTT transport]
    F --> H[BLE transport]
    G -- QoS 0 --> I[Spotflow MQTT Broker]
    H --> J[BLE gateway]
    J --> I
    I --> K[Spotflow Observability Platform]

Loading

The MQTT transport uses MQTT QoS 0. The BLE transport sends the same CBOR payloads over a Spotflow GATT service for a gateway to relay to Spotflow.

---
title: Spotflow Data Flow
---
flowchart LR
    processor[Spotflow Backend Processor]
    transport[spotflow transport]

    A[Zephyr logging] --> B[Spotflow Logging Backend]
    B --> C[Encode CBOR]
    C --> D[Log message queue]
    D --> processor

    E[Application metrics] --> F[Spotflow Metrics Backend]
    F --> G[Encode CBOR]
    G --> H[Metrics message queue]
    H --> processor

    I[Cloud configuration] --> J[Spotflow Config Backend]
    J --> processor

    K[Extract coredump] --> L[Coredumps Backend]
    L --> M[Encode CBOR]
    M --> N[Coredumps message queue]
    N --> processor

    processor --> transport
    transport --> O[MQTT transport]
    transport --> P[BLE transport]
    O --> Q[Spotflow MQTT Broker]
    P --> R[GATT TX and RX streams]
    R --> S[BLE gateway]
    S --> Q

Loading

BLE currently supports logs, metrics, and cloud configuration. Coredumps are currently sent only over MQTT.

Configuration

Most properties of the device SDK are currently configured in build time using Kconfig - see the help text of each option for more information.

The device SDK also provides a way to dynamically configure certain properties from the cloud using the Spotflow portal. Currently, there is only one such property:

• Minimal severity of sent log messages (the level of sent log messages)

The configuration process uses the mechanism of desired and reported values that are stored for each device in the Spotflow platform and are synchronized either directly over MQTT or through a BLE gateway. If Zephyr Settings subsystem is enabled, the device SDK uses it to persist the last active configuration.

---
title: Configuration from the Spotflow Platform
---
sequenceDiagram
participant ZS as Zephyr Settings
participant SDK as Device SDK
participant GW as BLE gateway
participant PM as Spotflow Platform

Note left of SDK: On startup
SDK ->> ZS: Try load initial configuration
ZS ->> SDK: Configuration loaded / defaults used

alt Direct MQTT transport
    Note right of SDK: On MQTT connection activation
    SDK ->> PM: Send reported configuration
    SDK ->> PM: Subscribe to desired configuration

    loop When desired configuration changes
        PM ->> SDK: Send desired configuration
        SDK ->> ZS: Persist configuration
        SDK ->> PM: Send reported configuration
    end
else BLE gateway transport
    Note right of SDK: On BLE session activation
    SDK ->> GW: Send reported configuration via TX stream

    loop When desired configuration changes
        PM ->> GW: Deliver desired configuration
        GW ->> SDK: Write desired configuration via RX stream
        SDK ->> ZS: Persist configuration
        SDK ->> GW: Send reported configuration via TX stream
        GW ->> PM: Relay reported configuration
    end
end

Loading

For BLE transport, the device exposes session metadata as a readable GATT characteristic and exchanges configuration data through framed TX and RX stream messages.

Over-the-air (OTA) updates

Spotflow OTA updates let the cloud ask a device to install one or more firmware versions. Each device-specific request is an update attempt identified by an attempt ID. It contains an ordered manifest of artifacts, where each artifact describes one firmware image and the version to install. The SDK processes the artifacts in manifest order, persists their results, and reports those results to the Spotflow platform.

An artifact marked as main firmware updates the firmware that runs the Spotflow SDK. The SDK can handle it automatically using MCUboot: it downloads the image, requests a test upgrade, reboots, and waits for the application to confirm that the new image works. Other artifacts are delegated firmware: the SDK invokes an application callback that performs the update, typically for an external MCU. Main firmware can also be delegated when automatic handling is disabled.

The implementation is split into the following components:

---
title: Implementation Responsibilities for OTA Updates
---
flowchart TD
    processor[Processor:<br/>Receiving cloud-to-device messages<br/>Sending device-to-cloud messages]
    facade[Public API]
    core[Core:<br/>State and worker]
    firmware[Firmware handlers:<br/>Automatic main and delegated]
    downloader["Downloader:<br/>HTTP(S), retrying, pausing, canceling"]
    persistence[Persistence:<br/>State and results]
    platform[Platform wrappers:<br/>MCUboot, flash memory, build ID parsing]

    style processor stroke-dasharray: 5 5

    processor --> core
    facade --> downloader
    facade --> core
    core --> firmware
    core --> persistence
    core --> processor
    firmware --> downloader
    firmware --> platform
    firmware --> persistence

Loading

• Public API enables application code to influence automatic main-firmware updates (observing, pausing, aborting) and implement delegated firmware updates.
• Processor handles communication with the Spotflow platform, including cloud-to-device (C2D) requests and device-to-cloud (D2C) results.
• Core contains the management of the update state and the delegation of complex tasks to a worker thread.
• Firmware handlers run on the worker thread and contain the logic for both automatic main-firmware updates and application-provided delegated updates.
• Downloader provides a resilient download mechanism for firmware updates. It is used internally by the automatic firmware update handler and can be used directly by the user code as well.
• Platform wrappers provide an interface for low-level features so that they can be easily faked in tests.
• Persistence of update state, results, installed versions, and main-firmware probation is handled by the Zephyr Settings subsystem.

A successful automatic main-firmware update looks like this:

---
title: Happy Path of a Main-Firmware Update
---
sequenceDiagram
    participant Cloud as Spotflow Cloud
    participant Processor
    participant Worker as Update Worker
    participant Settings as Zephyr Settings
    participant DL as Downloader
    participant Boot as MCUboot
    participant App as Application

    Cloud->>Processor: Send manifest for update attempt
    Processor->>Worker: Accept attempt
    Worker->>Settings: Persist accepted attempt
    Worker->>DL: Download image<br />to secondary slot
    Worker->>Settings: Persist probation
    Worker->>Boot: Request test upgrade
    Note right of Worker: Reboot device, new<br />image is unconfirmed
    App->>Worker: spotflow_confirm_main_firmware_image()
    Worker->>Boot: Confirm image
    Worker->>Settings: Persist installed version<br />and successful result
    Worker->>Processor: Prepare cumulative results
    Processor->>Cloud: Report attempt results

Loading

Choose the next document according to what you want to do:

• Integrate OTA updates: Over-the-air updates with Zephyr
• Try OTA updates: Sample for OTA updates, an end-to-end application with MCUboot sysbuild
• Maintain the SDK: Implementation design notes for OTA updates, including state, concurrency, persistence, recovery, and testing rationale

Build ID

In order to match core dumps with symbol files, our Zephyr module provides a piece of information called build ID. The build ID is computed as a hash of the bytes loaded to the device; therefore, it uniquely identifies the firmware image. Our Zephyr module adds a build command that computes the build ID and patches it into the .elf file as the following binary descriptor:

• ID: 0x5f0 (5f resembles sf - Spotflow, 0 stands for our first binary descriptor ID)
• Type: bytes
• Length: 20

You can retrieve the build ID from the .elf file using the following command:

west bindesc custom_search BYTES 0x5f0 zephyr.elf

Note: If the architecture of the device doesn't support binary descriptors (notably, ESP32), retrieving the build ID from the .elf file is more complex. In this case, it's necessary to use the symbol table, see the function get_build_id_from_elf in script tests.

Because Zephyr doesn't allow to insert a post-build command between the compilation of zephyr.elf and the generation of derived files such as zephyr.hex and zephyr.bin, our build command patches these files as well. In particular, the files with the following extensions are patched:

• .elf
• .hex
• .bin
• .strip
• .exe when not targeting native simulator (it's just a copy of the .elf file)

The files with the following extensions (and others that might be introduced in the future) are not patched, so the build IDs stored in them are filled with zeros:

• .lst (assembly listing)
• .uf2
• .s19
• .exe when targeting native simulator

Feedback

Any comments, suggestions, or issues are welcome. Create a Github issue or contact us at hello@spotflow.io, LinkedIn or Discord.