feat: add storage and OCI lifecycle latency benchmark

[jim-junior]

This is the initial PR for submiting deliverables for Phase 2 — Midterm milestone of urunc-dev/urunc#759 thats adds the following:

• The harness program skeleton i.e the general structure on the harness program, This involes
• Benchmarking code for OCI Lifecycle latency
• Benchmarking code for Storage
• Jupyter Notebooks for analysis of benchmark results

Current Evaluation Suite Support

The table below summarizes the benchmark areas, metrics, and supporting tools currently implemented in this phase of the evaluation suite.

Area	Metrics Currently Supported	Tools / Dependencies
OCI Lifecycle Latency	Container lifecycle command latency for create, start, and delete, measured in milliseconds (ms)	skopeo, umoci, containerd events
Storage Performance	Read and write throughput in MB/s per runtime; read and write IOPS; P99 tail latency for read and write operations; relative performance compared with the runc baseline	fio

Harness Skeleton

The harness skeleton is a Go cli program. It works by taking in a experiment.yml manifest file, it validates it against the schema, generates an execution plan and then executes the benchmarks based. It also accompanies multiple helper commands ad demonstrated below.

$ ./harness --help
A tool to benchmark OCI rutimes and compare their performance. It provides a set of commands to run benchmarks, collect results, and generate reports.

Usage:
  evaluation_suite [flags]
  evaluation_suite [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  doctor      Check whether the host is ready to run benchmark experiments
  help        Help about any command
  plan        Generate a concrete trial plan from an experiment manifest
  run         Run benchmark trials from an experiment manifest
  validate    Validate an experiment manifest

Flags:
  -h, --help      help for evaluation_suite
  -v, --version   version for evaluation_suite

Use "evaluation_suite [command] --help" for more information about a command.

$  ./harness run --help
Run benchmark trials from an experiment manifest

Usage:
  evaluation_suite run [flags]

Flags:
      --dry-run              Use fake adapter and fake collector without running live commands
  -f, --file string          Path to experiment manifest
  -h, --help                 help for run
      --limit int            Limit number of trials to run
  -o, --output string        Output format: summary or json (default "summary")
      --results-dir string   Directory where run artifacts are written (default "results")

The experiment manifest is defined as follows in the example below(note that memory and CPU are not included)

runtimes:
  - name: runc
    handler: io.containerd.runc.v2

  - name: kata
    handler: io.containerd.kata.v2

  - name: runsc
    handler: io.containerd.runsc.v1

  - name: urunc
    handler: io.containerd.urunc.v2


experiments:

  lifecycle:
    workloads:
      default:
        image: docker.io/library/nginx:alpine

      other:
      - image: docker.io/library/minimalc:test
        runtime: urunc

  storage:
    workloads:
      default:
        image: docker.io/jimjuniorb/fio:0.1
        snapshotter: overlayfs
      other:

      - image: docker.io/jimjuniorb/fio-urunc:0.4
        runtime: urunc
        snapshotter: devmapper

OCI Lifecycle latency benchmark

For OCI lifecycle latency, the approach used for measuring is as follows. We listen to containerd runtime tasks for the rutimes runc, kata and urunc. For gVisor(runsc) we are running the runsc cli command with a bundle and other required arguments then measuring the execution time it takes, This is because runsc does not abide by the Runtime v2 spec.

Aditionaly for urunc, when measuring the start OCI command, we ignore the /tasks/start event because as mentioned in the previous weekly sync it publishes this task earlier than when the container starts. To solve this, we created a minimal C program that starts and exits immediately with 0 code. Its code is in workloads/minimal-c/main.c

Storage Benchmark

As for storage, we built custom images with fio included in them.

For runtimes kata, runc and runsc, we are using the image docker.io/jimjuniorb/fio:0.1

As for urunc we are using the docker.io/jimjuniorb/fio-urunc:0.4 image.

For urunc, the kernel included in the image harbor.nbfc.io/nubificus/urunc/linux-kernel-qemu:v6.14 was misssing certain system calls that fio depends on, this required me to build a custom kernel that would be used in this benchmark. To do this i adjusted the kernel config @cmainas used at https://gist.github.com/cmainas/223e1525496dd2c8e08dbf8bab41df80 and added support for shmget syscall by enabling the CONFIG_SYSVIPC and its dependencies. The final kernel build config file can be found in workloads/fio-kernel/.config

Analysis Notebooks

For result analysis, i opted to use Jupyter notebooks. I find them simpler and more powerful to use for analysis. Currently there are only two notebooks, fio_runtime_analysis.ipynb (for storage) and oci_lifecycle_latency_analysis.ipynb located in the notebooks directory.

For now the benchmark results(which are stored in a run.json file) should be copied and put in the same directory as the notebooks when doing analysis.

These notebooks generate reports and easy to read graphs.

Sample results

---

LLM usage

Used LLMs mainly to assist on the process/pipeline of how to build a custom linux kernel

Checklist

• I have read the contribution guide.
• If LLMs were used: I have read the llm policy.

[cmainas]

Hello @jim-junior ,

thank you for the PR. We can easily merge this as a base, however we do not need to have release CI for the time being. Also, I have a few questions:

• How do we trigger the generation of the results based on the notebooks?
• Is the main.py app used / related to notebooks?

and some more generic comments:

• Would it be possible to add in the README how to use the current version? Maybe this one will also clarify the notebooks question.

[cmainas]

Hello @jim-junior ,

thank you for the changes, just one more comment.

[cmainas]

I think this file still needs to get cleaned up.

[jim-junior]

Hello @cmainas

Unfortunately if you want to run the notebooks locally, that file will be required, it is used by the package manager uv to resolve dependencies required in the project., However. I can delete it and update the readme.md to ensure that the notebooks are only run in Google colab(which is a cloud workspace that allows running of notebooks in the cloud)

The pyproject.toml in python serves the same purpose that as go.mod in go projects.

Should I still head on to delete it and opt for colab?

[cmainas]

I see, then we should keep it. Actually, we should not rely to external services, so we should definitely keep it. My bad.

[jim-junior]

I have also added the table for what evaluations the PR supports above in the description and you had requested in urunc-dev/urunc#759 (reply in thread)

[cmainas]

I have also added the table for what evaluations the PR supports above in the description and you had requested in urunc-dev/urunc#759 (reply in thread)

Great, there is one more request. All commits must be signed-off. Feel free to clean up a bit the commits (squashing some of them) too.