[vectorset] Runbook support - #1349
Conversation
There was a problem hiding this comment.
Pull request overview
Adds Big ANN Benchmarks–style “runbook” execution to the vectorset binary by introducing a runbook parser, dataset catalog/spec loading, a driver-based runner that executes insert/delete/replace/search steps, and JSON report emission for later analysis.
Changes:
- Added runbook parsing (
Runbook/Recipe/Operation) and aRunnerthat executes runbooks with optional dataset filtering and progress reporting. - Added dataset catalog + dataset spec parsing/loading (including step ground-truth loading) to support runbook-driven benchmarking.
- Added report structs and JSON output; updated CLI to include a new
runsubcommand and added example dataset specs/docs.
Reviewed changes
Copilot reviewed 14 out of 22 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| vectorset/src/test_utils.rs | Adds a small helper for creating temporary YAML files used by unit tests. |
| vectorset/src/runner.rs | Implements the runbook execution engine, operation orchestration, progress, and report generation. |
| vectorset/src/runbook.rs | Implements runbook YAML parsing, validation, and unit tests. |
| vectorset/src/report.rs | Defines JSON-serializable report structures for run results. |
| vectorset/src/main.rs | Adds run subcommand wiring, connection config changes, and integrates catalog/runbook/runner flow. |
| vectorset/src/garnet.rs | Introduces a Driver implementation for Garnet-backed vector set operations. |
| vectorset/src/driver.rs | Adds the Driver trait abstraction used by the runner. |
| vectorset/src/dataset.rs | Adds dataset spec parsing, data loading, and step ground-truth loading utilities. |
| vectorset/src/catalog.rs | Adds dataset catalog loading from a directory of dataset spec YAML files. |
| vectorset/README.md | Documents the new run command and updates general usage guidance. |
| vectorset/datasets/wikipedia-1M.yaml | Adds an example dataset spec entry for the Wikipedia 1M dataset. |
| vectorset/datasets/wikipedia-100K.yaml | Adds an example dataset spec entry for the Wikipedia 100K dataset. |
| vectorset/Cargo.toml | Adds dependencies required for runbooks, reporting, and dataset parsing; adds tempfile for tests. |
| vectorset/Cargo.lock | Locks newly introduced dependencies. |
| vectorset/.gitignore | Ignores generated reports/ output directory. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| pub enum RunnerError { | ||
| #[error("Nothing to do; empty runbook or nothing matched filter")] | ||
| NothingToDo, | ||
| #[error("data set missing: {0}")] |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1349 +/- ##
==========================================
- Coverage 91.57% 91.55% -0.03%
==========================================
Files 521 521
Lines 99598 100347 +749
==========================================
+ Hits 91211 91872 +661
- Misses 8387 8475 +88
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Adds big-ann-benchmarks style runbook support to vectorset.