Skip to content

Use a ~/.scarfbench directory to store and retrieve the dataset as a sensible default #4

Description

@rahlk

Steps to follow--

  1. Define .scarfbench layout (benchmark, evals, logs, tmp).
  2. Add one shared path-resolution policy: flag > env > .scarfbench default.
  3. Add bootstrap logic to create/validate .scarfbench directories.
  4. Make bench pull default to .scarfbench/benchmark.
  5. Make bench list and bench test default to .scarfbench/benchmark.
  6. Make eval run default eval-out to .scarfbench/evals and benchmark input to .scarfbench/benchmark.
  7. Keep existing flags working as explicit overrides (backward compatible).
  8. Standardize path normalization/error handling across all commands.
  9. Add tests for default-path behavior and override behavior.
  10. Update CLI help + README examples to document new defaults.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions