Skip to content

Evaluation: Separate out v2 evaluation endpoints (Langfuse-free) #1050

Description

@Ayush8923

Is your feature request related to a problem?
The current evaluation API relies on Langfuse, which adds unnecessary dependencies and complexity. We need a fully Kaapi-native v2 evaluation API that separates from the existing v1 endpoints without changes to their functionality.

Describe the solution you'd like

  • Create a new /api/v2 endpoint that mirrors the v1 run trigger with the same request body. Ensure v2 runs are native-only.
  • Implement a Langfuse-free dataset upload that stores the CSV in S3 and keeps original items only, while marking dataset metadata accordingly.
  • Enable v2 runs on duplicated datasets to expand items at load time with unique IDs. Ensure v1 datasets read S3 data without modifications.
  • Reuse the existing chunked fast-eval pipeline and add a marker to indicate judge runs for proper evaluation skipping.
  • Ensure v2 dataset loading directly accesses S3 instead of Langfuse.
Original issue

Describe the current behavior
Introduce a versioned v2 evaluation API surface that is fully Kaapi-native (no Langfuse dependency), leaving the existing v1 endpoints byte-for-byte unchanged. This issue covers only the endpoint/pipeline separation and the Langfuse-free dataset flow. The native ground-truth LLM-as-a-judge scoring is tracked separately in #959 and must not be added here.

Describe the enhancement you'd like

  • a replica of the v1 run trigger, mounted under a new /api/v2 prefix (settings.API_V2_STR). Same request body as v1. v2 fast runs are native-only: no cosine, no embeddings, no Langfuse sync/dependency.
  • a Langfuse-free dataset upload. Same multipart shape as v1, but stores the CSV in S3 only, creates the evaluation_dataset row with langfuse_dataset_id NULL, keeps original items only (no physical duplication), and records duplication_factor + a run-time-duplication marker in dataset_metadata.
  • a v2 run on a run-time-duplicated dataset expands each original item ×duplication_factor at load time (unique id per copy). A v1-created (pre-duplicated, Langfuse-backed) dataset reads its S3 data as-is without re-multiplying.
  • reuse the existing chunked fast-eval pipeline; add an is_judge_run marker on evaluation_run so the aggregate (which runs by eval_run_id only) knows to skip cosine/embeddings/Langfuse for v2. Langfuse client resolution (chunk, sizing, aggregate) must not require credentials for a v2/S3-backed run.
  • v2 dataset item loading reads from S3 (object store) instead of Langfuse.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions