Skip to content

feat(api): add non-generative task RPCs#19

Draft
connorcarpenter15 wants to merge 1 commit into
mainfrom
feat/non-generative-tasks
Draft

feat(api): add non-generative task RPCs#19
connorcarpenter15 wants to merge 1 commit into
mainfrom
feat/non-generative-tasks

Conversation

@connorcarpenter15

Copy link
Copy Markdown
Member

Summary

  • add typed Embed, Classify, and Score RPCs to the existing OpenEngine service
  • add common task inputs, dense and sparse embedding tensors, batch correlation, task usage, and grouped query/candidate scoring
  • add per-model, per-task capability discovery for inputs, granularities, value semantics, normalization, limits, modalities, priority, and LoRA
  • document batch-atomic lifecycle, validation, score semantics, and deterministic gateway-side reranking

Why

Generate cannot represent embeddings, classifier outputs, reward values, or model-native pair scores. Distributed frameworks therefore need engine-specific side channels for common non-generative inference.

This change establishes a portable typed surface across TensorRT-LLM, SGLang, and vLLM. It deliberately keeps reranking out of the engine protocol: gateways derive it from scalar Score results by stable-sorting, limiting, and reattaching gateway-owned documents.

Closes #10.

Validation

  • buf build
  • buf lint
  • buf format --diff --exit-code
  • buf breaking --against '.git#ref=origin/main'
  • Protobuf descriptor compilation with protoc
  • Markdown lint
  • Documentation link check
  • Commits include a DCO sign-off

Protocol checklist

  • The canonical schema and API documentation are synchronized
  • Presence, defaults, and invalid states are defined
  • Acceptance, unary terminal, cancellation, and error behavior are defined
  • Discovery and capability impacts are documented
  • The change is additive against origin/main

Additional context

  • Issue [Protocol]: Add portable non-generative inference task RPCs #10 was revised before implementation to retain Score and define Rerank as derived gateway/client behavior.
  • Successful task batches are atomic and preserve request order while also returning presence-aware original indexes.
  • Scalar, vector, and token-level scores cover native relevance/similarity, classifier/reward outputs, and causal label-token scoring.

Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Protocol]: Add portable non-generative inference task RPCs

1 participant