Skip to content

feat(benchmarking): enhance inference-perf-bench tooling - #540

Open
alizaidis wants to merge 1 commit into
mainfrom
feat/inference-benchmarking
Open

feat(benchmarking): enhance inference-perf-bench tooling#540
alizaidis wants to merge 1 commit into
mainfrom
feat/inference-benchmarking

Conversation

@alizaidis

Copy link
Copy Markdown
Collaborator

Description

This PR enhances our implementation of inference-perf-bench load-testing utility to be more robust, flexible, and resilient to missing configuration values.

Key Changes

  • Smart Defaults: Added explicit fallback mechanisms in configure_benchmark.sh to dynamically derive APP_LABEL, INFERENCE_BASE_URL, and BENCHMARK_STAGE_DURATION if they aren't provided in the environment. This prevents structurally invalid Kustomize templates from being generated during CI or local testing.
  • Directory Fallbacks: Updated run_benchmark.sh so that if a model-specific configuration directory doesn't exist for the current model, the script will gracefully fall back to the base benchmark configuration directory rather than failing out.
  • HuggingFace Secret Integration: Formally added secretproviderclass-huggingface-tokens.tpl.yaml to the benchmark template generation flow to ensure the benchmarking jobs can mount tokens necessary for testing gated models.
  • Dynamic Templating Fixes: Ensured benchmarking.tpl.env dynamically interpolates the proper target namespaces and service accounts instead of relying on hardcoded environment names.

Motivation and Context

These utility improvements make the benchmarking suite significantly easier to use when testing new or custom models. By handling missing environment variables and missing model-specific directories natively, it removes manual boilerplate required from developers attempting to load-test their deployments.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature / Tooling (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

@alizaidis
alizaidis force-pushed the feat/inference-benchmarking branch from 2690528 to 93f4418 Compare August 7, 2026 19:25

@syeda-anjum syeda-anjum left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants