feat(llmd): improvements to capture runtime configuration, refactor telemetry-collector, and fix for load-balancer time-out on large benchmarks - #536
Open
syeda-anjum wants to merge 11 commits into
Open
Conversation
- Add --wait-timeout ${WAIT_TIMEOUT:-21600} to Stage 7,8 execution in run_benchmark.sh to prevent default 1-hour timeout on large sweeps
- Add HINT diagnostic message when endpoint preflight fails with HTTP 503 or 'unconditional drop overload'
- Add TIP to SKILL.MD regarding GCP Gateway 30s Load Balancer timeouts on large multi-stage sweeps
… across all 3 well-lit path gateway modules - Override upstream llm-d Gateway default 300s timeout with 3600s (1 hour) for InferencePool services - Ensures large multi-stage and high-concurrency benchmark runs do not fail with HTTP 504 Gateway Timeout - Applies cleanly to optimized-baseline, precise-prefix-cache-routing, and predicted-latency-routing
…s tpu-v6e-2x4 across all three llm-d guides - Set TENSOR_PARALLEL_SIZE=8 in runtime.env - Set compute-class to tpu-v6e-2x4 in patch-nodeselector.yaml - Set TPU resource requests and limits to 8 in patch-resources.yaml - Applies to optimized-baseline, precise-prefix-cache-routing, and predicted-latency-routing
…d MAX_MODEL_LEN ceiling in tune_workload.py - Fix Kustomize JSON patch path to target container index 0 instead of non-existent container index 1 - Include prompt (input) sequence length in KV cache memory calculation (max_in + max_out) - Remove quantization calculation and FP8 fallback logic from automatic tuning - Replace hardcoded 32,768 ceiling on MAX_MODEL_LEN with 131,072 to support modern 128k context models
…model_specs.json max context ceiling and accurate KV cache sizing - Document 6-element tuple in model_specs.json hosting model max supported context length - Explain KV cache memory sizing against total sequence length (max_in + max_out) - Clarify Kustomize extra args injection targeting container index 0 (modelserver) - Update run_benchmark.sh warning message for agentic_code_generation long-context workloads
… ceiling to skills TROUBLESHOOTING.md - Document HTTP 504 Gateway Timeout symptom, 300s upstream default cause, and GCPBackendPolicy timeout resolution - Document VLLMValidationError / broken pipe symptom on long-context prompts, artificial ceiling cause, and model_specs.json max context resolution
…e and automate DCGM telemetry collection in run_benchmark.sh
…tecture and Managed Prometheus container accelerator metrics
…esolved runtime ConfigMap envs (TP, KV offloading) and well-lit path guide name
…ive node label architecture detection, scaling state, and multi-pass substitution
syeda-anjum
marked this pull request as ready for review
August 7, 2026 00:47
gushob21
reviewed
Aug 7, 2026
| # limitations under the License. | ||
| apiVersion: v1 | ||
| kind: Pod | ||
| kind: ConfigMap |
Collaborator
There was a problem hiding this comment.
highly recommend to make this a separate script :)
| use_kustomize = true | ||
| } | ||
|
|
||
| resource "local_file" "llmd_ob_gcp_backend_policy_override" { |
Collaborator
There was a problem hiding this comment.
this should be done via kustomization overlay.
| use_kustomize = true | ||
| } | ||
|
|
||
| resource "local_file" "llmd_ppcr_gcp_backend_policy_override" { |
Collaborator
There was a problem hiding this comment.
this should be done via kustomization overlay.
| use_kustomize = true | ||
| } | ||
|
|
||
| resource "local_file" "llmd_plr_gcp_backend_policy_override" { |
Collaborator
There was a problem hiding this comment.
this should be done via kustomization overlay.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Key Changes
telemetry-collector.yaml&run_benchmark.sh):kind: Podand inline multiline Pythonargs:with a KubernetesConfigMap(telemetry-collector-script) and a native batchJob(telemetry-collector,apiVersion: batch/v1).inference-perf -a) prior to GCS uploadcapture_vllm_config.py):spec.nodeName) to extract GKE NAP labels (cloud.google.com/compute-class: gpu-h100-80gb-high-x1andcloud.google.com/gke-accelerator: nvidia-h100-80gb)dep.status.readyReplicas) and detects activeHorizontalPodAutoscalertargets.