Skip to content
#

serving-systems

Star

Here are 10 public repositories matching this topic...

JohnScheuer / request-cancellation-overhead-bench

Star 1

Discrete-event simulation benchmark for LLM request cancellation handling, measuring wasted compute, KV residency overhead, and survivor latency recovery across polling and push-based strategies.

performance-engineering benchmark streaming simulation latency inference systems throughput request-cancellation discrete-event-simulation kv-cache llm llm-serving batch-scheduling machine-learning-infrastructure serving-systems
  • Updated Jul 25, 2026
  • Python

manishklach / KVFabric

Star 1

Exploratory AI infrastructure project modeling semantic KV-cache orchestration, memory tiering, and HBM/CXL movement tradeoffs for long-context LLM inference.

compression sram dram dma hbm memory-hierarchy cache-management prefetching systems-architecture kv-cache runtime-systems cxl long-context llm-inference ai-infrastructure systems-research inference-systems memory-orchestration serving-systems accelerator-research
  • Updated May 19, 2026
  • Python

JohnScheuer / graceful-degradation-bench

Star 1

Discrete-event simulation benchmark for graceful degradation strategies in LLM serving: output capping, CoT disabling, model downgrade, and selective drop under progressive pressure.

performance-engineering benchmark multi-tenant simulation inference systems slo quality-of-service discrete-event-simulation kv-cache llm graceful-degradation llm-serving batch-scheduling machine-learning-infrastructure model-routing serving-systems overload-management
  • Updated Jul 26, 2026
  • Python

JohnScheuer / decode-preemption-bench

Star 0

Discrete-event simulation benchmark for decode preemption policies in LLM serving: when to interrupt a running decode to admit an urgent prefill, and whether checkpoint or recompute is cheaper.

performance-engineering benchmark simulation latency scheduling inference checkpoint systems decode slo fairness preemption discrete-event-simulation prefill kv-cache llm llm-serving machine-learning-infrastructure recompute serving-systems
  • Updated Jul 26, 2026
  • Python

JohnScheuer / moe-activation-memory-bench

Star 0

Monte Carlo simulation benchmark for MoE expert activation memory variability and admission control under non-deterministic routing — including routing skew amplification, top_k sweep, and policy comparison across uniform, skewed, and very_skewed distributions

performance-engineering benchmark simulation scheduling inference systems moe monte-carlo-simulation mixture-of-experts admission-control llm llm-serving machine-learning-infrastructure expert-routing serving-systems load-imbalance activation-memory routing-skew
  • Updated Jul 26, 2026
  • Python

JohnScheuer / inference-observability-bench

Star 0

Simulation benchmark for LLM serving observability design: which minimum set of signals detects all major failure modes before SLO violation, and what does each signal miss?

performance-engineering benchmark monitoring simulation latency alerting inference systems slo signal-analysis observability fragmentation mlops set-cover kv-cache llm llm-serving machine-learning-infrastructure serving-systems
  • Updated Jul 26, 2026
  • Python

JohnScheuer / capacity-planning-sim

Star 0

Discrete-event simulation for LLM serving capacity planning: how many GPUs for a p99 TTFT SLO under ShareGPT traffic, batching fragmentation, and autoscaling lag — with cliff analysis and three levels of planning rigor.

performance-engineering benchmark simulation latency capacity-planning inference systems slo autoscaling cost-optimization discrete-event-simulation kv-cache llm sharegpt llm-serving batch-scheduling machine-learning-infrastructure serving-systems gpu-capacity
  • Updated Jul 26, 2026
  • Python

JohnScheuer / sliding-window-attention-kv-bench

Star 0

Analytical benchmark for sliding window attention KV cache management: quality vs window size tradeoffs, SWA vs eviction comparison, prefix sharing interaction, and operational window recommendations across four attention distributions

performance-engineering benchmark simulation inference systems attention mistral eviction analytical-model kv-cache llm long-context llm-serving machine-learning-infrastructure sliding-window-attention prefix-sharing serving-systems kv-cache-management
  • Updated Jul 26, 2026
  • Python

JohnScheuer / model-ab-testing-sim

Star 0

Simulation benchmark for canary deployment strategies in LLM model updates, measuring safety, rollback behavior, bad exposure, and rollout speed across good, bad, and marginal candidate scenarios

performance-engineering benchmark simulation rollback inference systems ab-testing canary-deployment rollout model-deployment model-versioning mlops guardrails llm llm-serving machine-learning-infrastructure serving-systems serving-safety
  • Updated Jul 26, 2026
  • Python

Improve this page

Add a description, image, and links to the serving-systems topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the serving-systems topic, visit your repo's landing page and select "manage topics."

Learn more