A complete inference runtime for open-weight large language models, enabling efficient execution through streaming weights, quantization, and memory-aware scheduling
machine-learning deep-learning numpy inference quantization inference-engine memory-mapping inference-optimization kv-cache ai-inference cpu-inference large-language-models llm model-runtime safetensors llm-inference gguf streaming-inference gguf-runner streaming-inference-engine
-
Updated
Aug 3, 2026 - Python