Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built a...
- Rust
- Web Servers, Proxies & Databases
- Apache-2.0
- #ai
- #llamacpp
- #llm
1.7k stars
Last commit