Conference SessionIntermediate60 min

Building an AI Inference Gateway on Kubernetes: Model Routing, Rate Limiting & Cost Control

Running many AI models without a gateway creates duplicated auth, weak rate limits, and hidden costs. We cover model routing with A/B traffic splitting, token-based rate limiting per team, fallback chains, and per-team cost attribution. "Building an AI Inference Gateway on Kubernetes: Model Routing, Rate Limiting & Cost Control" turns that scenario into a session where Rohit Mishra walks through gateway patterns for routing and control.

About This Session

Running multiple AI models on Kubernetes without a unified gateway means duplicated auth, no rate limits, and opaque costs. This talk builds an inference gateway from scratch using Envoy, Kubernetes CRDs, and Prometheus. We cover model routing with A/B traffic splitting, token-based rate limiting per team, fallback chains, and per-team cost attribution. Includes a live demo deploying the gateway and routing traffic across models in real time.