Conference SessionIntermediate60 min

A Practical Guide to Running LLMs on Kubernetes

Serving an LLM on Kubernetes has sharp edges. Includes a live demo deploying an open-weight LLM on Kubernetes and serving inference traffic with real-time GPU metrics. That is the entry point for "A Practical Guide to Running LLMs on Kubernetes", where Rohit Mishra shows how to handle GPU scheduling, model weights, health checks, autoscaling, and live inference in practice.

About This Session

The path from “I have a model” to “it’s serving production traffic on my cluster” is full of sharp edges. This talk is an end-to-end walk-through: GPU device plugins and node scheduling, model weight distribution strategies for 10–140 GB artifacts, health checks that survive multi-minute model loads, and GPU-aware auto-scaling. Includes a live demo deploying an open-weight LLM on Kubernetes and serving inference traffic with real-time GPU metrics.