What Actually Happens When You Send an LLM Request to Kubernetes?

Transcript

[music] [music] >> Hello, hello, hello everyone. I hope you all are awake like the So today I'm going to talk about what happens what actually happens when you send an LLM request to Kubernetes. So is there anyone who has deployed open weight models on Kubernetes or their own private infrastructure? So >> Wait it's loading.

>> Okay. Okay. So little bit intro about myself. So I'm Atharva. I work as an AI and infra engineer at CLD9 AI. So earlier this year I was Google Summer of Code Mente at org called Open Science Lab. Right now I am an open telemetry community member and I work on lots of and CNCF force projects. Some of it includes like open everts. So let's get started.

So if you have ever deployed open weight models on Kubernetes or any cloud hosted GPUs, you must have got the problem where the platform team ship LLM features fast. But the production ask basic question where does the request really go? Let's say if you have an your app let's say it's a chat app you send a post request to route slash V1 slash chat slash completions. But you you didn't get the response. Then you didn't know what actually break. Is it a DNS, gateway, autoscaler, or >> Direct.

>> is it a pod? So it happens in a like errors like 404 not found, something like 504 which is a gateway timeout and sometimes 409 which is like a too many request. So this is how it breaks. It is never about one pod really. It happens in a multiple steps. So when an request goes from an client or your SDK, it first goes through your DNS service, then the gateway which can be a either HTTP route or the model proxies which we have some projects like agent router which is an envoy AI gateway project. Then it

finally reaches your model pod which the model has the weights which is either deployed on VLLM with the help of GPUs or Olama on the CPU. So it never happen like the request goes from this all steps and it breaks either in one of the steps. So what's really a solution? Like as you have seen here, we have errors like 404 error at the gateway, 504 error when the model doesn't come up live. It's cold start issue. Then we have sometimes a 409 error where where we have counted

the number of request but not the number of tokens. So how to fix that? So has the So has the project called provider caser which I was working on. So at Open Everest there is a provider SDK where you can deploy pretty much every data plane and we control it using our control plane. We call it an instance. You just tell us what's your desire like I want to deploy an LLM model. We convert that instance into a controller using provider KServe. Then it reconciles into either LLM inference service or the another inference service which has

ML models. Then the KServe controllers handle all the deployments, HPAs, services, routes. So you just declare the intent what you want and you get an endpoint back which you can pretty much send a curl request, send an UI request or API request. So it's fine if you don't use KServe. We can use KubeAI where the gateway can be a ingress or hand rolled where you can use Nginx. If you use model routing, it can be either through proxy or inference pool endpoint peeker.

You can also have scale from zero like using the projects called KNative or Keda. So if you have custom hand rolled Kubernetes, you can use HPA with minimum of one replicas. So the model pods pretty much stays the same like vLLM or Olama in the case of KubeAI. So yeah, from now onwards if an LLM request fail, you have to map the entire path, not the brand.

So yeah, that's my >> [music]