Stop twisting every dial: One open Inference tuning lesson
Devcon 8 · 5 Nov 2026 · 3:29
Aqsa Aqeel, a software engineer on an inference team at DigitalOcean, opens a lightning talk on tuning vLLM, the open source model server, when it has more than twenty settings and the temptation is to change them all. The case is a roughly 400-billion-parameter Qwen model on eight GPUs whose pods crash-looped under live traffic. Removing the AMD Aiter kernel library from the setup stopped the pods from starting at all. The recording ends as GPU memory utilization is introduced as the setting worth checking, so the lesson itself is not heard.
Speakers
Topics
From Devcon 8, hosted by Ethereum Foundation.
Transcript
Hi everyone, I am Aksa. I work as a software engineer at DigitalOcean, and I work on the inference team. So recently, we have been serving a trillion tokens per day. And when you work at that scale and you're on-call, there are so many problems that you face. And there are so many things that you can change. So at DO, we use open-serving vLLM. And when you have something like vLLM, there are 20-plus knobs that you can change. And the temptation is to just keep changing everything. But today, I will be talking about one on-call problem that I solved , and where to look in what scenario
that could maybe help you in such a situation. So, let's get into it. Now, I won't be showing you a benchmark table, but I will bring you up to speed with a scenario. So the victim here is a model, Qwen 3.5 , which had 400 billion parameters. It was served on eight GPUs spread across a single machine. So, we saw that the pod would come up, but under live traffic, it was crash-looping.
Like, everything would work fine, but then the pods would crash, like it happens in general with OOMs. So, we looked into the logs and found the culprit, that was Aiter. So, Aiter is AMD's fast kernel library. It's supposed to make the mathematical calculations faster, but it just wasn't working on this particular model. So we see an easy fix here, right? We just removed Aiter. Right? So after removing it, what we saw was, initially it was crashing under traffic , but now the pods were not even starting.
Right, when the pods would go towards completion. The GPU workers would just crash, and then it would restart, and then it would just go into a crash loop. So this brings us to a sort of confusion . Something that was supposed to help with math wasn't working on this model; we removed that, and now we were in a bigger problem. So the next thing that you see is with vLLM, you have multiple parameters and you have the temptation to just go and change it.
You would think of, you know, decreasing the concurrency, cutting off the context length, and there are so many things. You can keep trying and failing. And when you are on-call, you are in a situation where you just want to mitigate this, so you can look into the problem later. But there is one thing, the GPU memory utilization, which you never look at. Because, uh uh, every guide on the internet says to, uh, set it at 90, 90 or 95%, because you don't want to lose out on GPU hardware, because.