Eleven model families on one GPU, and the promise we didn’t make
Before setting customer limits for our forecasting API, I measured what one GPU can actually serve. The numbers ruled out the target we had in mind.
One GPU is one slot
More workers don’t create more GPU. They queue for the same one, and each extra model process duplicates the weights in memory. Capacity planning starts from a single execution slot.
The target, against the measurements
The proposal was 50 maximum-size forecasts in under a minute. I ran the largest valid request for each model.
A single request could take two minutes, and one model timed out at five. We didn’t make the promise. Existing limits stay in place until there is evidence for a better number.
The memory guard that cried wolf
Admission control refuses a model load that would run out of memory. After a few loads it began refusing safe work. It was counting the kernel’s file cache, left behind by reading checkpoints, as memory in use. That cache is reclaimed on demand, and it was most of what the guard saw. The fix counts only memory that can’t be reclaimed.
Loading Yug the same way every time
One on-demand download of Yug, the lab’s model, failed in testing. So the checkpoint is now pinned to one version, verified by hash, and cached at deploy time. If the cache is missing, the service refuses to start instead of downloading mid-request.
| Yug, after pinning and caching | Time |
|---|---|
| First request, model not loaded | 7.2 s |
| Next requests | 0.10 s |
Nearly all of the cold time is loading. What decides capacity is which models stay in memory, not how fast one forecast runs.
Limits
- Most rows are single runs, not tail latencies.
- Measured in an isolated copy of production, not under real traffic. A new image or model version needs fresh numbers.