Akash Kokare/Notes

Engineering note · 28 October 2026 · Birla AI Labs

Eleven model families on one GPU, and the promise we didn’t make

Before setting customer limits for our forecasting API, I measured what one GPU can actually serve. The numbers ruled out the target we had in mind.

One GPU is one slot

More workers don’t create more GPU. They queue for the same one, and each extra model process duplicates the weights in memory. Capacity planning starts from a single execution slot.

The target, against the measurements

The proposal was 50 maximum-size forecasts in under a minute. I ran the largest valid request for each model.

  1. Chronos-2 base cold18.5 s
  2. TimesFM 200M warm ×2620.4 s
  3. Yug cold, uncached27.0 s
  4. Chronos large cold43.1 s
  5. Chronos tiny warm ×10063.0 s
  6. TimesFM 200M cold120 s
  7. TabPFN-TS cold> 300 s
Largest valid request per model. Log scale; the line is the proposed 60-second target.

A single request could take two minutes, and one model timed out at five. We didn’t make the promise. Existing limits stay in place until there is evidence for a better number.

The memory guard that cried wolf

Admission control refuses a model load that would run out of memory. After a few loads it began refusing safe work. It was counting the kernel’s file cache, left behind by reading checkpoints, as memory in use. That cache is reclaimed on demand, and it was most of what the guard saw. The fix counts only memory that can’t be reclaimed.

Loading Yug the same way every time

One on-demand download of Yug, the lab’s model, failed in testing. So the checkpoint is now pinned to one version, verified by hash, and cached at deploy time. If the cache is missing, the service refuses to start instead of downloading mid-request.

Yug, after pinning and cachingTime
First request, model not loaded7.2 s
Next requests0.10 s

Nearly all of the cold time is loading. What decides capacity is which models stay in memory, not how fast one forecast runs.

Limits

← Akash Kokare Questions? Email me