Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

GPU & accelerated compute

Why model training and inference run on GPUs and other accelerators, why memory and batching decide how much work one device does, how latency limits that batching, how to scale a GPU pool whose new machines take minutes to arrive, and what one request really costs on it.

Why is the GPU 30% busy when the queue is full?

A contact-centre software company transcribes its customers' recorded calls so supervisors can search them, and promises each transcript within 15 minutes. As shifts end and recordings pour in, the on-call engineer asks in the team channel: 1,800 calls waiting, the GPU dashboard says 30% busy, and the autoscaler wants more GPUs. Why are we buying GPUs we don't use?

Each of the eight GPU workers runs one loop: download a recording, decode the audio on the CPU, cut it into 30-second segments, send one segment to the speech model, wait for the text, send the next. The 30% is the figure most GPU dashboards show by default, the one nvidia-smi reports: the share of time during which any work was running on the GPU. For the other 70% it ran nothing, waiting on downloads and decoding. And when it did run, one segment kept only a fraction of the chip busy, which that figure cannot show: 100% means something was running all the time, not that the whole chip was.

What the chip is good at

A speech model, like most neural networks, spends its time multiplying large matrices: the audio's features against the model's learned weights, layer after layer. A server CPU has dozens of cores built for varied, branching work. A GPU has thousands of simpler arithmetic units that apply one operation to many numbers at once, with its own fast memory, VRAM, beside them. Google's TPUs and AWS's Inferentia and Trainium are built for this arithmetic alone; accelerator covers them all. Training and large-model inference, running the model on an input, are far faster on them than on CPUs.

They are also scarce and expensive, so accelerated compute gets a pool of its own. The upload API stays stateless and cheap and scales on request rate within seconds; the GPU workers scale on their own signal. Put both in one autoscaling group and they pull against each other: a CPU target adds GPU machines when the upload API is busy, and adds none while recordings pile up, because a worker waiting on its GPU shows little CPU.

One segment at a time wastes the chip

The first fix is to stop starving the GPU: CPU workers download, decode and cut recordings ahead of time and keep ready segments waiting, so the GPU never idles while a file downloads.

The second is a batch: 16 segments in one call instead of one. For much of a model's work, the time goes to moving the weights from VRAM to the arithmetic units, and a batch reuses each weight it fetches for all 16 segments. In this service's measurements, which are illustrative, one segment took 0.2 seconds and 16 took 0.6: five times the throughput from the same device. That holds up to a point; once the arithmetic units are full, a bigger batch takes proportionally longer. With both fixes the dashboard reads above 90% and the end-of-shift queue drains inside the promise. Without batching, a model like this one stays far below what the hardware can do, which is why a GPU per request is so expensive: each device works one input at a time and idles between them.

Keep the GPU fed

CPU workers prepare segments ahead, the speech model runs them on the GPU pool in batches, and the autoscaler sizes that pool from the queue, not from CPU.

Memory sets the ceiling

Why not batch 256, then? Because the batch has to fit in VRAM. The weights come first: 1.5 billion parameters at 16 bits each take about 3 GB. Each segment adds its own working memory, every layer's intermediate results and the text decoded so far, and the runtime takes a share. The batch you can run is what remains after the weights, divided by what one segment needs, which assumes segments of similar length, as these 30-second ones are; leave a margin and check it on real inputs, because a batch that does not fit does not run slowly: it fails with an out-of-memory error and has to be retried smaller.

So memory, often before arithmetic, decides which GPU a model needs. Data-centre cards range from about 16 GB to well over 100 GB, and the larger ones cost more per hour. A model too big for one card is split across several, which then exchange data on every pass, or quantized to fewer bits per weight, with an accuracy cost to measure. Large language models are tighter still, since each request's working memory grows with every token generated; whether to host one yourself is weighed in Serving LLMs: gateways, fallback & token budgets.

A live caption cannot wait for a full batch

Then the company adds live captions: a supervisor listening to a call wants the words on screen within about a second. A caption stuck behind 16-segment batches and an 1,800-recording queue is useless.

Batching trades latency for throughput: bigger batches, and longer waits to fill them, get more audio through each GPU while every request waits longer. So the trade is made per kind of traffic. Bulk recordings have 15 minutes of slack and get large batches. Live captions get chunks of a fraction of a second, small batches and a batching window of a few milliseconds, the dynamic batching that inference servers offer as configuration (Model serving: inference, versions & safe rollout), on GPUs of their own or ahead of bulk work. They also need headroom, because waiting climbs steeply as a device nears fully busy, as Servers, not utilization, decide the queue shows. The bulk pool can run hot because its queue is its buffer; the live pool cannot.

A pool that takes minutes to grow

For the bulk pool, the autoscaler's signal is queue depth: the minutes of audio waiting, divided by what the pool transcribes per minute, says how long the current backlog would take if nothing new arrived. Recordings keep arriving, so pair it with the age of the oldest waiting recording: grow the pool before that age, plus a cold start and the transcription itself, can reach 15 minutes, and whenever recordings arrive faster than the pool transcribes them, since then the backlog only grows.

A new GPU worker is not ready in seconds. The cloud must find a GPU machine, if the region has one free; it boots, pulls a container image carrying gigabytes of GPU libraries, and loads the weights from object storage into VRAM. That cold start often takes minutes. Sizing the whole GPU pool to the end-of-shift peak wastes money the rest of the day, and starting from zero misses the wave. Keep a warm baseline plus queue-depth autoscaling: new GPU capacity can take minutes to arrive (or be unavailable), so the baseline must absorb bursts. A scheduled scale-up before a known wave and images with the weights already on disk shorten the gap; idle replicas above the baseline scale down when the queue empties.

Bulk work can also run on spot (or preemptible) GPUs, discounted capacity the cloud can take back at short notice. A reclaimed worker's recording runs again when the queue redelivers messages nobody acknowledged and the worker acknowledges only after storing the transcript, so writing a transcript twice must be harmless, as Message queues & async workers explains; live captions stay on the on-demand baseline.

Paying by the hour, selling by the minute

A GPU is billed per hour whether it works or waits; the company is paid per minute of audio. Say, as an illustration rather than a price quote, a GPU worker costs US$2 an hour. The first design transcribed about 45 hours of audio per GPU-hour, over 4 cents per audio hour; batched and fed, about 700 hours, under a third of a cent. But calls arrive in the working day and the baseline runs all night.

So compute this for your own workload: the GPU's hourly price, divided by the work one GPU completes per hour at full batches multiplied by the share of that capacity actually used over the day. Here, US$2 ÷ (700 × 0.4) is about 0.7 cents per audio hour. Measure throughput at the batch size your memory allows, take that share from a week of real traffic, add the pool's other running costs, and compare with what a hosted model charges for the same audio: your own pool wins only when it is busy enough. A re-uploaded recording answered from its stored transcript, found by a hash of the audio and the model version, costs no GPU time at all.

Then build that pool in the Self-Hosted LLM Inference Challenge. Its models generate text rather than transcripts, but it asks for the same mechanics: GPU servers loading open-weight models, batching to keep utilization high, and a GPU pool autoscaled on queue depth and latency that scales idle replicas down. On your Grid, write on the Autoscaler its signal and its warm minimum, on the GPU Compute node the batch size its memory allows, and on the edge that loads the weights how long a cold start takes.

Practice this