Forty-eight vCPUs, seven per cent busy
The back end of a language-learning app ran on twelve virtual machines, one per workload, each with 4 vCPUs and 16 GiB of memory. Each had been sized, when its workload was written, for that workload's own busiest moment: the pronunciation-scoring API at the evening peak in its largest market, the audio transcoder on the day a new course is published, the weekly progress reports on Sunday night, the streak reminders every evening, an in-memory lesson cache, and seven smaller services. A cost review averaged their CPU over a month and found 7 per cent. The company paid for 48 vCPUs and used, on average, between three and four.
The sizing was not careless. Each machine had to cover its own peak, and the peaks fell at different times. Compute Resource Consolidation puts that to use: run several workloads on shared compute, so capacity that one of them leaves idle is there for another. The number that matters is then not the average but the combined peak: the twelve workloads' usage added together minute by minute, and the busiest minute of that sum. The team measured 14 vCPUs, against the 48 that twelve separate peaks had required, and chose three hosts of 8 vCPUs, 24 in all.
Sharing can go as deep as one process, where tasks are modules of one program and share memory, crashes and every deploy, or stop at separate virtual machines on shared hardware, each with its own kernel, which is how cloud providers already consolidate. The team chose the middle: separate containers on a small Kubernetes cluster. There, packing works from numbers each workload declares. A request is the CPU and memory it reserves, and the scheduler places it only on a host with that much unreserved; a limit is the ceiling it cannot pass once it runs. How the scheduler does this belongs to Containers & orchestration; for consolidation, the measured peak shows how much the workloads use, but their requests decide how many fit on a host.
The transcoder next door
The first packing put workloads wherever they fitted, with no requests or limits set, and the transcoder landed on the same host as the scoring API. For a week nothing happened. Then a Japanese course was published, the transcoder started converting 3,000 recordings, and the scoring API's 99th-percentile latency rose from 120 ms to 900 ms for the learners using it at that moment. Nothing about the API had changed. The transcoder kept every vCPU busy and filled the host's disk with temporary files, and the API's requests waited behind it for CPU time and for the disk.
That is a noisy neighbour: a workload that degrades others on the same host by consuming what they share. Co-locating latency-sensitive and batch workloads on the same host, without resource limits or priorities, lets a noisy batch job starve request-serving tasks of CPU or I/O. The mix itself is fine once something arbitrates between them, and the team set three things:
- A request sized for the API's peak. When the CPU is contended, the kernel shares it in proportion to the requests, so the API gets roughly the share it reserved, whatever the transcoder wants.
- A limit on the transcoder. It stops at 4 vCPUs even when more are idle, which leaves headroom for the API's next burst.
- A lower priority, and preemptible batch. When the API scales out and the cluster has no room for a new replica, the scheduler evicts transcoder pods to make room, and their jobs run again later. A batch job built to be interrupted costs a delay, not a lost result.
The disk was the harder part. An ephemeral-storage limit evicts a pod that fills more local disk than allowed, but Kubernetes' standard resource settings cap neither disk I/O nor network bandwidth, nor the CPU caches that workloads on a host share. A job heavy on those can slow its neighbours while staying inside every limit, so the transcoder's temporary files moved to a disk of their own.
Memory does not stretch the way CPU does: a container past its memory limit is killed, and a host that runs out evicts or kills pods, starting with those using more than they requested. So a pool can lend idle CPU freely, but each memory request should be close to the workload's real peak.
The pool lends idle CPU to the transcoder and takes it back when the API needs it. The quiz runner, which executes code teachers upload, runs in a pool of its own.
One host, many outages
Sharing capacity also shares failure. When one of the three hosts failed a hardware check and was drained for repair, every workload on it restarted at once on the two hosts left. They had room because, before consolidating, the team had checked that the CPU and memory requests of the replicas each service needs fit on two hosts after each host's system reserves; 16 vCPUs against an observed peak of 14 would not have shown that. That is the arithmetic a consolidated pool has to keep with one host gone, and the replicas of each service should sit on different hosts, which a spread or anti-affinity rule tells the scheduler, or one host takes a whole service down.
Consolidation and Bulkhead pull in opposite directions: a bulkhead partitions resources so a failure stays in its compartment, consolidation removes partitions to save capacity, and every decision about what shares a host sits somewhere on that line. At the scale of a whole system, how many customers share one copy of the stack, the same trade drives Deployment Stamps.
Code you did not write, on the same kernel
The last candidate for the shared pool was the quiz runner, which executes small scripts that teachers upload to build custom exercises. It would have fitted easily. But containers on a host share its kernel, so a script that escaped its container through a kernel flaw would land on the machine that runs the scoring API, with its credentials and the learners' recordings. Consolidating workloads consolidates their security boundary too: on one host, the weakest workload's isolation is everyone's. The quiz runner went to its own pool of lightweight virtual machines, and Sandboxing untrusted code: isolation, limits & egress covers how that isolation is built.
Four questions before two workloads share
By the end, the team asked four things of each pair of workloads before letting them share:
- Do they scale and peak together? Then the combined peak is close to the sum, and sharing saves little.
- Do they deserve the same trust? Code nobody vetted, like the quiz runner's scripts, needs a pool of its own.
- Do they live as long? A long-running service holds its share of the pool all day. A job that runs five minutes a day leaves capacity idle only where capacity is held for it, such as a machine of its own; in a shared pool its request is reserved only while it runs, so short, scheduled work fits in the capacity long-running services leave free, at low priority, or on Serverless & edge functions, which bill per invocation and for the time it runs.
- Do their shapes fit together? A host runs out of whichever resource fills first, so the CPU-heavy transcoder and the memory-heavy lesson cache fill a host better together than two of either.
The Serverless Function Platform Challenge is consolidation under a strict brief: many tenants' functions on one shared fleet. Answer the four questions on the Grid: tenant code in lightweight VM sandboxes, because none of it is trusted; functions scaled to zero when idle, with a pre-warmed pool for the short-lived ones; and a per-tenant concurrency limit where invocations enter the fleet, written on that edge, so one tenant's burst cannot starve the others.