Selective Consumer

Can the GPU nodes skip the thumbnails?

Someone asked that in the render farm's team channel on a Monday. The studio's farm takes all its work from one queue, render-jobs, which carries two kinds of job. Thumbnails are preview frames: CPU only, about 2 seconds each, thousands a day. Print frames are large-format renders for posters: each needs a 48 GB graphics card and runs about 6 minutes. Twenty CPU nodes and four GPU nodes take from the same queue, and the queue hands each job to whichever node asks next, which is Competing Consumers at work.

A GPU node would spend an afternoon on thumbnails while 30 print frames waited. Worse, a CPU node would take a print frame, run out of memory 10 minutes in and crash; the job went back on the queue because it was never acknowledged, and the next node to take it could be another CPU node. One fix is two queues, which comes up again below. The other keeps one queue and lets each node say what it will take.

A condition the broker checks for you

The submitter now stamps every job with a property, tier, set to gpu or cpu. Each GPU node connects to render-jobs with the condition tier = 'gpu', and each CPU node with tier = 'cpu'. That condition is a selector: the consumer hands it over when it starts listening, and the messaging system, not the consumer, evaluates it and offers the consumer only the messages for which it is true. A consumer that works this way is a Selective Consumer.

A thumbnail is never offered to a GPU node, and it is not lost either: it stays on the queue until a CPU node takes it. That is the difference from a Message Filter, which is a step in the flow with one output that throws away whatever does not match, so nothing after it ever sees those messages.

The tier has to be on the envelope

The selector reads tier, not the job's contents. In JMS, the Java messaging API, a message selector is written in a subset of SQL's conditional syntax and may refer only to header fields and properties, never to the body. Azure Service Bus subscription filters likewise evaluate a message's properties. The broker decides without parsing payloads, so the check works whatever the body's format.

The cost lands on the sender. The memory requirement used to live only in the job's JSON; now the submitter copies it into tier, and the two copies can disagree: a print frame stamped cpu by a buggy submitter goes to a CPU node and crashes it, because the selector believes the property. Some services, Amazon SNS subscription filter policies among them, can also match fields inside a JSON body; where they cannot, a fact that lives only in the body is invisible to every selector.

On a queue, the selector picks from one shared pile: a print frame one GPU node takes is gone for the other three. On a publish-subscribe topic, where each subscription gets its own copy, the same kind of condition, usually called a filter or a rule (Azure Service Bus attaches rules to each subscription), decides which messages enter a subscription at all; Message Channel sets the two kinds of channel side by side.

Forty frames nobody selected

In March the studio won a billboard campaign. Its frames need more than 48 GB, so the pipeline team gave them a new tier, gpu-xl, for two new machines with 80 GB cards that would select tier = 'gpu-xl'. The submitter change shipped on a Tuesday; the machines slipped a week.

Forty billboard frames went onto render-jobs and stayed there. No running consumer's selector matched them, so none was offered them. Every node stayed busy with its own tier. The only sign was a queue depth that never dropped below 40, on a queue whose depth rises and falls all day.

On a queue, a message no consumer selects is not rejected; it waits, until someone looks or until its time-to-live runs out. Then, depending on the broker and the queue's configuration, it is discarded or moved to a Dead Letter Channel. On a topic it can be quieter still: a message that matches no subscription's filter may reach no one and leave nothing to find.

The farm added three defences, and each has a catch:

• An alarm on the age of the oldest message, not only on depth: a job that has waited an hour while consumers keep taking others is the signal.
• A catch-all consumer whose selector is the complement of the others, alerting on anything it receives and setting it aside for resubmission. Its selector names every known tier, so it must be updated with each new one, and once gpu-xl is added it no longer sees those jobs: in March it would have taken none of the forty frames, so a known tier with no running consumer is left to the age alarm. It must also catch a job with no tier. In JMS, an absent property is null, a comparison with null is neither true nor false, and a message is delivered only when the selector is true, so tier NOT IN ('gpu', 'cpu', 'gpu-xl') alone misses such a job. The catch-all needs tier IS NULL OR tier NOT IN ('gpu', 'cpu', 'gpu-xl').
• A check at the sender: the submitter refuses a tier not on the list of known tiers. That catches a typo or a forgotten property, but not the March case, where gpu-xl was valid and its machines were not running yet.

When two queues are simpler

Selectors also cost the broker work: each message may be checked against the selector of every consumer that could take it. On a queue, when the messages at the head match no waiting consumer, the broker has to search further along, and in some brokers a large enough block of unmatched messages leaves consumers idle while matching ones wait behind it. Selectors behave best when the kinds are few, the conditions are simple comparisons on a property, and every message has someone ready to take it.

The farm could instead have run two queues, render-thumbnails and render-print: each with its own depth to watch, so a backlog shows which kind of job is stuck. The price is that the submitter must know which queue each job goes to, and a new tier means a new queue to create and wire up. A Content-Based Router takes that decision out of the submitter: everything goes to one channel, and the router reads each job and forwards it to exactly one destination, with the routing rules in one place.

Selective consumers fit when the kinds are few and consumers' interests change more often than the senders do. Separate channels, with or without a router in front, fit when the kinds are stable and each deserves its own capacity, alarms and retention.

Three questions before the first selector

• Does your broker evaluate selectors at all? A Kafka consumer reads every record of the partitions assigned to it, and an Amazon SQS queue has no per-consumer condition; there, use separate topics or queues, or filter in the consumer after delivery, where nothing steers a skipped message to a consumer that wants it the way a selector does.
• Is the deciding fact in a property the sender sets, or only in the body?
• Which consumer takes each value of that property, and who hears about a value nobody takes?

No challenge uses the Selective Consumer node yet. The Content-Based Router challenge asks for the design on the other side of this choice, a router that sends each message of a mixed channel to exactly one destination, so the third question turns into a requirement there: put the render farm through it with gpu-xl added, and the router must have a route, or a default, for that tier.