Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Pipes and Filters

Split a processing job into filters that each make one transformation and pass a message through pipes to the next, so steps can be reordered, reused or given more instances, held together by the message format they share.

Photo p-30551, start to finish

A landlord adds a kitchen photo to listing 8812 on a rental site. Before tenants can see it, the upload pipeline writes five log lines:

  1. scan p-30551: no malware found, 180 ms
  2. strip-metadata p-30551: GPS position and camera serial removed, 12 ms
  3. resize p-30551: 1600, 800 and 320 pixels wide, 1,900 ms
  4. watermark p-30551: 3 images stamped, 240 ms
  5. store p-30551: 3 files written, listing 8812 updated, 90 ms

Each line comes from a different program, and each program sees only the message handed to it. Eight months earlier the same work was one 600-line function, processListingPhoto, and the log had a single line: photo processed, 2,450 ms.

A swap that took two weeks

The complaint came from tenants browsing on phones. The site's logo was stamped on the full-size photo before it was shrunk, so on the 320-pixel thumbnail it was an unreadable smudge. The fix sounded like a swap: resize first, then stamp each size. Inside processListingPhoto it was not. Resize checked a photo.stamped flag that the watermark code set on a shared object, and used a rotation the metadata code had left in a local variable halfway down the function. Moving one block meant tracing every variable it touched and every later line that read one of them.

Filters, pipes and nothing shared

The team rebuilt the function as Pipes and Filters. Each filter does one transformation: it reads a message from its input, does its one job, and writes a message to its output. A pipe carries one filter's output to the next filter's input. A filter knows the message it receives and the message it produces, never which filter is on the other end, so the order of the steps lives in the wiring between them, not inside any of them.

The listing photo pipeline

Each filter is a Component, and each edge is a pipe, labelled with what it carries. The order of the steps exists only in these edges.

A pipe can be as plain as a value handed from one function to the next inside one process, or a queue between separate services, and which one you pick changes what the pipeline can do. Message Channel covers channels in their own right, and Subprocesses & reusable steps shows the same idea drawn as a flow diagram. Not every step has to be one message in, one message out: sending each message one way or another by its content is a Content-Based Router, and collecting related messages back into one is an Aggregator, each a step you can place in such a pipeline.

The message is the contract

With the steps apart, the swap became an edit to the edges. It still failed the first time. watermark had been written to receive one image, and after resize the message carries three. The filters no longer depend on each other's code, but each one depends on the shape of the message it reads, and that shape is the real contract of the pipeline. The team settled on one format for every pipe: the listing id, the photo id, a list of image files with their width and storage key, and a short record of the steps already applied. watermark now stamps every file in the list, however many there are, and from then on it can sit before or after resize without a change to its code.

Renaming a field in the format means changing every filter that reads it, the same reach as changing a function signature with many callers. When one step's output really is in another format, a filter whose only job is converting it keeps the others unchanged; that is a Message Translator.

Independent code, dependent order

A month later some photos of listing 8812 appeared on their side. Phone cameras often save the pixels in the sensor's orientation and record the correct rotation in a metadata tag, which viewers apply when they display the image. strip-metadata removed every tag, that one included, and resize, now downstream of it, had nothing left to read. No filter reached into another, and the pipeline was still wrong, because one step erased information a later step needed.

Filters you can reorder without editing them do not make every order correct. In this pipeline, scanning still has to come before any filter opens the file. For the rotation, the fix was to make strip-metadata apply the rotation to the pixels before dropping the tag: its one job became removing metadata without changing how the photo looks, and its output no longer depends on what runs after it. A filter whose output is complete in itself can move without breaking the filters after it, though the new order still has to give the result you want, as the watermark showed.

Watch outSharing state between filters brings the old function back. If resize writes its file list into a shared database row and watermark reads that row instead of the message, the pipe no longer carries everything: watermark also depends on that row and on when resize writes it, and moving either filter can break the other without a line of their code changing.

Where the 1,900 milliseconds go

resize takes 1.9 seconds, and the other four filters take about half a second together. While the pipeline ran in one process, one photo at a time, an agency uploading 300 photos of a new building waited for 300 resizes in a row.

Once the pipes are queues and each filter runs as its own process, the slow filter can get more instances reading from the same input pipe, each photo normally handled by one of them, while the fast filters keep one each. Four resize instances get through the agency's batch in roughly a quarter of the time a single resize instance needs, as long as there is CPU to spare and the filters after it keep up; otherwise the bottleneck moves downstream, to the first filter after it that cannot keep up. With pipes inside one process, the filters share its CPU and memory, and giving only resize more workers takes threads or tasks managed inside that process. Meanwhile the queue in front of resize absorbs the burst, and if photos keep arriving faster than it drains, that queue keeps growing; Backpressure & load shedding covers the ways to push back.

Parallel instances also give up the order of arrival. With four resize workers, photo 3 can be stored before photo 1, so the order of a listing's gallery has to travel in the message: the format now carries each photo's position. A queue that redelivers failures brings one more case, a truncated JPEG that makes resize fail on every attempt; after a few tries it belongs in a Dead Letter Channel, where someone can look at it.

Swapping two filters on paper

The Pipes and Filters Challenge asks for a source, filters that each do one transformation, the pipes between them, and filters that can be reordered or reused without changing each other, in at most six nodes: the upload form as a Client and the five filters as Components, each named for its one job, with the edges standing for the pipes; a Message Channel node makes a pipe explicit, at the cost of one filter. Then run the reorder test on your own drawing: swap resize and watermark. If only edges move, and no filter's name or label has to change, the drawing shows no filter tied to its neighbour, though the message each one reads still has to fit the new order. If a label such as "shrink the stamped image" has to be rewritten, that filter knows too much about the one before it.

Practice this