Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Content moderation at scale

How a platform screens what users post: classifier scores split into three lanes per category, the uncertain kept hidden where exposure is costly, text and image verdicts combined, a human queue ordered by harm and bounded in time, and appeals and audits that measure wrong removals and missed violations and feed the models.

Nine and a half hours for a €380 iPhone

On a Saturday at 21:40, an account created that evening on a second-hand marketplace listed a sealed iPhone 15 Pro for €380, half what the same phone fetched elsewhere on the site. The first photo was a retailer's product shot; the description said bank transfer only. Like every listing, it went live the moment the seller pressed publish; screening ran afterwards.

The scam model scored the description 0.71. One rule served every kind of problem: 0.90 or more removes a listing, under 0.50 leaves it alone, anything between goes to the review queue. So it joined the queue, last, behind 6,200 items, mostly phone numbers pasted into descriptions and sofas listed under electronics. Reviewers took them in order of arrival. A reviewer opened the iPhone at 07:10 on Sunday and removed it in under a minute. By then it had 1,100 views, and four buyers had paid; the marketplace learned that from their support tickets.

The model, the queue and the reviewer each did their job. What failed was the design around them: what a score of 0.71 could cause, whether the listing should be visible while it waited, and which item a reviewer opens next.

What a score of 0.71 may cause

A classifier gives a score, a number that rises with the model's confidence that the content breaks a given rule. What to do with that range has a usual answer, three lanes, sized by confidence: high scores are removed automatically, low scores are published automatically, and the middle goes to a person. The economics of the system is mostly the size of that middle multiplied by what a review costs.

The two thresholds around the middle are a trade no setting avoids. At 400,000 listings a day, a middle of 6% is 24,000 reviews, about 440 reviewer-hours at 55 an hour. Lower the auto-remove threshold to shrink it and more honest listings are deleted unseen, the false positives; raise the threshold under which listings publish and more scams go up unexamined, the false negatives.

One pair of thresholds for every rule was the first mistake of that Saturday, because the cost of an error depends on the category. Removing an honest listing for a toy cap gun costs a seller one sale; leaving a real firearm up for an hour is harm of another order. A scam that takes payment off the platform costs buyers money within the hour; a phone number in a description costs little for each hour it stays up. So each category gets its own thresholds, tuned on its own labelled examples, and per surface if the audiences differ. And its own answer to the next question: may the listing be seen while it waits?

Hidden until the fast pass answers

Screening has to happen before publication for the categories where minutes of exposure are unacceptable, and can happen after for the rest. A pass before publication has the seller waiting on it, so it must be fast, cheap and hosted close to where the upload lands; a pass after it reads from a stream and can take longer, within the exposure its category tolerates. Many systems do both: a fast pass inline, a heavier one asynchronously, and a re-scan of live content when a model is retrained.

On the marketplace now, the photos land in object storage the public cannot read, the listing is saved as hidden, and the Ingest API acts on the fast classifiers' scores per category. Clean on every rule, it goes live; high on one, it is removed and the seller told why. In the middle of a high-severity rule, such as weapons or off-platform payment, it stays hidden until a reviewer decides; in the middle of a low-severity one, it goes live and is reviewed later. If the inline call times out, keeping the listing hidden until the asynchronous pass decides costs that seller a slower publish and exposes no buyer. Holding has a price: an honest seller in the middle band now waits for a person, so the queue's speed becomes a promise to sellers.

Scored inline, scored again, the middle to a person

The Ingest API keeps the listing hidden until the fast pass scores it, and records its automatic decisions. Every listing then reaches the deep pass with those scores; what is still uncertain, or held for a person, goes to reviewers, ranked by harm and reach.

Three photos and a description, one decision

On the iPhone listing, the hosted image classifier, a service such as Amazon Rekognition or Azure AI Content Safety that scores generic harms like nudity and violence, found nothing wrong with a phone in a box; scams are the marketplace's own rule, scored by its own model trained on its own labelled listings. Combining those verdicts usually starts with keeping them per category, judging each score against its own model's thresholds, and taking the strongest signal for each: averaging the text's 0.71 with three clean photo scores into one number would have pulled it down, though the photos said nothing about scams. Some signals only count together, such as half the usual price, an account a few hours old and a payment instruction: a job for rules over the combined signals, or for one model trained on all of them. And some checks are too slow for the seller to wait on. Reading text out of photos (OCR) would have found the third photo's handwritten WhatsApp number, put there because the marketplace already stripped phone numbers from descriptions; it runs in the deep pass behind the stream, which can still remove the listing, or send it to a person, minutes after it went live.

First in line by harm, not by arrival

A review queue without a time budget is a backlog with a nicer name. Give each category a target, minutes for weapons and off-platform payment, a day for a misfiled sofa, and order the queue by predicted severity and reach, not by arrival time. On a marketplace, reach is views and messages, and it grows while a live listing waits; every item's priority, hidden ones included, also rises as it nears its category's deadline. Ordered that way, the iPhone would have passed the pasted phone numbers long before morning.

Alarm on the age of the oldest item in each category, not on the depth of the queue: 6,000 misfiled sofas can wait a few hours, one weapons listing waiting 20 minutes is an incident. When a live item passes its deadline with no reviewer free, the system can hide it until someone looks, page more reviewers, or both. Then measure what matters: not how many items were reviewed, but how long violating content stayed visible, at the 99th percentile.

Watch outReviewer wellbeing is a design constraint: blur by default, cap consecutive exposure, and never route the worst categories, injured animals, explicit images uploaded on purpose, through a general queue. A design that burns out its reviewers has a throughput problem whether or not the diagram shows it.

The seller who says you got it wrong

On Monday a collector's listing for a 1970s toy cap gun is removed automatically under weapons, and the collector appeals. Without an appeal path the mistake would be permanent and invisible, reaching support as a churn ticket instead of the model as a label. Appeals also yield labels: an overturned decision is a candidate false positive, an upheld one a candidate violation, and either becomes training data once a check of its review agrees. Route each appeal to a more senior reviewer, never the one who made the call, since the same reviewer re-reading their own call is not a second opinion, and feed the checked labels back into training. Without that loop, the classifiers fall behind as the adversaries keep moving: the handwritten WhatsApp number was itself an answer to a filter.

Many wrongly removed sellers leave without appealing, so appeals undercount mistakes; the share of automatic removals that were wrong comes from reviewers re-checking a small random sample of automatic removals each day, per category. A random sample of automatically published listings per category, checked the same way, finds the violations that went live unflagged, the false negatives. Store the score, the model version and the reason with every action: when a new scam model is rolled back, that record says which removals came from it. Model serving: inference, versions & safe rollout covers releasing versions safely.

Count the hours a listing stayed up

Open the Content Moderation at Scale challenge and draw the three lanes explicitly, auto-remove, auto-publish and human queue, with the thresholds on the edges between them and a note on which categories stay hidden while they wait. Beside your Grid, write the four numbers you would watch: how long violating content stayed visible at the 99th percentile, the oldest item's age per category against its budget, the share of content in the middle lane, and how often each automatic lane was wrong, per category. Comment System is the cheap-content, high-volume version of the problem, and Photo Sharing Feed puts the screening pass on an upload path that also makes thumbnails.

Practice this