Real systems don't fail politely. They fail at 2am, under a retry storm, with cold caches, while one availability zone is dark. But the way we usually evaluate architecture — in design reviews and in interviews — is by looking at the sunny-day diagram. RodGrid's stress presets and diagnosis challenges exist to close that gap: break the design on purpose and see who can diagnose it.
Twelve ways to hurt a healthy system
The traffic simulator accepts a stress scenario on top of whatever load you dial in. Grid-wide presets hit a whole class of nodes at once:
- Traffic surge ×3 and Flash crowd ×10 — three or ten times the traffic you set: can the design take the spike on headroom, or does it need queues, load shedding or autoscaling?
- Cache stampede — every cache misses at once and the stores behind them inherit the read load.
- Cold cache — the everyday post-deploy version: 30% of the normal hit rate.
- AZ loss — the tiers you run lose the replicas they had in the lost zone, and cross-zone routing adds latency.
- Slow 3rd-party — your external dependencies all have a bad day at once; the lesson is isolation.
- Retry storm — failures breed retries breed load, the feedback loop that turns a blip into an outage.
Node-targeted presets pin the failure to one component you pick:
- Node loss — it goes dark entirely.
- OOM kill — it keeps running out of memory and restarting: capacity collapses, latency spikes, errors climb.
- Thread-pool exhaustion — requests queue behind blocked workers.
- Network partition — about half its calls fail, after burning timeout latency.
- Hot partition — one skewed shard takes most of the load.
Diagram components: Web App, Load Balancer, Application Service, Redis, PostgreSQL.
A healthy read-heavy stack. Open a Grid like this in RodGrid, turn on the simulator, and toggle Cache stampede — the database inherits every read the cache used to absorb; watch whether it, or something else, reaches its limit.
Diagnosis challenges
Some challenges hand you a system that is already failing: a seeded starter design, a stated SLO, and an armed stress scenario. Your job is the on-call job — figure out why it's failing under that scenario and redesign it until the SLO holds. When you submit, the server rebuilds the scenario and re-runs the simulation on your final Grid, so the grade reflects what your design actually does under fire, not what a description of it claims.