Reactor

One wake-up, as the kernel sees it

Six system calls, trimmed from an strace of one Linux process:

1. epoll_wait(5, …) = 2: the thread has slept until two of its sockets had something, and the kernel names those two.
2. accept4(3, …) = 9013: the first is the listening socket, so a client is connecting, and its connection becomes descriptor 9013.
3. epoll_ctl(5, EPOLL_CTL_ADD, 9013, …): the process asks to be told when 9013 has data.
4. read(812, …) = 64: the second is socket 812, holding 64 bytes.
5. read(812, …) = -1 EAGAIN: nothing more is waiting there.
6. epoll_wait(5, …): back to sleep until something else is ready.

The process is a telemetry gateway holding TCP connections to 20,000 field devices, each sending a 64-byte reading every 30 seconds, and one thread does all of its networking. No thread belongs to a device: a single call waits on all 20,000 device sockets and the listening socket, and what runs next depends on which socket woke it. That structure is the Reactor pattern, described by Douglas Schmidt in the mid-1990s. Why one thread is enough, and why its handlers must never block, is the ground Event Loop stands on; this article opens up the machinery under that loop.

Who does what in those six lines

Schmidt's description has four parts, and each line of the trace belongs to one of them:

• Handles are what the operating system can watch: here, socket descriptors such as 3, 812 and 9013.
• The synchronous event demultiplexer is the one call that blocks until at least one handle is ready and reports which: epoll_wait here, select, poll or kqueue elsewhere. It is synchronous because the calling thread waits inside it, and it is the only place that thread is meant to wait.
• The reactor, which Schmidt called the initiation dispatcher, keeps a registry of which handler wants which kind of event on which handle, calls the demultiplexer, and dispatches each reported event to its handler.
• Event handlers are the application's code: one that accepts connections on the listening socket, and one per device connection that parses readings.

Together they repeat one cycle: register interest (a handle, an event type, a handler), wait in the demultiplexer, dispatch each ready event, then wait again. Handlers may change the registry while they run. In the trace, the accept handler created socket 9013 and registered a read handler for it, so the next wait already watches 20,001 device sockets; a handler that sees a device close its end of the connection (its read returns 0) removes its registration and closes the socket.

Diagram components: Client, Client, Reactor, Handler, Handler.

The reactor waits on every socket in one call and dispatches each ready one to the handler registered for it. Accepting a connection ends with a new registration, not a new thread.

Ready is not done

What epoll_wait reported for socket 812 is readiness: the kernel's word that a read now would not block. The 64 bytes stayed in the kernel's receive buffer until the handler's own read copied them out.

First, readiness says nothing about message boundaries. TCP delivers a stream of bytes, so one reading can arrive as 40 bytes in one event and 24 in the next; the handler keeps a buffer per connection and parses only complete readings. Second, readiness is a hint. The Linux manual page for select warns that a socket can be reported readable and then have nothing to read, for instance when the data failed its checksum and was discarded. So the sockets stay non-blocking, and EAGAIN just means not yet: the handler returns and the reactor waits again.

Told once, or told until you act

epoll reports readiness in one of two ways, chosen per registration. Level-triggered, the default, behaves like select and poll: as long as 812 has unread data, every epoll_wait reports it again. Edge-triggered (EPOLLET) reports changes: it fires when new data arrives, not while data is still there. kqueue offers a similar choice through its EV_CLEAR flag.

The gateway's team switched to edge-triggered to cut wake-ups and kept their handler, which read at most 1,024 bytes per event. Then a firmware update made each device send a 3 KB status report after connecting and wait for an acknowledgement:

1. The device sends 3 KB, and epoll_wait reports socket 812 once.
2. The handler reads 1,024 bytes and returns, as it always had.
3. The other 2 KB stay in the receive buffer, but no new data arrives, so no new event does either.
4. The device waits for its acknowledgement, the gateway waits for an event, and the connection sits idle until a timeout closes it.

Under level-triggered the same handler works: the next wait reports 812 again. Under edge-triggered a handler must keep reading until EAGAIN, and the accept handler must keep accepting until EAGAIN, since several devices connecting between two waits can produce a single event. Draining has a cost too: a device that floods its socket can hold the thread while thousands of others wait. The usual answer is to read a bounded amount and keep the handle on the reactor's own list to revisit after the others, because the kernel will not report it again until more data arrives. Level-triggered forgives a handler that stops early, at the price of repeated reports; edge-triggered reports each change once, and a handler that stops before EAGAIN without revisiting the socket can stall its connection.

The socket that is always writable

With writes, level-triggered is the one that bites. The gateway pushes configuration changes to devices, and an early version registered every connection for writable as well as readable. The idle process then used a full core. A socket is writable whenever its send buffer has room, which for an idle connection is nearly always, so every epoll_wait returned at once with thousands of writable sockets and nothing to write.

So write interest is registered only while output is pending. The handler writes directly; if the write takes everything, nothing is registered. If it returns EAGAIN or writes only part, the rest goes into that connection's output buffer and writable is added to its registration, until the buffer is empty and the interest is removed again. A device that reads more slowly than the gateway writes makes that buffer grow, so it needs a limit and a rule for what happens at it, which is where Back-Pressure picks up.

20,000 sockets watched, two ready

select takes bitmasks indexed by descriptor number, and with glibc on Linux those masks end at FD_SETSIZE, 1,024, so descriptor 9013 cannot safely be passed to it at all. poll has no such ceiling, but both hand the kernel the whole list on every call, the kernel checks every entry, and the program scans the list again for the ready ones: 20,000 entries of work per wake-up to find two. epoll on Linux and kqueue on the BSDs and macOS split the work the way the pattern does: registration happens once, the kernel keeps the interest set, and a wait returns only the handles that are ready. Its cost then follows the ready events rather than the watched ones, which pays off most when, as here, nearly every connection is idle.

One reactor per core

A basic reactor runs its handlers on one thread, so it keeps at most one core busy. The common variant gives each core its own: a main reactor watches only the listening socket and hands each new connection to one of several sub-reactors, each with its own demultiplexer and thread, where the connection stays for its whole life. Each connection's handlers still run one at a time, and only state shared with other threads, such as other sub-reactors or a worker pool, needs protection.

Proactor: being told it is finished

The contrasting design, Proactor, starts the read itself, handing the operating system a buffer, and is told when the read has completed with the bytes already in that buffer, as Windows I/O completion ports (IOCP) and Linux's io_uring do. Completion also works for regular files, which select and poll always report as readable and epoll refuses to watch, so a reactor cannot stop a disk read from blocking.

The gateway, in ten nodes or fewer

Redraw the gateway for the Reactor Pattern challenge: three Clients whose readable and new connection Events lead to one Reactor, which dispatches to accept and read Handlers, with an edge from the accept handler to the read handler it registers. No Client gets a thread of its own; if the Thread Pool node appears, it hangs off a Handler that offloads slow work, and Proxy and Actor Model stay out, because an actor owning its state behind a mailbox is another design, Actor Model. The notification modes and the write-interest rule are not graded; they matter when the reactor is real.