I write about distributed systems, trading systems, and pragmatic software engineering. Originally published on my own site; this is the canonical source.
The Crash Window: Why I Picked Double-Counting Over Data Loss
A service I worked on processed inventory messages. Millions per minute. Each message carried a list of SKU changes. The service filtered, accumulated counts, and wrote them to a distributed cache. The message queue guaranteed at-least-once delivery. That guarantee is the problem. At-least-once means the queue redelivers a message if it thinks you did not process it. Crash during processing, ack timeout, partition rebalance. All trigger redelivery. Duplicates are not exceptional. They are the baseline. ...
Cache Invalidation: Three Patterns and the Cost of Getting It Wrong
A version-control service I worked on kept a small list in Redis: which version of the code bundle was sitting on the shared NFS volume. Every request checked the list first. Hit meant the file was local and the executor could load it. Miss meant a download from object storage. One morning the cleanup job ran. It deletes old versions when NFS fills up. It deleted v1.42 from NFS. The list still said v1.42 existed. The cache lied, and a lying cache is worse than a missing one. A miss sends you to the source of truth. A false hit tells you the file is there, the loader opens it, and you get an error three steps later with no clear culprit. ...
Where Distributed Systems Complexity Comes From
Three machines in a rack are still three machines. They become a distributed system when they have to finish one job together, and that requirement is where the trouble starts. On a single machine you get three things for free. Every thread shares the same memory, so a write lands and everyone sees it. One clock puts every event in order. One lock protects the shared state. Move that job across a network and all three disappear. ...