The fundamentals of distributed systems, before I write any code
To kick this series off, it makes sense to get a broad understanding of the fundamentals of distributed systems before I jump into coding and playing around. Normally I learn best by getting my hands dirty, but there are a lot of complex moving parts and new concepts here, so a high-level understanding first is probably the right call.
What I’m learning from
After some research, and with Claude’s help designing a learning plan, these are the materials I’m leaning on to get started:
- Fundamentals of Distributed Systems, Michael Perry, on Pluralsight.
- MIT 6.824 Distributed Systems, Professor Robert Tappan Morris, on YouTube.
Where I’m starting from
Truth be told, I do have some experience with the basics. While looking for permanent roles I’ve come across a lot of technology I hadn’t heard of before, and naturally I had to research it. When I apply to a company I’d really like to work for, I like to investigate them and build a small mock-up of something relevant to their domain using some of their technology, and submit it with the application. A little initiative goes a long way.
Because of that I’ve built applications on messaging technologies such as Azure Service Bus and RabbitMQ, and the concepts they introduced (idempotency, retries, dead-letter queues, observability and telemetry) are ones I feel I understand well now. It’s still best to get into the weeds of the specifics.
So, as things stand today, I’ve watched Michael Perry’s course and the first few MIT lectures. Combined with what I already knew, they’ve highlighted the following fundamentals as I see them right now. I’ll likely write a post on each one in more detail, with the specific technology I’ve been exploring and a demo solution, but here they are in brief.
Why distributed systems to begin with?
Performance. The most obvious reason is to harness the power of multiple computers rather than running everything on a single machine. Running operations in parallel means higher throughput, faster applications and less time users spend waiting on us. It also means we can scale services horizontally quite easily.
Fault tolerance. Having multiple instances of the same application running means that if one instance dies, another can pick the work up straight away and the service carries on. Combine that with data replication and we’re resistant to a lot of errors.
Why not distributed systems?
Firstly, they’re complex. Designing, connecting and maintaining them is tough work and needs a lot of up-front thought.
Secondly, the network is unreliable. On a single machine a method call either works or throws an exception. Across a network, a request can be lost, delayed, or succeed without you ever hearing back, and you can’t always tell which. A huge amount of distributed systems design comes down to handling that uncertainty, which is exactly why idempotency comes up so often.
The ideas that keep coming back
Idempotency
One of, if not the most important concept. An operation run twice should produce the same result and not duplicate data. A common way to enforce this across a distributed system is a client-generated id. Without one, a retry goes like this:
- An application calls a server, say a banking service, to make a payment.
- The request reaches the server, but midway through the service dies.
- The client never heard back, so it doesn’t know whether the payment succeeded, and retries.
- It turns out the payment was made; the server just crashed before replying. The payment is now duplicated.
With a client-generated id:
- When sending the original request, the client generates a unique id, sends it with the payment details and keeps it locally.
- The request reaches the server and, once again, the server dies.
- This time the client knows the id it generated, so when it retries it sends the same id again.
- The server checks whether a payment with that id already exists. If not, it creates it. If it does, it returns the existing payment. No duplication.
Immutability
Once data is written, you don’t change it; you add new facts. A bank is a good example again. Rather than storing a single balance and overwriting it each time money moves, you store every transaction as a record that never changes, and the balance is just the sum of those records.
This really matters in distributed systems because data that never changes is safe to copy, for example when replicating across servers for redundancy. It also gives you a full audit history for free, which is great for debugging, and it pairs nicely with idempotency: if a retry tries to add a record that already exists, it’s easy to spot and ignore.
Consistency, strong and eventual
This one is a direct side effect of replication. Once you’ve got copies of the same data on multiple machines, you have to ask: if I update one copy, when does everyone else see the change? There are two broad answers.
- Strong consistency. Every read sees the latest write, whichever copy you ask. Simple to reason about, but every write has to wait for the copies to agree, which makes it slower and more fragile when parts of the network are struggling.
- Eventual consistency. Copies are allowed to be briefly out of date, but they all catch up. Much faster and more available, but for a short window two people might see different answers.
Banking is a nice example yet again. Your account balance really needs to be strongly consistent; nobody wants to spend the same £50 twice because two servers disagreed. But your spending insights, or a “you’ve spent £X on coffee this month” summary, can happily be a few seconds behind and nobody will notice.
Observability
This is something I didn’t appreciate until I realised just how many moving parts a distributed system has. In a single application, if something goes wrong you check the logs and step through the code. In a distributed system a single user request might pass through five or six services, so the question becomes “where did it break?”.
Digging into this with OpenTelemetry, it’s usually split into three parts:
- Structured logs. What happened inside a single service, in detail.
- Metrics. Numbers over time: request rates, error rates and response times.
- Traces. Following a single request across every service it touches, using a correlation id passed along with each call.
Traces really hammered the point home. Seeing a request flow from service to service on a timeline, with the slow or failing step highlighted, turns what would otherwise be guesswork into something much easier to debug. I’ve played with this in the .NET Aspire dashboard, which gives it to you out of the box, and that naturally led to Azure Monitor, Prometheus and Grafana, which I’m sure I’ll get to throughout the series.
The technologies I’m about to get my hands on
Messaging platforms
This was quite a cool one. I’d heard of these but hadn’t really understood their role. The key idea is that a publisher sends a message to the platform and simply carries on with its work rather than waiting for a response. From playing with Azure Service Bus and RabbitMQ, the big wins are:
- Decoupling. The sender doesn’t need to know or care who’s listening, or whether they’re online right now.
- Fan-out. One event, such as “payment completed”, can trigger notifications, analytics and fraud checks at the same time.
- Resilience. If a consumer goes down, the messages wait for it instead of failing.
Kafka is the one I’m exploring now, and it works a little differently. Posts on that as I play around more.
Containers and Kubernetes
I’d used containers before and understood the benefit (no more “it works on my machine”), but I’d never really looked at Kubernetes. After a surface-level dive I now see the power of it. Kubernetes manages all of your containers for you, with the key idea being desired state: you tell it “I want three copies of this service running” and it constantly works to make that true.
If a container crashes it restarts it. If a machine dies it moves the work elsewhere. When you deploy a new version it can roll it out gradually. That’s a lot of the fault tolerance from earlier, handled for you.
RPC
This sounded scarier than it was, until I realised I’d used it before in my .NET WCF days. The idea is an abstraction over the network that lets one service call another as if it were a local method, without building requests by hand. Instead of constructing an HTTP request, sending it and parsing the response, you call something like accounts.GetBalance(id) and generated code handles the rest.
The catch is that it’s a leaky abstraction. It looks like a local call, but it can be slow, time out, or leave you unsure whether it actually ran. So you still need timeouts, and retries are only safe if the operation is idempotent. A theme is emerging.
gRPC is the most popular framework for this today. You define your services in contract files, it generates the client and server code for you in basically any language, and it sends compact binary messages, which makes it a lot faster than JSON. That matters for high-throughput services.
What has stuck so far
- The network is unreliable, so design for requests that are lost, late or silently successful.
- Idempotency, with client-generated ids, is the answer to almost every retry question.
- Immutable records are safe to copy and give you an audit trail for free.
- Consistency is a trade-off you choose per piece of data, not a setting you turn on.
- You can’t debug what you can’t observe: logs, metrics and traces, from the start.
There are likely more important parts I’ve had initial exposure to, but these are the ones that have stuck in my head so far. In future posts I’ll dive deeper into each of them, to show my understanding and to document the learning so I can come back to it later.