mixed · reliability
Production failures: the things that page you
One class of real production failure per week: how it presents, why it happens, how it is diagnosed and what prevents it.
·
By Amit Chakraborty — Kolkata, India · Remote worldwide
What you will be able to do
Recognise a failure mode from its symptoms and know the first three things to check.
Syllabus — 24 chapters
- 1. The three in the morning page — Read an unfamiliar dashboard under pressure and decide what to look at first.
- 2. Thundering herd: when the cache expires all at once — Recognise correlated cache expiry and apply jitter and request coalescing.
- 3. Retry storms — See how a well-intentioned client takes down a recovering service, and bound it.
- 4. Connection pool exhaustion — Trace requests queueing on a resource nobody is watching.
- 5. The slow query that only appears at scale — Find the query whose plan changed when the table grew.
- 6. Clock skew, timezones and the off-by-one-hour outage — Handle time correctly in storage, transport and display.
- 7. Certificate expiry — Understand why this still happens and how to make it impossible.
- 8. DNS: blamed often, guilty sometimes — Diagnose resolution problems instead of asserting them.
- 9. Memory leaks and the process that keeps restarting — Identify a leak from process metrics and confirm it in a heap profile.
- 10. Disk full — Trace the log, the temporary file and the retention policy nobody set.
- 11. Deadlocks and lock contention — Read a deadlock report and change the access order that caused it.
- 12. The deploy that only breaks for one percent — Find a failure that averages away in your dashboards.
- 13. Cascading failure and the circuit breaker — Stop one failing dependency from taking down everything upstream.
- 14. Silent data corruption — Detect wrongness that produces no error at all.
- 15. Backpressure and queues that grow forever — Make a system shed load instead of accumulating it.
- 16. Rate limits and the third party that changed — Survive a dependency changing its terms without warning.
- 17. Feature flags that became the bug — Manage flag state as production configuration with real risk.
- 18. The migration that locked the table — Change a large schema without taking the service down.
- 19. Split brain and the cost of consensus — Understand what a partition does to a distributed system.
- 20. Timeouts longer than the caller will wait — Set timeouts as a coherent budget across a call chain.
- 21. The rollback that could not roll back — Recognise irreversible changes before making them.
- 22. Cache invalidation, for real — Work through the actual hard cases rather than the joke.
- 23. The outage you could not see — Close observability gaps found the expensive way.
- 24. Writing the postmortem — Produce a blameless document that changes something.
0 of 24 published. A new chapter every Friday morning. Free to read.
Learn it free, elsewhere too
The documentation, videos, books and practice sites worth your time on this subject. All free. None of it is mine.