The 3 AM Page
Three services down. Three different failure modes. One maintenance window caused all of them.
The Scenario
The auth service is completely down. Three separate changes landed during a maintenance window: a TLS secret was re-created with double-base64 encoding, memory limits were tightened and are now OOMKilling pods under warm-up load, and a new NetworkPolicy is blocking egress to the Vault service that issues tokens. Each failure cascades into the next. You need to triage which fix to apply first, or you'll spend 45 minutes fixing in the wrong order.
What You'll Learn
How to triage multiple simultaneous failures and sequence fixes correctly
Identifying double-base64 encoding in Kubernetes TLS secrets
Why OOMKill and NetworkPolicy failures can look identical from the outside
Reading multi-container pod events when only one container is failing
Tools You'll Use
Real-World Context
Maintenance window incidents with multiple simultaneous changes are the hardest to debug because each symptom can explain the others. This scenario models the real pattern where teams fix one thing and the service still doesn't recover.
Ready to debug this?
Free account required - sign up with GitHub or Google in 10 seconds
Play The 3 AM Page