The Volume That Won't Let Go
Database replica is down. The persistent volume is stuck on a node that no longer exists.
The Scenario
A spot instance was reclaimed by AWS 10 minutes ago. The StatefulSet rescheduled the postgres-replica-2 pod to a new node, but the pod has been stuck in ContainerCreating for 9 minutes. The PVC shows Bound but the pod events show Multi-Attach error: volume is already exclusively attached to one node. A stale VolumeAttachment object from the terminated node is preventing the EBS volume from attaching to the replacement node.
What You'll Learn
Why EBS volumes show Multi-Attach errors after spot instance reclamation
Finding and deleting stale VolumeAttachment objects that block re-scheduling
The difference between PVC Bound status and actual volume attachment readiness
Setting up spot instance drain handlers to avoid stale attachment objects
Tools You'll Use
Real-World Context
Stale VolumeAttachment objects after spot instance reclamation are a known AWS EBS + Kubernetes failure mode. The fix is simple once you know what to look for, but the symptom - a healthy-looking PVC that still won't attach - is extremely confusing.
Ready to debug this?
Free account required - sign up with GitHub or Google in 10 seconds
Play The Volume That Won't Let Go