Kubernetes - Failed to detach Persistent Volume
This is an entry in our Engineering Notes. We record troubleshooting steps and advice from our experience using similar tools to your teams.
Problem: Failed to detach
Our team recently ran into an issue in our Kubernetes cluster where
volumeattachment's failed to detach and caused a flood of errors in our
logging systems. The errors were thrown repeatedly in the logs of the
csi-attacher is a sidecar container that is responsible
for attaching and detaching volumes to and from nodes. In our case,
csi-attacher was part of the
csi-cinder-controllerplugin pod in
The side effect of this error was a dramatic increase in resource utilization on the node.
request.go:530 Throttling request took 1.385373635s, request: PATCH:https://10.254.0.1:443/apis/storage.k8s.io/v1beta1/volumeattachments/csi-5321e82004036bca2c98cd2254de8568283a1e72a36cc21b1df53d2667de54e3
csi_handler.go:439 Saved detach error to "csi-5321e82004036bca2c98cd2254de8568283a1e72a36cc21b1df53d2667de54e3
csi_handler.go:99 Error processing csi-5321e82004036bca2c98cd2254de8568283a1e72a36cc21b1df53d2667de54e3": failed to detach: persistentvolume "pvc-5a4edc0a-20bf-418d-92a9-32dbfc952082" not found
controller.go:175 Started VA processing csi-5321e82004036bca2c98cd2254de8568283a1e72a36cc21b1df53d2667de54e3
csi_handler.go:89 CSIHandler: processing VA csi-5321e82004036bca2c98cd2254de8568283a1e72a36cc21b1df53d2667de54e3
csi_handler.go:140 Starting detach operation for "csi-5321e82004036bca2c98cd2254de8568283a1e72a36cc21b1df53d2667de54e3"
csi_handler.go:147 Detaching csi-5321e82004036bca2c98cd2254de8568283a1e72a36cc21b1df53d2667de54e3
csi_handler.go:428 Saving detach error to csi-5321e82004036bca2c98cd2254de8568283a1e72a36cc21b1df53d2667de54e3
Solution: Force delete
The problem was that we had volume attachments that were defined in Kubernetes
but not in OpenStack. This caused an issue with the
csi-attacher because it
was trying to detach the volume from the node but it was not able to find the
volume in OpenStack.
To fix this issue, we had to delete the volume attachments from Kubernetes. However,
we were not able to delete the volume attachments because it still attempted to
remove the volume from OpenStack. To get around this, we had to edit the
volumeattachment and remove the
finalizers. This allowed the CSI driver to
delete the volume attachment from Kubernetes automatically.
kubectl get pv
kubectl get volumeattachment
VolumeAttachment with the issue
Verify that the volume attachment doesn't have a volume associated with it.
kubectl edit volumeattachment csi-5321e82004036bca2c98cd2254de8568283a1e72a36cc21b1df53d2667de54e3
Remove the finalizers and save the file
If the volume attachment has not been automatically deleted, you should be able to delete it now
kubectl delete volumeattachment csi-5321e82004036bca2c98cd2254de8568283a1e72a36cc21b1df53d2667de54e3