I'm seeing all kinds of openstack failures in codfw1dev today. VMs can't be created or deleted, nova-compute on cloudvirt2001-dev shows as down, etc. etc.
I did a rabbitmq wipe and restart and that doesn't seem to have helped. Services do claim that the can connect to rabbit but most of the time API calls get marked as pending and just stay that way forever.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T344302 codfw1dev openstack services mostly can't communicate | |||
| Open | None | T344440 Add some monitoring/non-paging alerts to codfw1dev | |||
| Resolved | fgiunchedi | T350010 Deploy 'cloud' Prometheus instance to codfw |
This turned out to be ceph getting full.
It seems I left over the ceph cluster with one osd less some time ago (around christmas I think), while doing some tests to have a cookbook to take out + upgrade + take in osds.
As we replicate per-rack, the node that only had one OSD daemon left got full quicker than the rest, and that brought the cluster to read only mode.
We rebuilt the osd (osd.0) and let the cluster rebalance, and it's back up and running. @Andrew confirmed that things are mostly working now, but I will let him do more tests/decide if this is fixed.
Note that we had to manually remove the PV/VG from that specific osd to be able to ceph-volume lvm zap it and bring it back online.