Change Details

On 2023-02-13 we had a total outage of the Cloud Ceph cluster (`cloudceph*.eqiad.wmnet`). Still investigating.

On 2023-02-13 (around 16:30 UTC) we had a total outage of the Cloud Ceph cluster (`cloudceph*.eqiad.wmnet`). To prevent any data corruption, we did shut down all the OpenStack hypervisors (`cloudvirt*.eqiad.wmnet`), effectively turning off Cloud VPS and its related services (Toolforge, etc.) See the comments below for root cause and resolution.

On 2023-02-13 (around 16:30 UTC) we had a total outage of the Cloud Ceph cluster (`cloudceph*.eqiad.wmnet`). To prevent any data corruption, we did shut down all the OpenStack hypervisors (`cloudvirt*.eqiad.wmnet`), effectively turning off Cloud VPS and its related services (Toolforge, Still investigatingetc.) See the comments below for root cause and resolution.