Note that the "Public Cloud" panel of our service provider is crashing since a couple of hours causing inability of create any new service.

(This image is not under CC-BY-SA or GPL)

503 Service Unavailable

I will retry ASAP.

Hi @valerio.bozzolan, from this tweet

https://twitter.com/olesovhcom/status/1369535787570724864

it seems that all of SBG will be down at least for the day.

Thank you @Joe,

Thank you to @Nemo_bis who was able to allocate a new virtual machine, I've migrated the website from one datacenter to another one, now online:

https://www.wikimedia.it/
https://wikimedia.it/
https://wikimedia.it/
http://wikimedia.it/

Now I'm trying to restore other services from our tracked configuration:

rWIIN wikimedia-it-wmit-infrastructure

It will be a long day. asd

Upstream update some seconds ago.

Status of Strasbourg Datacenter
SBG1 : Network Room is OK - 4 room destroyed - 8 Rooms OK
SBG2 : Destroyed
SBG3 : UPS Down - Check server still in progress
SBG4 : No physical impact

No restart today for SBG1,SBG3 and SBG4

Plan for the next 2 weeks:
1)Rebuilding 20KV for SBG3
2)Rebuilding 240V in SBG1/SBG4
3)Verifying DWDM/routers/switchs in the network room A (SBG1).
Checking the fibers Paris/Frankfurt
4)Rebuilding the network room B (SBG5) cheking fiber Paris/Frankfurt
― http://travaux.ovh.net/?do=details&id=49484

The good part is that we should have a lot of cute snapshots safely stored in the datacenter SBG3

That's my understanding too per https://www.ovh.com/manager/public-cloud/#/pci/projects/ <redacted> /billing/history/2021/2 . So we should be able to recover "everything" by 2021-03-19 at the latest, per http://travaux.ovh.net/?do=details&id=49484 :

The restoration of the site's power supply of SBG1 and SBG4 is estimated for Monday, March 15. A recovery for SBG3 is estimated for Friday, March 19

Just my opinion but I think a few of the services and microsites on the machine can wait 9 days. In the meanwhile we may want to consider the main Wordpress instance as read-only, to make it easier to later pick either side as new master.

In T277008#6902236, @Nemo_bis wrote:

In the meanwhile we may want to consider the main Wordpress instance as read-only, to make it easier to later pick either side as new master.

I thought so too but in the meanwhile it seems the staff has some important work in the queue (e.g. for the bando wiki-docente 2021). We can also assist them to:

1. allow to update www.wikimedia.it
2. wait for upstream news about our snapshot in SBG3
3. re-import that complete snapshot once ready, but overriding the specific data of www.wikimedia.it

If I understand https://www.ovh.com/fr/images/sbg/index-en.html correctly, we'll know more about the location(s) of the snapshots in the coming 24-48 hours.

In T277008#6905929, @Nemo_bis wrote:

If I understand https://www.ovh.com/fr/images/sbg/index-en.html correctly, we'll know more about the location(s) of the snapshots in the coming 24-48 hours.

I have understood the same thing, but what is the meaning of room? The number of SBGx (so if SBG2 if destroyed and our backup is not in SBG1, Room 4) our data are not lose.
But, if the backups of SBG2 are in another Rooms of SBG2...

I'm sharing here yesterday's update:

From http://travaux.ovh.net/?do=details&id=49484 (Tuesday, 16 March 2021, 18:09PM)

SBG-3 Situation:

• Servers undamaged

Electrical restart:

• Temporarily repowered on 12th March and will be restored permanently on 16th March

Network restart:

• New network room to be deployed and powered on 16th March
• Internal network to be redeployed on 17th March
• Server restart : Provisional ETA: Monday, 22 March for gradual restart

Update

http://travaux.ovh.net/?do=details&id=49484

SBG1 datacenter is currently closed and secured.
SBG2 datacenter is closed and secured.
SBG4 datacenter is closed and secured as a precautionary measure.

The recovery scenario will be defined during the weekend for SBG1 and SBG4 data centers, always focusing on security.

SBG3 is partially restored:
Teams continue to work on the SBG3 data center, as of today, respecting security measures.

Update

http://travaux.ovh.net/?do=details&id=49484

SBG3 Public Cloud Instance 86%*
The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Find out more about our cleaning process via this link (here).

#iosperiamochemelacavo
(only for Italian people, sorry)

Update

http://travaux.ovh.net/?do=details&id=49484

Public Cloud Instance - 90%*
Services not yet returned to customers will be available again on Monday 29/03 and Tuesday 30/03.

What's left to do here really?

In T277008#7153070, @Nemo_bis wrote:

What's left to do here really?

Unfortunately I don't have a complete overview anyway I'm quite sure we were missing just WMI Framadate. Everything else seems fine to me and since lot of weeks we have not received any more complaints.

About WMI Framadate, here the related updated documentation:

https://wiki.wikimedia.it/wiki/Framadate/Technical_documentation