[from a swift POV, just need to check cluster OK afterwards]

backup1009 is the main backup node for bacula on eqiad. Most backups happen during the night- so just monitoring that it came back and new backups happen normally would be enough.

db1204 is the main mediabackups metadata db- ideally, media backups are stopped while maintenance happens and restored afterwards to avoid errors.

In T365995#9883497, @jcrespo wrote:

backup1009 is the main backup node for bacula on eqiad. Most backups happen during the night- so just monitoring that it came back and new backups happen normally would be enough.

db1204 is the main mediabackups metadata db- ideally, media backups are stopped while maintenance happens and restored afterwards to avoid errors.

Is there a procedure for that so we know how to do so?

Is there a procedure for that so we know how to do so?

Sadly, there is not. The code changes for implementing that "the proper way" are trivial, I didn't want to implement the simple current way & I am scared of changing any backup code logic just days before me leaving (traditionally, it always introduced some regression), so I will just encourage to join me the session next week about debugging mediabackups I had with Arnaud, where I was going to go over typical issues and maintenance.

I will try - but just in case @ABran-WMF please take some notes!

Mentioned in SAL (#wikimedia-operations) [2024-07-09T10:29:47Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1192 db1198 db1199 T365995', diff saved to https://phabricator.wikimedia.org/P66039 and previous config saved to /var/cache/conftool/dbconfig/20240709-102947-root.json

Mentioned in SAL (#wikimedia-operations) [2024-07-09T11:15:48Z] <btullis> drained dse-k8s-worker1006.eqiad.wmnet ready for T365995

Mentioned in SAL (#wikimedia-operations) [2024-07-09T11:17:19Z] <btullis> set cephosd cluster into noout mode to prevent rebalancing for T365995

Mentioned in SAL (#wikimedia-operations) [2024-07-09T11:18:24Z] <btullis> depooled druid1010 for T365995

databases are ready

Mentioned in SAL (#wikimedia-operations) [2024-07-09T14:26:14Z] <hnowlan> kubectl drain kubernetes1061.eqiad.wmnet kubernetes1061.eqiad.wmnet kubernetes1061.eqiad.wmnet kubernetes1061.eqiad.wmnet kubernetes1061.eqiad.wmnet kubernetes1061.eqiad.wmnet kubernetes1061.eqiad.wmnet mw1492.eqiad.wmnet mw1492.eqiad.wmnet (T365995)

kubernetes* and mw* are ready

Switch upgrade completed without issue. All connected hosts are back online and responding to ping now, thanks all for the help.

Swift looks OK, thanks.

Repooling databases

@cmooney got to be closed?