Page MenuHomePhabricator

Restore POPs server interfaces and cables
Closed, ResolvedPublic

Description

All the POPs server interfaces have been deleted from prod Netbox, which deleted their cables connecting them to the switches.

Those cables contain manually entered cable IDs (and other characteristics), which were not stored anywhere else (and took time to input).
It's also being used to generate switches interfaces in existing CRs.

They need to be restored.
Maybe audit at least POPs to make sure nothing else got deleted.

Event Timeline

ayounsi triaged this task as High priority.Tue, May 19, 6:28 AM
ayounsi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, May 19, 6:28 AM
Volans added a comment.EditedTue, May 19, 9:25 AM

I'm checking the changelog and evaluating how safe it is to restore directly the daily DB dump that we have:

# PRE
-rw-r--r-- 1 postgres postgres 594483 May 18 01:31 psql-all-dbs-20200518.sql.gz
-rw-r--r-- 1 postgres postgres 599825 May 18 21:12 psql-all-dbs-20200518.sql.gz  # from netboxdb2001

# POST
-rw-r--r-- 1 postgres postgres 711593 May 19 01:31 psql-all-dbs-20200519.sql.gz

We also have CSV dumps but the have revealed harder to restore so far due to Netbox exporting some fields raw and some resolved with a nicer representation of foreign keys and one-to-many/many-to-many fields.

Decided to restore the last backup available (May 18 21:12) and manually replay the 6 actions performed between then and the delete actions.

Mentioned in SAL (#wikimedia-operations) [2020-05-19T10:29:14Z] <volans> start Netbox restore - T253091

Mentioned in SAL (#wikimedia-operations) [2020-05-19T10:32:44Z] <volans> flushed all Netbox caches (manage.py invalidate all) - T253091

@ayounsi thanks for noticing, all data should have been restored. Could you please also do a second verification and confirm it's all there?

ayounsi closed this task as Resolved.Tue, May 19, 12:02 PM
ayounsi assigned this task to Volans.

I looked at the last actions I did yesterday and POP servers links and can't see anything missing, thanks!

Thanks for fixing this. I did see some interfaces with cables and tried to avoid them when cleaning up, but I suppose that I missed a few of them. Apologies.