All the POPs server interfaces have been deleted from prod Netbox, which deleted their cables connecting them to the switches.
Those cables contain manually entered cable IDs (and other characteristics), which were not stored anywhere else (and took time to input).
It's also being used to generate switches interfaces in existing CRs.
They need to be restored.
Maybe audit at least POPs to make sure nothing else got deleted.
I'm checking the changelog and evaluating how safe it is to restore directly the daily DB dump that we have:
# PRE -rw-r--r-- 1 postgres postgres 594483 May 18 01:31 psql-all-dbs-20200518.sql.gz -rw-r--r-- 1 postgres postgres 599825 May 18 21:12 psql-all-dbs-20200518.sql.gz # from netboxdb2001 # POST -rw-r--r-- 1 postgres postgres 711593 May 19 01:31 psql-all-dbs-20200519.sql.gz
We also have CSV dumps but the have revealed harder to restore so far due to Netbox exporting some fields raw and some resolved with a nicer representation of foreign keys and one-to-many/many-to-many fields.
Decided to restore the last backup available (May 18 21:12) and manually replay the 6 actions performed between then and the delete actions.
Mentioned in SAL (#wikimedia-operations) [2020-05-19T10:29:14Z] <volans> start Netbox restore - T253091
Mentioned in SAL (#wikimedia-operations) [2020-05-19T10:32:44Z] <volans> flushed all Netbox caches (manage.py invalidate all) - T253091
@ayounsi thanks for noticing, all data should have been restored. Could you please also do a second verification and confirm it's all there?
I looked at the last actions I did yesterday and POP servers links and can't see anything missing, thanks!
Thanks for fixing this. I did see some interfaces with cables and tried to avoid them when cleaning up, but I suppose that I missed a few of them. Apologies.