Page MenuHomePhabricator

Netbox: generate CSV backups
Closed, ResolvedPublic

Description

Currently we're performing Postgresql backups of Netbox, but if a wrong edit is made it would be pretty hard to manually find the right values for a quick revert without having to restore the whole DB, potentially losing changes made by others.
We could, in addition to the DB backup, also perform backups in CSV form of the data, using the export to CSV function in Netbox.
A script in the netbox-deploy repo, that uses the already existing token should do the job and should be fairly simple to add.

Caveat: for the DCIM devices we should use the custom export all fields CSV method instead of the default one.

Things to be decided:

  • which objects to export (all?)
  • how frequent to perform the backup
  • in which structure
    • For this my suggestion would be something like:
netbox-csv-backups/
    2019-05-14/
        dcim.devices.csv
        dcim.sites.csv
        ....
  • how/when to rotate/compress the files

Event Timeline

Volans created this task.May 14 2019, 3:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 14 2019, 3:47 PM
Volans triaged this task as Normal priority.May 14 2019, 3:47 PM

Just to follow up on this. I did spend some time trying to figure out how to initiate a template-based export from hitting a URL. It seems as though there's no API-way, and hitting the URL endpoint doesn't work with a token authentication as far as I can tell.

That said, it'd be relatively trivial to use the API to export things, so I propose that we proceed by basically implementing the same functionality except via the API.

Change 518166 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/software/netbox-deploy@master] Add new dumpbackup.py script

https://gerrit.wikimedia.org/r/518166

Change 518166 merged by CRusnov:
[operations/software/netbox-deploy@master] Add new dumpbackup.py script

https://gerrit.wikimedia.org/r/518166

Mentioned in SAL (#wikimedia-operations) [2019-08-13T18:31:47Z] <crusnov@deploy1001> Started deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292

Mentioned in SAL (#wikimedia-operations) [2019-08-13T18:32:23Z] <crusnov@deploy1001> Finished deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (duration: 00m 36s)

Mentioned in SAL (#wikimedia-operations) [2019-08-13T18:32:31Z] <crusnov@deploy1001> Started deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292

Mentioned in SAL (#wikimedia-operations) [2019-08-13T18:33:09Z] <crusnov@deploy1001> Finished deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (duration: 00m 43s)

Mentioned in SAL (#wikimedia-operations) [2019-08-13T18:33:39Z] <crusnov@deploy1001> Started deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (fix perms)

Mentioned in SAL (#wikimedia-operations) [2019-08-13T18:33:49Z] <crusnov@deploy1001> Finished deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (fix perms) (duration: 00m 09s)

crusnov closed this task as Resolved.Aug 13 2019, 6:43 PM

this has been fully deployed now and tested. It is automated.