Not an emergency, so I did NOT go livehacking any permissions changes.

@RobH yep, known issue, the immediate fix was already scheduled in https://gerrit.wikimedia.org/r/c/operations/puppet/+/463820 but then we decided to go directly in the direction of using swift as a backend for attachments to avoid to setup and rsync between the two netbox hosts. For that I preferred to leave it "broken" on purpose to avoid having then to migrate existing attachments to swift. I just din't had yet time to set it up, I hope in the next week or two to be able set everything up, but please let me know also how much is a blocker so I can prioritize accordingly.

okay so this works, mostly, in labs when manually configured to operate against the deployment-prep Swift cluster. Netbox lets me upload images and shows them associated with the object in question - except that viewing fails because they are served from a URL in the swift cluster that is unavailable. We'll look at this part more next week no doubt.

There is a niggle, though. Netbox does a strange thing in that it imports only parts of configuration.py into the settings module, explicitly selecting them from the configuration module and sanitizing and fixing them, so I had to add a broad import to pull the swift settings also. I'm not sure of the best way to approach this in actual production. We could override the settings.py, or patch it to explicitly support the swift settings in general. Some discussion is needed.

Change 518785 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/software/netbox@master] Add a passthrough configuration system

https://gerrit.wikimedia.org/r/518785

Change 518785 merged by CRusnov:
[operations/software/netbox@master] Add a passthrough configuration system

https://gerrit.wikimedia.org/r/518785

Change 520296 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/puppet@production] netbox: Add parameters and settings for storing things in Swift

https://gerrit.wikimedia.org/r/520296

Mentioned in SAL (#wikimedia-operations) [2019-07-03T06:55:05Z] <godog> depool and roll-restart swift proxy - T209182

Change 520296 merged by CRusnov:
[operations/puppet@production] netbox: Add parameters and settings for storing things in Swift

https://gerrit.wikimedia.org/r/520296

Netbox has been deployed with the change that should enable this. We're testing.

Change 527576 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/puppet@production] netbox: Fix additional swift parameters.

https://gerrit.wikimedia.org/r/527576

Change 527576 merged by CRusnov:
[operations/puppet@production] netbox: Fix additional swift parameters

https://gerrit.wikimedia.org/r/527576

I have confirmed content-type is set correctly, however Swift sets a content-disposition to attachment which causes browser to download. Incoming patch to sttrip this at the apache level.

Netbox does not appear to have thumbnails, but there is a feature request open: https://github.com/netbox-community/netbox/issues/2050

Replication and backups: @fgiunchedi halp.

Change 532509 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/puppet@production] profile::netbox: Fix swift proxy content-disposition

https://gerrit.wikimedia.org/r/532509

Change 532509 merged by CRusnov:
[operations/puppet@production] profile::netbox: Fix swift proxy content-disposition

https://gerrit.wikimedia.org/r/532509

Had a chat with @crusnov about swift replication and backups, to summarize:

1. We can use swift container synchronization to sync eqiad -> codfw, similar to what the docker registry does. For two-way sync we need to verify that the right things happen when we set up codfw -> eqiad at the same time (e.g. uploading to both codfw/eqiad in an active/active fashion yields correct results)
2. For swift backups we can either (on a schedule, and assuming the container doesn't get "too big")
   1. Download the whole container locally on a (netbox?) host, and ask bacula to backup that path. Restore would be the inverse: bacula restore + swift upload
   2. Copy the whole container to a new swift container (unsynchronized with the other site I'm assuming) and keep the last N copies, restoring would mean copying back the backup swift container to the one netbox reads.

I'm assuming the container isn't expected to get too big (by big I mean > 100G) so we could do either option for backups.

Thanks for the update.

For the replication I guess one-way it's ok, we don't plan to have a/a backend in Netbox anyway (Postgres), but we were thinking to do a/a frontend (Django), although that means cross-DC connections, so is not yet set in stones.
But even in that scenario I don't see a big issue to point both a/a frontend to the single DB/storage active backend and as it would require a manual intervention to failover the DB backend seems ok to me that it would require a manual intervention to failover the storage backend too, pointing to the other swift cluster and inverting the replication.

Sounds good to me for the backups, I'll probably pick 2.A just to have those in our usual backup infrastructure and not depending on the same software where we keep the live ones. We'll most likely upload only pictures of racks and I don't expect it to be large at all.

So the task for this is:

• Automate downloading of swift container contents on Netbox frontends
• Setup bacula path to backup those contents

And then it will be resolved, afaict.

Hi everyone,

Since the last comment is from 2 years ago from a person no longer with the foundation, is there someone who can summarize what the state of this is?