Page MenuHomePhabricator

Setup Swift Storage for Netbox image (was: netbox won't allow me to upload photos of the rack)
Open, NormalPublic

Description

When attempting to upload a photo of the rear of the ulsfo rack 103.02.22, I received the following error:

<class 'PermissionError'>

[Errno 13] Permission denied: '/srv/deployment/netbox/deploy-cache/revs/1cd4d43dd1bdf32e81af0318f479890cf23e2ee5/src/netbox/media/image-attachments/rack_33_103.02.22_rear.jpg'

This likely needs some puppetization of the directory permissions, or a review if we want to allow uploads to it at all.

Event Timeline

RobH triaged this task as Normal priority.Nov 9 2018, 8:45 PM
RobH created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 9 2018, 8:45 PM
RobH added a comment.Nov 9 2018, 8:46 PM

Not an emergency, so I did NOT go livehacking any permissions changes.

Volans added a comment.Nov 9 2018, 8:56 PM

@RobH yep, known issue, the immediate fix was already scheduled in https://gerrit.wikimedia.org/r/c/operations/puppet/+/463820 but then we decided to go directly in the direction of using swift as a backend for attachments to avoid to setup and rsync between the two netbox hosts. For that I preferred to leave it "broken" on purpose to avoid having then to migrate existing attachments to swift. I just din't had yet time to set it up, I hope in the next week or two to be able set everything up, but please let me know also how much is a blocker so I can prioritize accordingly.

Peachey88 updated the task description. (Show Details)Nov 10 2018, 12:06 AM
crusnov renamed this task from netbox won't allow me to upload photos of the rack to Setup Swift Storage for Netbox image (was: netbox won't allow me to upload photos of the rack).Jun 7 2019, 2:46 PM
crusnov claimed this task.
crusnov added subscribers: fgiunchedi, Volans.

okay so this works, mostly, in labs when manually configured to operate against the deployment-prep Swift cluster. Netbox lets me upload images and shows them associated with the object in question - except that viewing fails because they are served from a URL in the swift cluster that is unavailable. We'll look at this part more next week no doubt.

There is a niggle, though. Netbox does a strange thing in that it imports only parts of configuration.py into the settings module, explicitly selecting them from the configuration module and sanitizing and fixing them, so I had to add a broad import to pull the swift settings also. I'm not sure of the best way to approach this in actual production. We could override the settings.py, or patch it to explicitly support the swift settings in general. Some discussion is needed.

Change 518785 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/software/netbox@master] Add a passthrough configuration system

https://gerrit.wikimedia.org/r/518785

Change 518785 merged by CRusnov:
[operations/software/netbox@master] Add a passthrough configuration system

https://gerrit.wikimedia.org/r/518785

Change 520296 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/puppet@production] netbox: Add parameters and settings for storing things in Swift

https://gerrit.wikimedia.org/r/520296

Mentioned in SAL (#wikimedia-operations) [2019-07-03T06:55:05Z] <godog> depool and roll-restart swift proxy - T209182

Change 520296 merged by CRusnov:
[operations/puppet@production] netbox: Add parameters and settings for storing things in Swift

https://gerrit.wikimedia.org/r/520296

Netbox has been deployed with the change that should enable this. We're testing.

Change 527576 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/puppet@production] netbox: Fix additional swift parameters.

https://gerrit.wikimedia.org/r/527576

Change 527576 merged by CRusnov:
[operations/puppet@production] netbox: Fix additional swift parameters

https://gerrit.wikimedia.org/r/527576

crusnov closed this task as Resolved.Aug 6 2019, 6:09 PM

Okay after some finagling, uploading (and downloading) images should work. A particularity of swift storage is that they download instead of viewing, but they work!

A later optimal solution would be to provide an end point for netbox to proxy through to swift, but this isn't as off the shelf and will require patching netbox and/or django-swift-storage.

Volans reopened this task as Open.EditedAug 14 2019, 1:34 PM

Re-opening as there is still some work to do given that as it is right now is less redundant that it was before.
Namely I think we need:

  • ensure images are replicated across both main datacenters
  • ensure Content-Type is properly set so that image attachments are shown instead of downloaded
  • decide a strategy to backup the attachments
  • check if Netbox supports/is supposed to show thumbs of attached images
  • check if on deletion the images are correctly removed from Swift too

I have confirmed content-type is set correctly, however Swift sets a content-disposition to attachment which causes browser to download. Incoming patch to sttrip this at the apache level.

Netbox does not appear to have thumbnails, but there is a feature request open: https://github.com/netbox-community/netbox/issues/2050

Replication and backups: @fgiunchedi halp.

Change 532509 had a related patch set uploaded (by CRusnov; owner: CRusnov):
[operations/puppet@production] profile::netbox: Fix swift proxy content-disposition

https://gerrit.wikimedia.org/r/532509

Change 532509 merged by CRusnov:
[operations/puppet@production] profile::netbox: Fix swift proxy content-disposition

https://gerrit.wikimedia.org/r/532509

Had a chat with @crusnov about swift replication and backups, to summarize:

  1. We can use swift container synchronization to sync eqiad -> codfw, similar to what the docker registry does. For two-way sync we need to verify that the right things happen when we set up codfw -> eqiad at the same time (e.g. uploading to both codfw/eqiad in an active/active fashion yields correct results)
  2. For swift backups we can either (on a schedule, and assuming the container doesn't get "too big")
    1. Download the whole container locally on a (netbox?) host, and ask bacula to backup that path. Restore would be the inverse: bacula restore + swift upload
    2. Copy the whole container to a new swift container (unsynchronized with the other site I'm assuming) and keep the last N copies, restoring would mean copying back the backup swift container to the one netbox reads.

I'm assuming the container isn't expected to get too big (by big I mean > 100G) so we could do either option for backups.

Thanks for the update.

For the replication I guess one-way it's ok, we don't plan to have a/a backend in Netbox anyway (Postgres), but we were thinking to do a/a frontend (Django), although that means cross-DC connections, so is not yet set in stones.
But even in that scenario I don't see a big issue to point both a/a frontend to the single DB/storage active backend and as it would require a manual intervention to failover the DB backend seems ok to me that it would require a manual intervention to failover the storage backend too, pointing to the other swift cluster and inverting the replication.

Sounds good to me for the backups, I'll probably pick 2.A just to have those in our usual backup infrastructure and not depending on the same software where we keep the live ones. We'll most likely upload only pictures of racks and I don't expect it to be large at all.