Page MenuHomePhabricator

Swift users and their usage
Open, Needs TriagePublic

Description

This is a list of users/use cases currently on Swift.

The ms cluster is media storage: two independent clusters in codfw and eqiad and 3x replication each. Whereas thanos is a multi-site cluster with 4x replication.

nameclusterbytes (eqiad+codfw)objectscontainersdescription
mwms560190777432472 + 5600725129683231773941102 + 177242249542972 + 42971mediawiki media storage
searchms62900126496 + 016805 + 01 + 0(elastic)search backups (TBD if still in use)
dockerms503293975765 + 503293975765215822 + 2158221 + 1docker registry storage
pagecompilationms000zim/page compilation (not used)
analyticsms94912827434 + 075209 + 00 + 0binary file exchange analytics -> prod (T219544)
netboxms1 + 024 + 01 + 0netbox attachments
performance/arclampms17922074574 + 1792286071423247 + 232634 + 4svg/logs storage for performance
flink/wdqsthanos2221255946811194flink/wdqs-updater pipelines checkpoints (T246004)
thanosthanos76442034809721140521thanos data
chartmuseumthanos40813923841chart museum storage
mlservethanos???model storage for machine learning
mapsthanos???maps tiles cache

Event Timeline

fgiunchedi mentioned this in Unknown Object (Task).Feb 12 2021, 11:12 AM

I would like to bring to your attention T138915. This is not a current user of Swift, but it seems like something like this, a misc-object storage cluster, would be the ideal location, rather than a relational database, which make backups and administration quite cumbersome.

I am not askinh for help to perform a migration now- this only should happen after current users have been done, but I wonder if its non-trivial size could be taken into account for future procurement.

Thank you for the note @jcrespo, definitely agreed object storage is far more suited for attachments. If OTRS has plans to migrate to moss then definitely we should be taking that into account for the next budget cycle

We should add https://wikitech.wikimedia.org/wiki/Shared_Data_Platform as a potential usage of object storage. This would be a very large usage.

Another potential usage is GitLab artifacts, adding @thcipriani for awareness.