Page MenuHomePhabricator

Provide Swift object store(s) for the labs projects
Closed, DeclinedPublic

Description

We could use a Swift installation to support the projects in the labs infrastructure. I have identified at least a couple use case: Beta cluster and CI.

Beta cluster

The first use case that has been pending for a while: it is to align Beta-Cluster-Infrastructure to production T64835: Setup a Swift cluster on beta-cluster to match production for hosting files/images.

IIRC we currently use an emulation of the media server service that pre date the deployment of Swift. It is running unpuppetized on deployment-upload with Nginx and a stalled MediaWiki code base (thumbs.php and others).

Related bugs:

Continuous integration

There are growing needs for CI to publish and store build artifacts such as logs, tarballs, generated documentation, code coverage reports.

The CI tracking bug is T101545: Provide infrastructure to store files by project/branch post-merge to compare with pre-merge, and blocks a few use cases:

T72945: Preview generated documentation in test pipeline for review
T64633: Jenkins: Set up perceptual diffs (visual regression testing)
T101542: [EPIC] Provide pre-merge reports on patchsets (tracking)
T101543: Provide (pre-merge) performance reports on patchsets
T101544: Provide (pre-merge) code coverage reports on patchsets

Imagine sending a patch to Gerrit and having Zuul report back with the URL to the coverage report and the generated documentation. That will surely help reviewers. Such content being temporarily, it would need to be discarded after some days.

After a merge occur, we publish some documentation to https://doc.wikimedia.org/ and a few coverage reports at https://integration.wikimedia.org/cover/ . That is currently done by having jobs rsync to a specific labs instance and then rsync from gallium (which hosts doc/integration.wikimedia.org). Documented at: https://www.mediawiki.org/wiki/Continuous_integration/Documentation_generation . Those would need to be stored indefinitely (in most cases).

Others

I believe other labs projects such as tools-labs could use a Swift object storage as well that would expose its material under a wmfusercontent.org URL automatically. For example the Meetbot reports http://tools.wmflabs.org/meetbot/ .

The staging cluster had an attempt to reproduce the commons/production Swift storage T91553.

Related Objects

Event Timeline

I found a previous update by @Andrew from November 2014:

To support Swift in labs I want to allow keystone/swift authentication for service users so that we can have project- or tool-wide swift accounts. This requires adding a second ldap backend to keystone, and multiple keystone auth backends was broken in Havana.
I just upgraded Labs to OpenStack Icehouse, and I /think/ that that bug is fixed in IceHouse so I may be able to proceed on this now.

Seems OpenStack version was a blocker. It has been upgraded since them so maybe that unblock Swift stores per labs projects.

jayvdb added a subscriber: jayvdb.Oct 15 2015, 9:09 AM
hashar updated the task description. (Show Details)Oct 19 2015, 7:46 PM
hashar set Security to None.
hashar triaged this task as Normal priority.Nov 9 2015, 1:10 PM
Restricted Application added a subscriber: Steinsplitter. · View Herald TranscriptJul 15 2016, 11:24 AM

This is now out of date as the beta cluster has it's own internal swift install.

hashar closed this task as Declined.Sep 3 2016, 12:29 PM

Last year some of the discussion was to eventually provide an object storage system for all labs project. Not sure whether there are any use cases beside Beta (done: T64835) and CI (T101545)

I am declining the idea of a shared infra.

For CI we would go the same route Beta did: a Swift cluster in the integration project. Lets use sub task T101545 for that.