Page MenuHomePhabricator

Deploy wikimedia/portals with scap3
Closed, ResolvedPublic1 Estimated Story Points

Description

In order to be able to show the portal pages from Git as described in T110070, we need to deploy the contents of the repository wikimedia/portals to some space either in /srv/mediawiki or any other static directory. This deployment should be happening for any server that has roles mediawiki::web::beta_sites or mediawiki::web::sites.

Event Timeline

Smalyshev raised the priority of this task from to High.
Smalyshev updated the task description. (Show Details)
Smalyshev added a project: Discovery-ARCHIVED.
Smalyshev added a subscriber: Smalyshev.
mmodell added a subscriber: mmodell.

I'd really like to see this use Scap for deployment.

For our planning purposes, what kind of timeline should we plan for to be able to deploy the portal via scap3?

I'd really like to see this use Scap for deployment.

If they want content on all Apache servers this seems like a job for scap/sync-dir today. All of the docroots are managed by scap in /srv/mediawiki/docroot

When we initially discussed it in #wikimedia-releng on IRC, there were shitloads of several conflicting opinions and disagreements, leading me to end up with an extreme lack of understanding, that is to say that I had/have essentially no clue what is actually desired and/or required.

If the portals are to live in /srv/mediawiki (which is what I originally suggested) and get deployed by wikidev, then indeed, it could be deployed with regular scap and there really isn't much to do here.

Again, during IRC discussion, there seemed to be a great deal of disagreement about whether it would be ok to require the help of a SWAT deployer to do the updates. I got the impression that the portals needed separate permissions, specifically, people without wikidev group membership should be able to deploy the portals, and do so independently from the mediawiki deployments.

Separate deployment groups, with separate shared ssh keys, is a feature that will be supported with Scap, however, it's unfortunately not ready to go quite yet. We could probably have that ready in the next week or two, optimistically.

When we initially discussed it in #wikimedia-releng on IRC, there were shitloads of several conflicting opinions and disagreements, leading me to end up with an extreme lack of understanding, that is to say that I had/have essentially no clue what is actually desired and/or required.

Then perhaps as product owner I can clear up this confusion. :-)

If the portals are to live in /srv/mediawiki (which is what I originally suggested) and get deployed by wikidev, then indeed, it could be deployed with regular scap and there really isn't much to do here.

Again, during IRC discussion, there seemed to be a great deal of disagreement about whether it would be ok to require the help of a SWAT deployer to do the updates. I got the impression that the portals needed separate permissions, specifically, people without wikidev group membership should be able to deploy the portals, and do so independently from the mediawiki deployments.

That is the ideal. But it is not a requirement for the MVP.

Separate deployment groups, with separate shared ssh keys, is a feature that will be supported with Scap, however, it's unfortunately not ready to go quite yet. We could probably have that ready in the next week or two, optimistically.

I would prefer to go with regular scap then, and migrate over to scap3 (with all its nice features) later. This has taken much longer than I'd hoped, so if there's anything we can do to get it working sooner, I'd definitely want to go with that.

As far as I know this is ready to go on our end, I just need the go-ahead and I can start deploying this repo along with the train.

added @MaxSem as a subscriber as he is now leading up the backend portion of the portals work.

MaxSem lowered the priority of this task from High to Low.Nov 3 2015, 12:53 AM

Lowering priority: with https://gerrit.wikimedia.org/r/#/c/248526/ , we're fine for now. Having this repository deployed via scap3 (and potentially automatically) would be nice, but it's not blocking us right now.

ksmith renamed this task from Create deployment for wikimedia/portals to Create scripts for automatic deployment for wikimedia/portals.Nov 3 2015, 12:55 AM

Does deploying MW with scap3 actually block this? Or can we migrate portals before we migrate MW itself?

portals are completely independent from MW, the submodule just happens to live in the mediawiki-config repository because that's also where the docroot used by our apache's lives. We can almost certainly migrate them before

mmodell renamed this task from Create scripts for automatic deployment for wikimedia/portals to Deploy wikimedia/portals with scap3.Mar 13 2016, 5:02 AM
mmodell removed mmodell as the assignee of this task.

Currently weekly portal deploys happen thanks to @Jdrewniak - how's the current process? Thanks.

Jdrewniak claimed this task.

@MarcoAurelio this task can certainly be closed out now.

The portals do use scap to deploy the portals/deploy repository, which is placed as a git submodule in mediawiki-config.

Scap is run via a shell script called deploy-portals at the root of that repo.

I wrote some of this down in the portals deploy documentation, but I need to update that to include the addition of the portals/deploy repository.