YES, we need to unify them into one set of config files that are actually templates that branch on prod vs labs (or even just use hiera to pick out the values necessary)

This is actually... a fair bit harder than I expected, partially because it could also break prod's apaches, and I'm not very familiar with apache config language...

In T1256#1091917, @yuvipanda wrote:

This is actually... a fair bit harder than I expected, partially because it could also break prod's apaches, and I'm not very familiar with apache config language...

In T1256#1092469, @greg wrote:

Who's help do you need?

Ping? :)

Chad has started on it I believe.

Out of curiosity, what would be the beta equivalent of wikimediafoundation.org (which was involved in the 2016-01-26 incident). It would be nice if http://wikimediafoundation.beta.wmflabs.org/ (or some such) existed as a beta mirror of that part of the configuration. (Or perhaps I just have the URL wrong, and that exists already?)

In T1256#1967920, @cscott wrote:

Out of curiosity, what would be the beta equivalent of wikimediafoundation.org (which was involved in the 2016-01-26 incident). It would be nice if http://wikimediafoundation.beta.wmflabs.org/ (or some such) existed as a beta mirror of that part of the configuration. (Or perhaps I just have the URL wrong, and that exists already?)

https://noc.wikimedia.org/conf/highlight.php?file=all-labs.dblist

There isn't one (currently). One of the issues involved in this is that beta doesn't have all the sites that production has (e.g. copies of all the random private wikimedia.org sites) and production doesn't have all the sites that beta has (e.g. upload.beta.wmflabs.org, which in production is not handled by apache, deployment.wikimedia, etc.)

My notes are at P2531, but they assume presence of https://gerrit.wikimedia.org/r/265642 and https://gerrit.wikimedia.org/r/265659

@Reedy So it seems that an equivalent to wikimediafoundation.org doesn't (yet) exist in labs?

In the incident report it was stated, "Ideally we'd be able to test apache config in beta. However, beta uses mostly separate apache config. Alex M has been trying to work on slowly merging it with production apache config - this patch was part of that effort. However, this specific issue looks like it would've been catchable in beta as wwwportals is now part of the common (both beta and prod) config."

But if there isn't actually an accessible host, I don't see how wwwportals can be tested in beta.

I'm not sure how wikimediafoundation.org is relevant here? It's a wiki, not a portal

The symptom of the problem were redirects to wikimediafoundation.org. Ideally (eventually) the beta configuration would match the production configuration closely enough that this problem would have manifested in beta, presumably as redirects to wikimediafoundation.beta.wmflabs.org. AIUI. But perhaps I'm misunderstanding.

Step 1 would have been for the puppet swat procedure to include testing of changes in Beta.... but I digress.

In T1256#1968013, @greg wrote:

Step 1 would have been for the puppet swat procedure to include testing of changes in Beta.... but I digress.

We should merge the production and beta apache configs together for that.

Hence this bug.

This follow-up task from an incident report has not been updated recently. If it is no longer valid, please add a comment explaining why. If it is still valid, please prioritize it appropriately relative to your other work. If you have any questions, feel free to ask me (Greg Grossmeier).

I've uploaded some patches recently to bring beta closer to prod in this area:

• https://gerrit.wikimedia.org/r/#/c/322413/ - mediawiki: move redirects site from prod_sites to sites
• https://gerrit.wikimedia.org/r/#/c/322416/ - beta sites: copy zerowiki config from prod
• https://gerrit.wikimedia.org/r/#/c/322417/ - beta: configure loginwiki the same as prod
• https://gerrit.wikimedia.org/r/#/c/322418/ - beta: Use wikimedia-common.incl for testwiki
• https://gerrit.wikimedia.org/r/#/c/322419/ - beta: copy metawiki config from prod instead of having our own
• https://gerrit.wikimedia.org/r/#/c/322424/ - beta: bring remnant.conf closer to prod version

Or we could go the other way: Turn production's config files into templates, make them work with beta then get rid of beta's ones. (except for the one or two extra sites beta hosts, I guess)

Actually I prefer the other way, uploading patches for that.

Change 322601 had a related patch set uploaded (by Alex Monk):
beta: Move beta-specific VHosts into their own apache config file

https://gerrit.wikimedia.org/r/322601

Change 322602 had a related patch set uploaded (by Alex Monk):
Move production apache config files to templates

https://gerrit.wikimedia.org/r/322602

Change 322603 had a related patch set uploaded (by Alex Monk):
Use production apache config on beta

https://gerrit.wikimedia.org/r/322603

Change 322604 had a related patch set uploaded (by Alex Monk):
Get rid of old beta_sites class now just containing a load of ensure => absent

https://gerrit.wikimedia.org/r/322604

Change 322601 merged by Filippo Giunchedi:
beta: Move beta-specific VHosts into their own apache config file

https://gerrit.wikimedia.org/r/322601

It looks like everybody dropped the ball for actually getting these reviewed. The next patch (after the last one was merged 18 months ago) is literally changing three files to templates so this is utterly ridiculous. I'm going to start out-of-band requesting reviews again.

Change 322602 abandoned by Alex Monk:
Move some production apache config files to templates

Reason:
Abandoning since it looks like ops plan to achieve pretty much this in a different way.

https://gerrit.wikimedia.org/r/322602

Change 322603 abandoned by Alex Monk:
Use production apache config on beta

Reason:
see dependency

https://gerrit.wikimedia.org/r/322603

Change 322604 abandoned by Alex Monk:
Get rid of old beta_sites class now just containing a load of ensure => absent

Reason:
see dependencies

https://gerrit.wikimedia.org/r/322604

Change 468583 had a related patch set uploaded (by Alex Monk; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki::web::beta_sites: convert to using mediawiki::web::vhost

https://gerrit.wikimedia.org/r/468583

Change 468583 had a related patch set uploaded (by Alex Monk; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki::web::beta_sites: convert to using mediawiki::web::vhost

https://gerrit.wikimedia.org/r/468583

Change 468583 merged by Giuseppe Lavagetto:
[operations/puppet@production] mediawiki::web::beta_sites: convert to using mediawiki::web::vhost

https://gerrit.wikimedia.org/r/468583