Page MenuHomePhabricator

Rework beta apache config
Closed, ResolvedPublic

Description

The current beta apache config is somewhat of a mess. It's diverged, and is well behind that of production.

After Giuseppe has merged and deployed (and it's working!) https://gerrit.wikimedia.org/r/#/c/170300/ I'm planning on rebuilding/rebasing the beta cluster apache config ontop of the current production one, so it's not so far diverged.

We can then probably try and then get some puppet magic to have it deploy 2 versions of the config (templates?), without having 2 physical versions on disk.

We might be actually able to do something along those lines to do the beta config from production... Maybe?

Event Timeline

Reedy created this task.Nov 12 2014, 5:57 PM
Reedy claimed this task.
Reedy raised the priority of this task from to Normal.
Reedy updated the task description. (Show Details)
Reedy changed Security from none to None.
Reedy added subscribers: greg, hashar, Reedy, zeljkofilipin.

YES, we need to unify them into one set of config files that are actually templates that branch on prod vs labs (or even just use hiera to pick out the values necessary)

This is actually... a fair bit harder than I expected, partially because it could also break prod's apaches, and I'm not very familiar with apache config language...

greg reassigned this task from Reedy to yuvipanda.Mar 5 2015, 4:41 PM

Who's help do you need?

This is actually... a fair bit harder than I expected, partially because it could also break prod's apaches, and I'm not very familiar with apache config language...

In T1256#1092469, @greg wrote:

Who's help do you need?

Ping? :)

Chad has started on it I believe.

greg reassigned this task from yuvipanda to demon.Mar 20 2015, 6:49 PM
greg moved this task from Backlog to In Progress on the Staging board.
demon removed demon as the assignee of this task.May 13 2015, 3:39 PM
demon added a subscriber: demon.
thcipriani moved this task from In Progress to Backlog on the Staging board.May 27 2015, 3:05 PM
Restricted Application added a subscriber: Luke081515. · View Herald TranscriptJul 13 2015, 7:26 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2015, 8:16 PM
cscott added a subscriber: cscott.Jan 26 2016, 10:58 PM

Out of curiosity, what would be the beta equivalent of wikimediafoundation.org (which was involved in the 2016-01-26 incident). It would be nice if http://wikimediafoundation.beta.wmflabs.org/ (or some such) existed as a beta mirror of that part of the configuration. (Or perhaps I just have the URL wrong, and that exists already?)

Out of curiosity, what would be the beta equivalent of wikimediafoundation.org (which was involved in the 2016-01-26 incident). It would be nice if http://wikimediafoundation.beta.wmflabs.org/ (or some such) existed as a beta mirror of that part of the configuration. (Or perhaps I just have the URL wrong, and that exists already?)

https://noc.wikimedia.org/conf/highlight.php?file=all-labs.dblist

There isn't one (currently). One of the issues involved in this is that beta doesn't have all the sites that production has (e.g. copies of all the random private wikimedia.org sites) and production doesn't have all the sites that beta has (e.g. upload.beta.wmflabs.org, which in production is not handled by apache, deployment.wikimedia, etc.)

My notes are at P2531, but they assume presence of https://gerrit.wikimedia.org/r/265642 and https://gerrit.wikimedia.org/r/265659

@Reedy So it seems that an equivalent to wikimediafoundation.org doesn't (yet) exist in labs?

In the incident report it was stated, "Ideally we'd be able to test apache config in beta. However, beta uses mostly separate apache config. Alex M has been trying to work on slowly merging it with production apache config - this patch was part of that effort. However, this specific issue looks like it would've been catchable in beta as wwwportals is now part of the common (both beta and prod) config."

But if there isn't actually an accessible host, I don't see how wwwportals can be tested in beta.

I'm not sure how wikimediafoundation.org is relevant here? It's a wiki, not a portal

The symptom of the problem were redirects to wikimediafoundation.org. Ideally (eventually) the beta configuration would match the production configuration closely enough that this problem would have manifested in beta, presumably as redirects to wikimediafoundation.beta.wmflabs.org. AIUI. But perhaps I'm misunderstanding.

greg added a comment.Jan 26 2016, 11:17 PM

Step 1 would have been for the puppet swat procedure to include testing of changes in Beta.... but I digress.

In T1256#1968013, @greg wrote:

Step 1 would have been for the puppet swat procedure to include testing of changes in Beta.... but I digress.

We should merge the production and beta apache configs together for that.

Hence this bug.

greg added a comment.Sep 29 2016, 7:42 PM

This follow-up task from an incident report has not been updated recently. If it is no longer valid, please add a comment explaining why. If it is still valid, please prioritize it appropriately relative to your other work. If you have any questions, feel free to ask me (Greg Grossmeier).

I've uploaded some patches recently to bring beta closer to prod in this area:

Or we could go the other way: Turn production's config files into templates, make them work with beta then get rid of beta's ones. (except for the one or two extra sites beta hosts, I guess)

Actually I prefer the other way, uploading patches for that.

Change 322601 had a related patch set uploaded (by Alex Monk):
beta: Move beta-specific VHosts into their own apache config file

https://gerrit.wikimedia.org/r/322601

Change 322602 had a related patch set uploaded (by Alex Monk):
Move production apache config files to templates

https://gerrit.wikimedia.org/r/322602

Change 322603 had a related patch set uploaded (by Alex Monk):
Use production apache config on beta

https://gerrit.wikimedia.org/r/322603

Change 322604 had a related patch set uploaded (by Alex Monk):
Get rid of old beta_sites class now just containing a load of ensure => absent

https://gerrit.wikimedia.org/r/322604

Change 322601 merged by Filippo Giunchedi:
beta: Move beta-specific VHosts into their own apache config file

https://gerrit.wikimedia.org/r/322601

hashar removed a subscriber: hashar.Jan 6 2017, 9:16 AM
Krenair added a comment.EditedJun 9 2018, 10:08 PM

It looks like everybody dropped the ball for actually getting these reviewed. The next patch (after the last one was merged 18 months ago) is literally changing three files to templates so this is utterly ridiculous. I'm going to start out-of-band requesting reviews again.

Change 322602 abandoned by Alex Monk:
Move some production apache config files to templates

Reason:
Abandoning since it looks like ops plan to achieve pretty much this in a different way.

https://gerrit.wikimedia.org/r/322602

Change 322603 abandoned by Alex Monk:
Use production apache config on beta

Reason:
see dependency

https://gerrit.wikimedia.org/r/322603

Change 322604 abandoned by Alex Monk:
Get rid of old beta_sites class now just containing a load of ensure => absent

Reason:
see dependencies

https://gerrit.wikimedia.org/r/322604

Change 468583 had a related patch set uploaded (by Alex Monk; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki::web::beta_sites: convert to using mediawiki::web::vhost

https://gerrit.wikimedia.org/r/468583

Change 468583 had a related patch set uploaded (by Alex Monk; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki::web::beta_sites: convert to using mediawiki::web::vhost

https://gerrit.wikimedia.org/r/468583

Change 468583 merged by Giuseppe Lavagetto:
[operations/puppet@production] mediawiki::web::beta_sites: convert to using mediawiki::web::vhost

https://gerrit.wikimedia.org/r/468583

Krenair closed this task as Resolved.Oct 22 2018, 2:17 PM
Krenair assigned this task to Joe.