Rework beta apache config
Closed, ResolvedPublic
Actions

Description

The current beta apache config is somewhat of a mess. It's diverged, and is well behind that of production.

After Giuseppe has merged and deployed (and it's working!) https://gerrit.wikimedia.org/r/#/c/170300/ I'm planning on rebuilding/rebasing the beta cluster apache config ontop of the current production one, so it's not so far diverged.

We can then probably try and then get some puppet magic to have it deploy 2 versions of the config (templates?), without having 2 physical versions on disk.

We might be actually able to do something along those lines to do the beta config from production... Maybe?

Details

Project	Branch	Lines +/-	Subject
operations/puppet	production	+103 -409	mediawiki::web::beta_sites: convert to using mediawiki::web::vhost
operations/puppet	production	+0 -77	Get rid of old beta_sites class now just containing a load of ensure => absent
operations/puppet	production	+80 -88	Use production apache config on beta
operations/puppet	production	+3 -3	Move some production apache config files to templates
operations/puppet	production	+57 -57	beta: Move beta-specific VHosts into their own apache config file

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T53494 Use Beta cluster as a true canary for code deployments (epic)
Open	None	T87220 Minimize infrastructure differences between Beta Cluster and production
Resolved	yuvipanda	T86644 Remove all ::beta roles in puppet
Resolved	Joe	T1256 Rework beta apache config

Event Timeline

Reedy created this task.Nov 12 2014, 5:57 PM

Reedy claimed this task.

Reedy raised the priority of this task from to Medium.

Reedy updated the task description. (Show Details)

Reedy added a project: Release-Engineering-Team.

Reedy changed Security from none to None.

Reedy added subscribers: greg, hashar, Reedy, zeljkofilipin.

zeljkofilipin removed a subscriber: zeljkofilipin.Dec 3 2014, 12:43 PM

yuvipanda added a parent task: T87220: Minimize infrastructure differences between Beta Cluster and production.Jan 20 2015, 8:22 AM

YES, we need to unify them into one set of config files that are actually templates that branch on prod vs labs (or even just use hiera to pick out the values necessary)

greg removed a parent task: T426: [Quarterly Goal] Create tested nightly builds.Mar 4 2015, 8:24 PM

This is actually... a fair bit harder than I expected, partially because it could also break prod's apaches, and I'm not very familiar with apache config language...

Who's help do you need?

In T1256#1091917, @yuvipanda wrote:

This is actually... a fair bit harder than I expected, partially because it could also break prod's apaches, and I'm not very familiar with apache config language...

In T1256#1092469, @greg wrote:

Who's help do you need?

Ping? :)

Chad has started on it I believe.

greg reassigned this task from yuvipanda to demon.Mar 20 2015, 6:49 PM

greg moved this task from Backlog to In Progress on the Staging board.

thcipriani mentioned this in T91548: Create staging-mw-app* (MW App servers).Mar 23 2015, 6:28 PM

Liuxinyu970226 added a subscriber: Liuxinyu970226.Apr 18 2015, 1:32 AM

demon removed demon as the assignee of this task.May 13 2015, 3:39 PM

demon added a subscriber: demon.

thcipriani moved this task from In Progress to Backlog on the Staging board.May 27 2015, 3:05 PM

thcipriani moved this task from To Triage to Backlog on the Beta-Cluster-Infrastructure board.Jul 13 2015, 7:26 PM

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptJul 13 2015, 7:26 PM

Krenair added a subscriber: Krenair.Aug 26 2015, 8:16 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2015, 8:16 PM

Krenair added a parent task: T86644: Remove all ::beta roles in puppet.Jan 26 2016, 8:53 PM

MZMcBride added a subscriber: MZMcBride.Jan 26 2016, 9:56 PM

greg added a project: Incident-20160126-WikimediaDomainRedirection.Jan 26 2016, 10:38 PM

Out of curiosity, what would be the beta equivalent of wikimediafoundation.org (which was involved in the 2016-01-26 incident). It would be nice if http://wikimediafoundation.beta.wmflabs.org/ (or some such) existed as a beta mirror of that part of the configuration. (Or perhaps I just have the URL wrong, and that exists already?)

In T1256#1967920, @cscott wrote:

Out of curiosity, what would be the beta equivalent of wikimediafoundation.org (which was involved in the 2016-01-26 incident). It would be nice if http://wikimediafoundation.beta.wmflabs.org/ (or some such) existed as a beta mirror of that part of the configuration. (Or perhaps I just have the URL wrong, and that exists already?)

https://noc.wikimedia.org/conf/highlight.php?file=all-labs.dblist

There isn't one (currently). One of the issues involved in this is that beta doesn't have all the sites that production has (e.g. copies of all the random private wikimedia.org sites) and production doesn't have all the sites that beta has (e.g. upload.beta.wmflabs.org, which in production is not handled by apache, deployment.wikimedia, etc.)

My notes are at P2531, but they assume presence of https://gerrit.wikimedia.org/r/265642 and https://gerrit.wikimedia.org/r/265659

@Reedy So it seems that an equivalent to wikimediafoundation.org doesn't (yet) exist in labs?

In the incident report it was stated, "Ideally we'd be able to test apache config in beta. However, beta uses mostly separate apache config. Alex M has been trying to work on slowly merging it with production apache config - this patch was part of that effort. However, this specific issue looks like it would've been catchable in beta as wwwportals is now part of the common (both beta and prod) config."

But if there isn't actually an accessible host, I don't see how wwwportals can be tested in beta.

I'm not sure how wikimediafoundation.org is relevant here? It's a wiki, not a portal

The symptom of the problem were redirects to wikimediafoundation.org. Ideally (eventually) the beta configuration would match the production configuration closely enough that this problem would have manifested in beta, presumably as redirects to wikimediafoundation.beta.wmflabs.org. AIUI. But perhaps I'm misunderstanding.

Step 1 would have been for the puppet swat procedure to include testing of changes in Beta.... but I digress.

In T1256#1968013, @greg wrote:

Step 1 would have been for the puppet swat procedure to include testing of changes in Beta.... but I digress.

We should merge the production and beta apache configs together for that.

Hence this bug.

JanZerebecki added a subscriber: JanZerebecki.Jan 27 2016, 9:19 AM

Glaisher added a subscriber: Glaisher.Jan 27 2016, 3:22 PM

greg added a project: Wikimedia-Incident.Jul 28 2016, 10:17 PM

greg moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.Jul 28 2016, 10:18 PM

This follow-up task from an incident report has not been updated recently. If it is no longer valid, please add a comment explaining why. If it is still valid, please prioritize it appropriately relative to your other work. If you have any questions, feel free to ask me (Greg Grossmeier).

I've uploaded some patches recently to bring beta closer to prod in this area:

https://gerrit.wikimedia.org/r/#/c/322413/ - mediawiki: move redirects site from prod_sites to sites
https://gerrit.wikimedia.org/r/#/c/322416/ - beta sites: copy zerowiki config from prod
https://gerrit.wikimedia.org/r/#/c/322417/ - beta: configure loginwiki the same as prod
https://gerrit.wikimedia.org/r/#/c/322418/ - beta: Use wikimedia-common.incl for testwiki
https://gerrit.wikimedia.org/r/#/c/322419/ - beta: copy metawiki config from prod instead of having our own
https://gerrit.wikimedia.org/r/#/c/322424/ - beta: bring remnant.conf closer to prod version

Or we could go the other way: Turn production's config files into templates, make them work with beta then get rid of beta's ones. (except for the one or two extra sites beta hosts, I guess)

Actually I prefer the other way, uploading patches for that.

Change 322601 had a related patch set uploaded (by Alex Monk):
beta: Move beta-specific VHosts into their own apache config file

https://gerrit.wikimedia.org/r/322601

Change 322602 had a related patch set uploaded (by Alex Monk):
Move production apache config files to templates

https://gerrit.wikimedia.org/r/322602

Change 322603 had a related patch set uploaded (by Alex Monk):
Use production apache config on beta

https://gerrit.wikimedia.org/r/322603

Change 322604 had a related patch set uploaded (by Alex Monk):
Get rid of old beta_sites class now just containing a load of ensure => absent

https://gerrit.wikimedia.org/r/322604

Krenair mentioned this in T86644: Remove all ::beta roles in puppet.Nov 24 2016, 1:39 AM

Change 322601 merged by Filippo Giunchedi:
beta: Move beta-specific VHosts into their own apache config file

https://gerrit.wikimedia.org/r/322601

hashar removed a subscriber: hashar.Jan 6 2017, 9:16 AM

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 7:05 PM

EddieGP added a subscriber: EddieGP.Apr 12 2018, 8:02 AM

Krenair mentioned this in T182927: Get letsencrypt wildcard cert for *.beta.wmflabs.org domains.Jun 9 2018, 9:35 PM

It looks like everybody dropped the ball for actually getting these reviewed. The next patch (after the last one was merged 18 months ago) is literally changing three files to templates so this is utterly ridiculous. I'm going to start out-of-band requesting reviews again.

Joe mentioned this in T196968: Re-organize the apache configuration for MediaWiki in puppet.Jun 12 2018, 7:30 AM

Krinkle removed a project: Staging.Jul 13 2018, 12:03 AM

Krinkle added a subtask: T196968: Re-organize the apache configuration for MediaWiki in puppet.

Krinkle removed a subtask: T196968: Re-organize the apache configuration for MediaWiki in puppet.

Change 322602 abandoned by Alex Monk:
Move some production apache config files to templates

Reason:
Abandoning since it looks like ops plan to achieve pretty much this in a different way.

https://gerrit.wikimedia.org/r/322602

Change 322603 abandoned by Alex Monk:
Use production apache config on beta

Reason:
see dependency

https://gerrit.wikimedia.org/r/322603

Change 322604 abandoned by Alex Monk:
Get rid of old beta_sites class now just containing a load of ensure => absent

Reason:
see dependencies

https://gerrit.wikimedia.org/r/322604

Change 468583 had a related patch set uploaded (by Alex Monk; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki::web::beta_sites: convert to using mediawiki::web::vhost

https://gerrit.wikimedia.org/r/468583

Change 468583 had a related patch set uploaded (by Alex Monk; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki::web::beta_sites: convert to using mediawiki::web::vhost

https://gerrit.wikimedia.org/r/468583

Change 468583 merged by Giuseppe Lavagetto:
[operations/puppet@production] mediawiki::web::beta_sites: convert to using mediawiki::web::vhost

https://gerrit.wikimedia.org/r/468583

Krenair closed this task as Resolved.Oct 22 2018, 2:17 PM

Krenair assigned this task to Joe.

Liuxinyu970226 removed a subscriber: Liuxinyu970226.Oct 27 2018, 7:33 AM

Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM