Setup a dedicated mediawiki host in Beta Cluster that we can use for security scanning
Closed, ResolvedPublic

Description

Sherif is doing great work scanning the Beta Cluster for security vulnerabilities, but we had to stop the scanning as it was negatively effecting other browser tests/user tests going on.

It'd be good if we could setup a separate mediawiki instance that only serves his scanning traffic, and thus leave the other two mediawikis to handle the browser tests and user tests that normally go on during the day.

Details

Reference
bz70181
bzimport raised the priority of this task from to High.
bzimport set Reference to bz70181.
greg created this task.Aug 29 2014, 4:56 PM
greg added a comment.Aug 29 2014, 4:57 PM

We'd like to get this going as soon as possible as the work is proving to be fruitful. Setting to High accordingly.

Joe added a comment.Aug 29 2014, 6:13 PM

Redirecting traffic based on a cookie in varnish can be subtle, although I expect beta to be much simpler than production, it's still something that will probably need some non-trivial effort.

Giving this a go.

gerritadmin wrote:

Change 158016 had a related patch set uploaded by Dduvall:
Labs: Varnish backend/director for isolated security audits

https://gerrit.wikimedia.org/r/158016

The new deployment-mediawiki03 instance is fully provisioned, and I've cherry picked the varnish patch on deployment-salt. I've verified that the instance receives traffic [only] if a "security_audit=1" cookie is set, but I'd appreciate a second set of eyes on it.

cherifmansour wrote:

Thanks Dan, will take a look tomorrow and test it, what is the url and domain I should hit?

The host should be the same (en.wikipedia.beta.wmflabs.org). You just need to make sure the requests contain a "security_audit=1" cookie.

To be on the safe side, you might want to ping the #wikimedia-qa IRC channel when you're ready to start, just so we can keep an eye on things.

cherifmansour wrote:

Will do

greg added a comment.Sep 10 2014, 5:58 PM

13:57 < bd808> mediawiki03 isn't in the scap pool yet I just noticed.
13:58 < bd808> so it has stale code

gerritadmin wrote:

Change 159520 had a related patch set uploaded by BryanDavis:
beta: add deployment-mediawiki03 to scap targets

https://gerrit.wikimedia.org/r/159520

I've cherry-picked the patch to deployment-salt.eqiad.wmflabs and the last scap deployment seems to have synced to deployment-mediawiki03.

dduvall@deployment-mediawiki03:~$ ls -ld /srv/mediawiki/
drwxr-xr-x 12 mwdeploy mwdeploy 4096 Sep 11 21:35 /srv/mediawiki/

gerritadmin wrote:

Change 159520 merged by Dzahn:
beta: add deployment-mediawiki03 to scap targets

https://gerrit.wikimedia.org/r/159520

greg added a comment.Sep 12 2014, 12:16 AM

Are we all good here, then?

hashar removed a subscriber: hashar.Nov 24 2014, 9:12 PM
Qgil removed a subscriber: Qgil.Nov 25 2014, 8:20 AM
greg updated the task description. (Show Details)Mar 24 2015, 10:26 PM
greg set Security to None.
hashar added a subscriber: hashar.Mar 25 2015, 9:13 AM

Still waiting for https://gerrit.wikimedia.org/r/#/c/158016/ to be merged.

That puppet change for varnish needs to be rebased. You can get it applied on the beta cluster varnish by cherry picking the change on the local puppet master. That will give additional confidence to ops to review it.

Note that the old mediawiki03 doesn't exist at all anymore...

So another option is to just hit apache directly on one host - just open that up to a public IP. but I guess that doesn't make it similar enough to prod...

The last scan found an issue in varnish, so there is benefit to having it
goo through he entire stack.

Don't we have the hhmv cookie->back end logic still in varnish? Isn't this
basically the same thing?

hashar removed a subscriber: hashar.Mar 25 2015, 2:58 PM
greg renamed this task from Setup a mediawiki03 (or what not) on Beta Cluster that we can direct the security scanning work to to Setup a mediawiki033 on Beta Cluster that we can direct the security scanning work to.Jun 25 2015, 5:18 PM

Change 158016 had a related patch set uploaded (by Greg Grossmeier):
beta: varnish backend/director for isolated security audits

https://gerrit.wikimedia.org/r/158016

greg added a subscriber: BBlack.Jun 25 2015, 7:20 PM

Note that the old mediawiki03 doesn't exist at all anymore...

Sure it does: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-mediawiki03.deployment-prep.eqiad.wmflabs :)

@csteipp: This'll probably need some help from @BBlack to finish in addition to whatever @dduvall can do. We can commit to helping as much as we can, with that caveat.

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJun 25 2015, 7:20 PM

It's a different mediawiki03! Just because it's the same name and runs the same code does not mean it is the same! THEY ARE PEOPLE, NOT INTERCHANGEABLE MACHINES, DAMMIT!

greg renamed this task from Setup a mediawiki033 on Beta Cluster that we can direct the security scanning work to to Setup a dedicated mediawiki host in Beta Cluster that we can use for security scanning.Jun 25 2015, 7:50 PM
greg added a project: Security-Team.

Change 223391 had a related patch set uploaded (by BryanDavis):
beta: include deployment-mediawiki03 in scap targets

https://gerrit.wikimedia.org/r/223391

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptJul 7 2015, 7:55 PM

Paired with @demon and @thcipriani in rewriting the patch as much of the Puppet configuration for caches has since been refactored. It's currently cherry-picked on deployment-salt and we verified that it's working correctly.

Note that the new patch checks for a X-Wikimedia-Security-Audit: 1 header instead of a security_audit=1 cookie, so whoever performs the scans should make sure to include that header for every request.

Change 223391 merged by Yuvipanda:
beta: include deployment-mediawiki03 in scap targets

https://gerrit.wikimedia.org/r/223391

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 4 2015, 4:25 PM

Change 158016 merged by BBlack:
beta: varnish backend/director for isolated security audits

https://gerrit.wikimedia.org/r/158016

BBlack added a comment.Aug 6 2015, 7:41 PM

Varnish part is merged (along with a bunch of minor followup fixes to figure out how to get it to not break the prod caches with no security_audit backend defined!)

dduvall closed this task as Resolved.Aug 6 2015, 8:18 PM

Thanks a ton, @BBlack

@csteipp, Beta Cluster should be good to go. Let me know if you have any other questions about the setup.

Mentioned in SAL (#wikimedia-releng) [2018-07-14T21:20:10Z] <Krenair> redirected security_audit traffic (see T72181) traffic from deployment-mediawiki06 to deployment-mediawiki-09 to fix puppet on varnish (06 was deleted in T192996)