Sherif is doing great work scanning the Beta Cluster for security vulnerabilities, but we had to stop the scanning as it was negatively effecting other browser tests/user tests going on.
It'd be good if we could setup a separate mediawiki instance that only serves his scanning traffic, and thus leave the other two mediawikis to handle the browser tests and user tests that normally go on during the day.
We'd like to get this going as soon as possible as the work is proving to be fruitful. Setting to High accordingly.
Redirecting traffic based on a cookie in varnish can be subtle, although I expect beta to be much simpler than production, it's still something that will probably need some non-trivial effort.
gerritadmin wrote:
Change 158016 had a related patch set uploaded by Dduvall:
Labs: Varnish backend/director for isolated security audits
The new deployment-mediawiki03 instance is fully provisioned, and I've cherry picked the varnish patch on deployment-salt. I've verified that the instance receives traffic [only] if a "security_audit=1" cookie is set, but I'd appreciate a second set of eyes on it.
cherifmansour wrote:
Thanks Dan, will take a look tomorrow and test it, what is the url and domain I should hit?
The host should be the same (en.wikipedia.beta.wmflabs.org). You just need to make sure the requests contain a "security_audit=1" cookie.
To be on the safe side, you might want to ping the #wikimedia-qa IRC channel when you're ready to start, just so we can keep an eye on things.
13:57 < bd808> mediawiki03 isn't in the scap pool yet I just noticed.
13:58 < bd808> so it has stale code
gerritadmin wrote:
Change 159520 had a related patch set uploaded by BryanDavis:
beta: add deployment-mediawiki03 to scap targets
I've cherry-picked the patch to deployment-salt.eqiad.wmflabs and the last scap deployment seems to have synced to deployment-mediawiki03.
dduvall@deployment-mediawiki03:~$ ls -ld /srv/mediawiki/
drwxr-xr-x 12 mwdeploy mwdeploy 4096 Sep 11 21:35 /srv/mediawiki/
gerritadmin wrote:
Change 159520 merged by Dzahn:
beta: add deployment-mediawiki03 to scap targets
That puppet change for varnish needs to be rebased. You can get it applied on the beta cluster varnish by cherry picking the change on the local puppet master. That will give additional confidence to ops to review it.
So another option is to just hit apache directly on one host - just open that up to a public IP. but I guess that doesn't make it similar enough to prod...
The last scan found an issue in varnish, so there is benefit to having it
goo through he entire stack.
Don't we have the hhmv cookie->back end logic still in varnish? Isn't this
basically the same thing?
Change 158016 had a related patch set uploaded (by Greg Grossmeier):
beta: varnish backend/director for isolated security audits
Sure it does: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-mediawiki03.deployment-prep.eqiad.wmflabs :)
@csteipp: This'll probably need some help from @BBlack to finish in addition to whatever @dduvall can do. We can commit to helping as much as we can, with that caveat.
It's a different mediawiki03! Just because it's the same name and runs the same code does not mean it is the same! THEY ARE PEOPLE, NOT INTERCHANGEABLE MACHINES, DAMMIT!
Change 223391 had a related patch set uploaded (by BryanDavis):
beta: include deployment-mediawiki03 in scap targets
Paired with @demon and @thcipriani in rewriting the patch as much of the Puppet configuration for caches has since been refactored. It's currently cherry-picked on deployment-salt and we verified that it's working correctly.
Note that the new patch checks for a X-Wikimedia-Security-Audit: 1 header instead of a security_audit=1 cookie, so whoever performs the scans should make sure to include that header for every request.
Change 223391 merged by Yuvipanda:
beta: include deployment-mediawiki03 in scap targets
Change 158016 merged by BBlack:
beta: varnish backend/director for isolated security audits
Varnish part is merged (along with a bunch of minor followup fixes to figure out how to get it to not break the prod caches with no security_audit backend defined!)
Mentioned in SAL (#wikimedia-releng) [2018-07-14T21:20:10Z] <Krenair> redirected security_audit traffic (see T72181) traffic from deployment-mediawiki06 to deployment-mediawiki-09 to fix puppet on varnish (06 was deleted in T192996)