Page MenuHomePhabricator

Unit tests for apache config/rewrites
Open, NormalPublic

Description

We have a huge amount of apache rewrites, it'd be really nice if we had a way to test them so that we know for an input domain we get an output domain

I think there is some simplification that can be done by ordering the rewrites, and then needing less conditionals on some of the later rewrites etc


Version: wmf-deployment
Severity: enhancement
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=43266

Details

Reference
bz55857

Event Timeline

bzimport raised the priority of this task from to Normal.Nov 22 2014, 2:14 AM
bzimport set Reference to bz55857.
bzimport added a subscriber: Unknown Object (MLST).
Reedy created this task.Oct 18 2013, 12:47 AM

meanwhile the Apache config moved from apache-config to the mediawiki module

elukey added a subscriber: elukey.EditedNov 16 2016, 8:51 PM

Idea worth to discuss imo: setting LogLevel rewrite:trace8 in the apache logs (only for testing) is really useful:

elukey@deployment-mediawiki06:~$ curl http://localhost --header "Host: wikipedia.beta.wmflabs.org"
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="http://www.wikipedia.beta.wmflabs.org/">here</a>.</p>
</body></html>

elukey@deployment-mediawiki06:/var/log$ sudo tail -f apache2.log | grep "\[rewrite:"  | cut -d " " -f 16-
applying pattern '^(.*)$' to uri '/wiki/Main_Page'
RewriteCond: input='en.wikipedia.beta.wmflabs.org' pattern='=commons.wikipedia.beta.wmflabs.org' => not-matched
pass through /wiki/Main_Page
init rewrite engine with requested uri /
applying pattern '.' to uri '/'
setting env variable 'RW_PROTO' to ''
applying pattern '.' to uri '/'
RewriteCond: input='' pattern='!=https' => matched
setting env variable 'RW_PROTO' to 'http'
applying pattern '^/([a-z]{2}|meta)/(.*)$' to uri '/'
applying pattern '/apple-app-site-association' to uri '/'
applying pattern '^(.*)$' to uri '/'
rewrite '/' -> 'http://www.wikipedia.beta.wmflabs.org/'
explicitly forcing redirect with http://www.wikipedia.beta.wmflabs.org/
escaping http://www.wikipedia.beta.wmflabs.org/ for redirect
elukey@deployment-mediawiki06:/var/log$ curl http://localhost --header "Host: en.wikipedia.beta.wmflabs.org" -i
HTTP/1.1 301 Moved Permanently
Date: Wed, 16 Nov 2016 21:00:21 GMT
Server: deployment-mediawiki06.deployment-prep.eqiad.wmflabs
X-Powered-By: HHVM/3.12.7
X-Content-Type-Options: nosniff
Cache-control: s-maxage=1200, must-revalidate, max-age=0
P3P: CP="This is not a P3P policy! See https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAutoLogin/P3P for more info."
Vary: Accept-Encoding,X-Forwarded-Proto,Cookie,Authorization
Location: https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page
Last-Modified: Wed, 16 Nov 2016 21:00:21 GMT
Backend-Timing: D=66998 t=1479330021366903
Content-Length: 0
Content-Type: text/html; charset=utf-8

elukey@deployment-mediawiki06:/var/log$ sudo tail -f apache2.log | grep "\[rewrite:"  | cut -d " " -f 16-
applying pattern '^/robots\\.txt$' to uri '/wiki/Main_Page'
applying pattern '^/$' to uri '/wiki/Main_Page'
applying pattern '^/favicon\\.ico$' to uri '/wiki/Main_Page'
applying pattern '^/apple-touch-icon\\.png$' to uri '/wiki/Main_Page'
applying pattern '^/w/skins/.*$' to uri '/wiki/Main_Page'
applying pattern '^/w/resources/.*$' to uri '/wiki/Main_Page'
applying pattern '^/w/extensions/.*$' to uri '/wiki/Main_Page'
applying pattern '^(.*)$' to uri '/wiki/Main_Page'
RewriteCond: input='en.wikipedia.beta.wmflabs.org' pattern='=commons.wikipedia.beta.wmflabs.org' => not-matched
pass through /wiki/Main_Page
init rewrite engine with requested uri /
applying pattern '^/$' to uri '/'
rewrite '/' -> 'fcgi://127.0.0.1:9000/srv/mediawiki/docroot/wikipedia.org/w/index.php'
forcing proxy-throughput with fcgi://127.0.0.1:9000/srv/mediawiki/docroot/wikipedia.org/w/index.php

It might be hacky but we could use mod_rewrite's error log output to unit test a specific set of HTTP requests that we expect Apache to handle in a specific way (not only returning the expected result but also with the expected steps).

hashar added a subscriber: hashar.Nov 16 2016, 9:27 PM

operations/apache-config.git had a test suite written by Tim. Based on curl it was exercising the apache conf quite nicely. You can find it in origin/HEAD^ in /test/ . Maybe we can try hooking it in Jenkins.

I also tried to do get a script doing an Apache config check against both puppet.git and mediawiki-config.git T72068 ( lame dirty code for Jenkins is https://gerrit.wikimedia.org/r/#/c/166033/2/jjb/operations-apache-config.yaml ). Did not go well though :-(