Page MenuHomePhabricator

Create temporary http -> https reverse proxy for MerlBot
Closed, ResolvedPublic

Description

Create a reverse proxy that will transform HTTP requests to HTTPS requests transparently for MerlBot. This is a temporary fix until the bot can be updated to properly use HTTPS natively.

Event Timeline

Restricted Application added subscribers: Zppix, Aklapper. · View Herald Transcript
bd808 triaged this task as High priority.Jun 7 2016, 6:08 PM
bd808 moved this task from Backlog to Doing on the Community-Tech-Tool-Labs board.

Change 293223 had a related patch set uploaded (by BryanDavis):
role::toollabs::merlbot_proxy

https://gerrit.wikimedia.org/r/293223

Change 293223 merged by Yuvipanda:
tools: Add role::toollabs::merlbot_proxy

https://gerrit.wikimedia.org/r/293223

Proxy is up and running at http://tools-merlbot-proxy.tools.eqiad.wmflabs:80

$ curl -v -X POST --proxy http://tools-merlbot-proxy.tools.eqiad.wmflabs:80 -A 'fake MerlBot' 'http://en.wikipedia.org/w/api.php' --data 'action=query&meta=tokens&type=login&format=json'
* Hostname was NOT found in DNS cache
*   Trying 10.68.16.208...
* Connected to tools-merlbot-proxy.tools.eqiad.wmflabs (10.68.16.208) port 80 (#0)
> POST http://en.wikipedia.org/w/api.php HTTP/1.1
> User-Agent: fake MerlBot
> Host: en.wikipedia.org
> Accept: */*
> Proxy-Connection: Keep-Alive
> Content-Length: 47
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 47 out of 47 bytes
< HTTP/1.1 200 OK
* Server nginx/1.11.1 is not blacklisted
< Server: nginx/1.11.1
< Date: Wed, 08 Jun 2016 03:03:29 GMT
< Content-Type: application/json; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Powered-By: HHVM/3.12.1
< X-Content-Type-Options: nosniff
< Cache-control: private, must-revalidate, max-age=0
< P3P: CP="This is not a P3P policy! See https://en.wikipedia.org/wiki/Special:CentralAutoLogin/P3P for more info."
< X-Frame-Options: SAMEORIGIN
< Vary: Accept-Encoding
< Set-Cookie: enwikiSession=erpum8vu7d4q8nljhc7dhqvafb1m6j4q; path=/; secure; httponly
< Set-Cookie: forceHTTPS=true; path=/; httponly
< Backend-Timing: D=47156 t=1465355009264366
< X-Varnish: 3376812114, 3972462337
< Via: 1.1 varnish, 1.1 varnish
< Age: 0
< X-Cache: cp1055 pass, cp1054 pass
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Set-Cookie: WMF-Last-Access=08-Jun-2016;Path=/;HttpOnly;secure;Expires=Sun, 10 Jul 2016 00:00:00 GMT
< X-Analytics: https=1;nocookies=1
< X-Client-IP: 10.68.16.208
< Set-Cookie: GeoIP=:::::v4; Path=/; secure; Domain=.wikipedia.org
< Labs-TLS-Bandaid: on
<
* Connection #0 to host tools-merlbot-proxy.tools.eqiad.wmflabs left intact
{"batchcomplete":"","query":{"tokens":{"logintoken":"b2d87319a225328c201164cd1da4712e57578b01+\\"}}}

And with java on the grid with java from P3219:

$ jsub -stderr -once -l release=trusty -mem 4g java -Dhttp.proxyHost=tools-merlbot-proxy.tools.eqiad.wmflabs -Dhttp.ProxyPort=80 ProxyTest
Your job 7235579 ("java") has been submitted
$ cat java.out

Sending 'POST' request to URL : http://en.wikipedia.org/w/api.php
Response Code : 200
null: [HTTP/1.1 200 OK]
Age: [0]
Cache-control: [private, must-revalidate, max-age=0]
Backend-Timing: [D=38530 t=1465355137835159]
X-Client-IP: [10.68.16.208]
Set-Cookie: [GeoIP=:::::v4; Path=/; secure; Domain=.wikipedia.org, WMF-Last-Access=08-Jun-2016;Path=/;HttpOnly;secure;Expires=Sun, 10 Jul 2016 00:00:00 GMT, forceHTTPS=true; path=/; httponly, enwikiSession=esdrat8n7rooll0oha744881iuq21ucb; path=/; secure; httponly]
Connection: [keep-alive]
Server: [nginx/1.11.1]
X-Powered-By: [HHVM/3.12.1]
X-Cache: [cp1066 pass, cp1054 pass]
X-Content-Type-Options: [nosniff]
X-Frame-Options: [SAMEORIGIN]
X-Varnish: [2450668964, 3972950569]
Strict-Transport-Security: [max-age=31536000; includeSubDomains; preload]
Vary: [Accept-Encoding]
Labs-TLS-Bandaid: [on]
Transfer-Encoding: [chunked]
Date: [Wed, 08 Jun 2016 03:05:37 GMT]
P3P: [CP="This is not a P3P policy! See https://en.wikipedia.org/wiki/Special:CentralAutoLogin/P3P for more info."]
X-Analytics: [https=1;nocookies=1]
Via: [1.1 varnish, 1.1 varnish]
Content-Type: [application/json; charset=utf-8]
{"batchcomplete":"","query":{"tokens":{"logintoken":"ed15209299aac02242934845b2dbd03e57578b81+\\"}}}

The proxy is up and running using Puppet managed configuration.