Page MenuHomePhabricator

Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm
Closed, ResolvedPublic

Description

As part of T120345: Set up mass visual diff testing with a custom install of mediawiki, I have created two labs VMs mw-base.wikitextexp and mw-expt.wikitextexp. On each VM, I've configured a multiwiki mediawiki-vagrant setup. Right now, they each have 41 wikis. So, I need 41 proxies set up for each VM.

One solution is to go horizon.wikimedia.org and fill out a form for each of the 82 proxies. That would work. But, if I add / remove wikis or if an instance goes down and I need to do this again, it would be somewhat painful.

So, filing this ticket for labs folks to figure out the best maintainable solution that might also arise in other contexts for other projects.

Event Timeline

ssastry created this task.Apr 8 2016, 9:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 8 2016, 9:23 PM
bd808 added a comment.Apr 8 2016, 10:20 PM

@Andrew and @yuvipanda allocated 2 floating IPs to the project and setup a wildcard DNS record for each: *.base.wikitextexp.wmflabs.org and *.expt.wikitextexp.wmflabs.org. The floating IPs were then assigned to the appropriate host via horizon.

Then we set the proper domain in /srv/mediawiki-vagrant/puppet/hieradata/local.yaml on each host:

mediawiki::multiwiki::base_domain: ".expt.wikitextexp.wmflabs.org"

and forced a puppet run to apply the change.

bd808 added a comment.Apr 8 2016, 10:32 PM

Mediawiki-Vagrant in Labs is setup to expect a reverse proxy in front of it that maps port 80/443 to 8080 on the host. MediaWiki is configured to generate links that do not include a port number. Since we don't have a proxy this causes problems. We need to either (a) change the Puppet config to set the proper port number, or (b) add a port 80 -> port 8080 reverse proxy on each host. Option A would require a dirty diff of Vagrantfile in /srv/mediawiki-vagrant. Option B should be a pretty simple nginx config.

bd808 closed this task as Resolved.Apr 8 2016, 11:21 PM
bd808 claimed this task.

Setup reverse proxy using nginx:

$ sudo apt-get install nginx-light
$ sudo vim /etc/nginx/sites-enabled/default
upstream wiki {
        server 127.0.0.1:8080;
}
server {
        listen 80 default_server;
        listen [::]:80 default_server ipv6only=on;
        location / {
                proxy_pass       http://wiki;
                proxy_set_header Host            $host;
                proxy_set_header X-Forwarded-For $remote_addr;
        }
}

Wikis on each vm can be seen via the MediaWiki-Vagrant error page:

Restricted Application added a project: User-bd808. · View Herald TranscriptApr 8 2016, 11:21 PM

@ssastry can I remove the proxies you added by hand? :D

@ssastry can I remove the proxies you added by hand? :D

Yes. :)

The proxies aren't resolving from promethium.

<subbu> YuviPanda, what might be going on with this .. the dns names for the vm proxies aren't resolving on promethium.
<subbu> ssastry@promethium:~$ curl -v http://nl.base.wikitextexp.wmflabs.org/wiki/Sydney
<subbu> * Hostname was NOT found in DNS cache
<subbu> *   Trying 208.80.155.182...
<subbu> i can access that url in my browser however.
<subbu> also, the old proxies i had added still resolve. ssastry@promethium:~$ curl -v http://nl-mw-base.wmflabs.org/wiki/Sydney
<subbu> * Hostname was NOT found in DNS cache
<subbu> *   Trying 10.68.21.68...
<subbu> * Connected to nl-mw-base.wmflabs.org (10.68.21.68) port 80 (#0)
Krenair added a subscriber: Krenair.EditedApr 9 2016, 9:12 PM

I was going to say use http_proxy=webproxy:8080, but actually it seems that host is labs baremetal (which I've never dealt with before) and I have no idea why you can't connect to that labs public IP from there. Probably something for network ops to look at.

bd808 reopened this task as Open.Apr 10 2016, 2:00 AM
bd808 removed bd808 as the assignee of this task.

The proxies aren't resolving from promethium.

The proxy is actually resolving (Trying 208.80.155.182...), but access to the external IP is blocked by something. iptables doesn't show any active rules on promethium.eqiad.wmflabs, so I would guess that the routing block is at some higher level.

"nl-mw-base.wmflabs.org" is using split horizon DNS. It resolves to 208.80.155.156 from outside the labs network and 10.68.21.68 from inside the network. "*.base.wikitextexp.wmflabs.org" resolves to 208.80.155.182 both inside and outside the labs network. I didn't see anything in the horizon interface that would let us setup different records for internal labs resolution.

Using export http_proxy=http://webproxy.eqiad.wmnet:8080 as @Krenair suggested doesn't work either. I think this is because webproxy is only usable from inside the production network.

Restricted Application added a project: SRE. · View Herald TranscriptApr 10 2016, 3:36 AM
Krenair added a comment.EditedApr 10 2016, 3:51 AM

"nl-mw-base.wmflabs.org" is using split horizon DNS. It resolves to 208.80.155.156 from outside the labs network and 10.68.21.68 from inside the network. "*.base.wikitextexp.wmflabs.org" resolves to 208.80.155.182 both inside and outside the labs network. I didn't see anything in the horizon interface that would let us setup different records for internal labs resolution.

It's not configurable in Horizon, there is a hardcoded list of projects whose instances get split horizon DNS. project-proxy is listed, wikitextexp is not. https://gerrit.wikimedia.org/r/#/c/268921/ will change this.

Using export http_proxy=http://webproxy.eqiad.wmnet:8080 as @Krenair suggested doesn't work either. I think this is because webproxy is only usable from inside the production network.

Yes, even being a baremetal host this is still in labs and therefore cannot be allowed access to webproxy.

Change 282511 had a related patch set uploaded (by Yuvipanda):
labs: Alias floating IPs in wikitextexp project as well

https://gerrit.wikimedia.org/r/282511

^ is what you need. I'll merge on Monday :)

While that may be a workaround, it doesn't fix the fact that promethium cannot connect to 208.80.155.182

akosiaris triaged this task as Medium priority.Apr 20 2016, 11:26 AM
akosiaris added a subscriber: akosiaris.

While that may be a workaround, it doesn't fix the fact that promethium cannot connect to 208.80.155.182

I think that baremetal labs are meant to be treated like VMs, hence all the usual restrictions hold true. In the VMs case, NAT would take care of that, but seems like there is no such thing here.

Change 282511 abandoned by Yuvipanda:
labs: Alias floating IPs in wikitextexp project as well

https://gerrit.wikimedia.org/r/282511

bd808 closed this task as Resolved.Dec 6 2018, 9:50 PM
bd808 claimed this task.

Closing this out. 2.5 years with no updates so... yeah.