Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ssastry
	Apr 8 2016, 9:23 PM

Description

As part of T120345: Set up mass visual diff testing with a custom install of mediawiki, I have created two labs VMs mw-base.wikitextexp and mw-expt.wikitextexp. On each VM, I've configured a multiwiki mediawiki-vagrant setup. Right now, they each have 41 wikis. So, I need 41 proxies set up for each VM.

One solution is to go horizon.wikimedia.org and fill out a form for each of the 82 proxies. That would work. But, if I add / remove wikis or if an instance goes down and I need to do this again, it would be somewhat painful.

So, filing this ticket for labs folks to figure out the best maintainable solution that might also arise in other contexts for other projects.

Details

	Subject	Repo	Branch	Lines +/-
	labs: Alias floating IPs in wikitextexp project as well	operations/puppet	production	+1 -0

Customize query in gerrit

Related Objects

Mentioned In: T211367: Setting up bulk proxies for two multi-wiki mw-vagrant labs vms
T201202: Replace bare-metal promethium.wikitextexp.eqiad.wmflabs with a VM or VMs
T120345: Set up mass visual diff testing with a custom install of mediawiki
Mentioned Here: T120345: Set up mass visual diff testing with a custom install of mediawiki

Event Timeline

ssastry created this task.Apr 8 2016, 9:23 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 8 2016, 9:23 PM

ssastry mentioned this in T120345: Set up mass visual diff testing with a custom install of mediawiki.Apr 8 2016, 9:32 PM

@Andrew and @yuvipanda allocated 2 floating IPs to the project and setup a wildcard DNS record for each: *.base.wikitextexp.wmflabs.org and *.expt.wikitextexp.wmflabs.org. The floating IPs were then assigned to the appropriate host via horizon.

Then we set the proper domain in /srv/mediawiki-vagrant/puppet/hieradata/local.yaml on each host:

mediawiki::multiwiki::base_domain: ".expt.wikitextexp.wmflabs.org"

and forced a puppet run to apply the change.

Mediawiki-Vagrant in Labs is setup to expect a reverse proxy in front of it that maps port 80/443 to 8080 on the host. MediaWiki is configured to generate links that do not include a port number. Since we don't have a proxy this causes problems. We need to either (a) change the Puppet config to set the proper port number, or (b) add a port 80 -> port 8080 reverse proxy on each host. Option A would require a dirty diff of Vagrantfile in /srv/mediawiki-vagrant. Option B should be a pretty simple nginx config.

Setup reverse proxy using nginx:

$ sudo apt-get install nginx-light
$ sudo vim /etc/nginx/sites-enabled/default

upstream wiki {
        server 127.0.0.1:8080;
}
server {
        listen 80 default_server;
        listen [::]:80 default_server ipv6only=on;
        location / {
                proxy_pass       http://wiki;
                proxy_set_header Host            $host;
                proxy_set_header X-Forwarded-For $remote_addr;
        }
}

Wikis on each vm can be seen via the MediaWiki-Vagrant error page:

Restricted Application added a project: User-bd808. · View Herald TranscriptApr 8 2016, 11:21 PM

@ssastry can I remove the proxies you added by hand? :D

In T132216#2192083, @yuvipanda wrote:

@ssastry can I remove the proxies you added by hand? :D

Yes. :)

Done.

The proxies aren't resolving from promethium.

<subbu> YuviPanda, what might be going on with this .. the dns names for the vm proxies aren't resolving on promethium.
<subbu> ssastry@promethium:~$ curl -v http://nl.base.wikitextexp.wmflabs.org/wiki/Sydney
<subbu> * Hostname was NOT found in DNS cache
<subbu> *   Trying 208.80.155.182...
<subbu> i can access that url in my browser however.
<subbu> also, the old proxies i had added still resolve. ssastry@promethium:~$ curl -v http://nl-mw-base.wmflabs.org/wiki/Sydney
<subbu> * Hostname was NOT found in DNS cache
<subbu> *   Trying 10.68.21.68...
<subbu> * Connected to nl-mw-base.wmflabs.org (10.68.21.68) port 80 (#0)

I was going to say use http_proxy=webproxy:8080, but actually it seems that host is labs baremetal (which I've never dealt with before) and I have no idea why you can't connect to that labs public IP from there. Probably something for network ops to look at.

In T132216#2193077, @ssastry wrote:

The proxies aren't resolving from promethium.

The proxy is actually resolving (Trying 208.80.155.182...), but access to the external IP is blocked by something. iptables doesn't show any active rules on promethium.eqiad.wmflabs, so I would guess that the routing block is at some higher level.

"nl-mw-base.wmflabs.org" is using split horizon DNS. It resolves to 208.80.155.156 from outside the labs network and 10.68.21.68 from inside the network. "*.base.wikitextexp.wmflabs.org" resolves to 208.80.155.182 both inside and outside the labs network. I didn't see anything in the horizon interface that would let us setup different records for internal labs resolution.

Using export http_proxy=http://webproxy.eqiad.wmnet:8080 as @Krenair suggested doesn't work either. I think this is because webproxy is only usable from inside the production network.

Krenair added a project: netops.Apr 10 2016, 3:36 AM

Restricted Application added a project: SRE. · View Herald TranscriptApr 10 2016, 3:36 AM

"nl-mw-base.wmflabs.org" is using split horizon DNS. It resolves to 208.80.155.156 from outside the labs network and 10.68.21.68 from inside the network. "*.base.wikitextexp.wmflabs.org" resolves to 208.80.155.182 both inside and outside the labs network. I didn't see anything in the horizon interface that would let us setup different records for internal labs resolution.

It's not configurable in Horizon, there is a hardcoded list of projects whose instances get split horizon DNS. project-proxy is listed, wikitextexp is not. https://gerrit.wikimedia.org/r/#/c/268921/ will change this.

In T132216#2193290, @bd808 wrote:

Using export http_proxy=http://webproxy.eqiad.wmnet:8080 as @Krenair suggested doesn't work either. I think this is because webproxy is only usable from inside the production network.

Yes, even being a baremetal host this is still in labs and therefore cannot be allowed access to webproxy.

Change 282511 had a related patch set uploaded (by Yuvipanda):
labs: Alias floating IPs in wikitextexp project as well

https://gerrit.wikimedia.org/r/282511

gerritbot added a project: Patch-For-Review.Apr 10 2016, 5:16 AM

^ is what you need. I'll merge on Monday :)

While that may be a workaround, it doesn't fix the fact that promethium cannot connect to 208.80.155.182

faidon removed a project: netops.Apr 11 2016, 2:39 PM

In T132216#2193397, @Krenair wrote:

While that may be a workaround, it doesn't fix the fact that promethium cannot connect to 208.80.155.182

I think that baremetal labs are meant to be treated like VMs, hence all the usual restrictions hold true. In the VMs case, NAT would take care of that, but seems like there is no such thing here.

bd808 removed a project: User-bd808.May 9 2016, 1:46 AM

Change 282511 abandoned by Yuvipanda:
labs: Alias floating IPs in wikitextexp project as well

https://gerrit.wikimedia.org/r/282511

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:43 PM

ssastry mentioned this in T201202: Replace bare-metal promethium.wikitextexp.eqiad.wmflabs with a VM or VMs.Oct 9 2018, 5:15 PM

ssastry mentioned this in T211367: Setting up bulk proxies for two multi-wiki mw-vagrant labs vms.Dec 6 2018, 7:04 PM