Setting up bulk proxies for two multi-wiki mw-vagrant labs vms
Open, Needs TriagePublic

Description

For visual diff testing, the parsing team has two different labs vms: mw-base.wikitextexp and mw-expt.wikitextexp that each run a multi-wiki mediawiki install with 41 wikis each. Back in 2016 when I set this up, Yuvi had help set up mass proxies for the 82 wikis on both these vms so that we could access urls like http://en.expt.wikitextexp.wmflabs.org/wiki/Main_Page and http://en.base.wikitextexp.wmflabs.org/wiki/Main_Page and so on. These proxies are not visible https://horizon.wikimedia.org/project/proxy/, but the two labs VM have a floating IP each that are visible on horizon.

Now, as part of T204566: cloudvps: wikitextexp project trusty deprecation, I am building new VMs, transferring custom settings, data in DBs so that we can retire the older VMs. I am now at the process where I need to assign new floating IPs and then assign bulk proxies as with the old VMs.

T132216: Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm might be related to this task.

So, some questions:

  1. I don't have quota to assign new floating IPs. So, assuming I get new floating IPs, I can manually create the 82 proxies in horizon .. not the end of the world.
  2. I can dissociate the floating IPs and attach them to the new VMs. Question: will this now magically point the proxies to the new vms? Or is more fiddling required?
  3. Any other suggestions or options?
ssastry created this task.Thu, Dec 6, 7:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, Dec 6, 7:04 PM
Krenair added a subscriber: Krenair.EditedThu, Dec 6, 7:22 PM

Proxies *and* floating IPs? The point of the proxy system is to avoid floating IPs. It looks like you just have DNS records for *.expt.wikitextexp.wmflabs.org and *.base.wikitextexp.wmflabs.org pointing at your floating IPs. Not being listed in horizon also suggests they do not exist. What makes you think proxies are involved here?

T132216: Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm is still open 2+ years later. Should that task have been closed at some point as working?


Assuming that the state from T132216 is the desired thing that we need to replicate with the new servers, here are relevant comments from the prior task:

@Andrew and @yuvipanda allocated 2 floating IPs to the project and setup a wildcard DNS record for each: *.base.wikitextexp.wmflabs.org and *.expt.wikitextexp.wmflabs.org. The floating IPs were then assigned to the appropriate host via horizon.

Then we set the proper domain in /srv/mediawiki-vagrant/puppet/hieradata/local.yaml on each host:

mediawiki::multiwiki::base_domain: ".expt.wikitextexp.wmflabs.org"

and forced a puppet run to apply the change.

Mediawiki-Vagrant in Labs is setup to expect a reverse proxy in front of it that maps port 80/443 to 8080 on the host. MediaWiki is configured to generate links that do not include a port number. Since we don't have a proxy this causes problems. We need to either (a) change the Puppet config to set the proper port number, or (b) add a port 80 -> port 8080 reverse proxy on each host. Option A would require a dirty diff of Vagrantfile in /srv/mediawiki-vagrant. Option B should be a pretty simple nginx config.

Setup reverse proxy using nginx:

$ sudo apt-get install nginx-light
$ sudo vim /etc/nginx/sites-enabled/default
upstream wiki {
        server 127.0.0.1:8080;
}
server {
        listen 80 default_server;
        listen [::]:80 default_server ipv6only=on;
        location / {
                proxy_pass       http://wiki;
                proxy_set_header Host            $host;
                proxy_set_header X-Forwarded-For $remote_addr;
        }
}

Wikis on each vm can be seen via the MediaWiki-Vagrant error page:

I think this means that we need to:

  1. Add quota to allow 2 more floating IPs for the project
  2. Attach floating ips to the new hosts
  3. Ensure that the new hosts have the necessary locally installed nginx reverse proxy
  4. Update the *.base.wikitextexp.wmflabs.org and *.expt.wikitextexp.wmflabs.org DNS entries to point to the new hosts
  5. Verify that things work as expected
  6. Release the old floating IPs
  7. Reduce the floating IP quota back to 2
  8. Delete the old vms
  9. Profit!
bd808 added a comment.Thu, Dec 6, 7:37 PM

I think this means that we need to:

  1. Add quota to allow 2 more floating IPs for the project
  2. Attach floating ips to the new hosts
  3. Ensure that the new hosts have the necessary locally installed nginx reverse proxy
  4. Update the *.base.wikitextexp.wmflabs.org and *.expt.wikitextexp.wmflabs.org DNS entries to point to the new hosts
  5. Verify that things work as expected
  6. Release the old floating IPs
  7. Reduce the floating IP quota back to 2
  8. Delete the old vms
  9. Profit!

Alternately, we could make this less of a special snowflake project by creating the 82 proxies via the project-proxy service. I think it would be possible to script this rather than making @ssastry click the buttons over and over in the Horizon UI. Then we could clean up the manually created wildcard DNS and also save a couple of floating IPs for another project.

If we go this direction then the proxied hostnames will be something like X-base-wikitextexp.wmflabs.org and X-expt-wikitextexp.wmflabs.org instead. Would that work @ssastry (assuming I help you get the config on the MediaWiki-Vagrant side working properly as well)? Do you have a list of the hostnames that need to be setup? I'm assuming it is a symmetrical set of X.base and X.expt entries for 41 language/project variations?

I think this means that we need to:

  1. Add quota to allow 2 more floating IPs for the project
  2. Attach floating ips to the new hosts
  3. Ensure that the new hosts have the necessary locally installed nginx reverse proxy
  4. Update the *.base.wikitextexp.wmflabs.org and *.expt.wikitextexp.wmflabs.org DNS entries to point to the new hosts
  5. Verify that things work as expected
  6. Release the old floating IPs
  7. Reduce the floating IP quota back to 2
  8. Delete the old vms
  9. Profit!

Alternately, we could make this less of a special snowflake project

+1 to this.

by creating the 82 proxies via the project-proxy service. I think it would be possible to script this rather than making @ssastry click the buttons over and over in the Horizon UI. Then we could clean up the manually created wildcard DNS and also save a couple of floating IPs for another project.

If we go this direction then the proxied hostnames will be something like X-base-wikitextexp.wmflabs.org and X-expt-wikitextexp.wmflabs.org instead. Would that work @ssastry (assuming I help you get the config on the MediaWiki-Vagrant side working properly as well)?

This works. I would just need to update the config files on parsing-qa-01 and url bases on wiki which is easy to do.

Do you have a list of the hostnames that need to be setup? I'm assuming it is a symmetrical set of X.base and X.expt entries for 41 language/project variations?

Yes, symmetrical. Here is the list of 41 wikis.

ar
ckb
cu
cv
de
en
enwikisource
enwikivoyage
enwiktionary
es
eswikisource
eswikivoyage
eswiktionary
fr
frwikisource
frwikivoyage
frwiktionary
he
hi
hy
is
it
itwikisource
itwikivoyage
itwiktionary
ja
kaa
ka
ko
lbe
ln
mzn
nl
pl
pnb
pt
ru
sv
uk
uz
zh

Mentioned in SAL (#wikimedia-cloud) [2018-12-07T02:00:08Z] <bd808> Added BryanDavis (self) as project admin for T211367

bd808 claimed this task.Fri, Dec 7, 3:44 AM
bd808 moved this task from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.
bd808 added a comment.EditedFri, Dec 7, 3:54 AM

Proxy entries created using a python cli tool I made to make it easier to automate the process:

$ for h in $(cat hosts); do
  ./webproxy.py --project=wikitextexp add ${h}-base-wikitextexp http://10.68.16.144:8080
  ./webproxy.py --project=wikitextexp add ${h}-expt-wikitextexp http://10.68.23.251:8080
done
$ ./webproxy.py --project=wikitextexp list
domain                                           backend
================================================ ========================
ar-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ar-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
ckb-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
ckb-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
cu-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
cu-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
cv-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
cv-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
de-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
de-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
en-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
en-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
enwikisource-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
enwikisource-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
enwikivoyage-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
enwikivoyage-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
enwiktionary-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
enwiktionary-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
es-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
es-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
eswikisource-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
eswikisource-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
eswikivoyage-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
eswikivoyage-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
eswiktionary-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
eswiktionary-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
fr-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
fr-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
frwikisource-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
frwikisource-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
frwikivoyage-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
frwikivoyage-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
frwiktionary-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
frwiktionary-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
he-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
he-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
hi-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
hi-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
hy-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
hy-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
is-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
is-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
it-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
it-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
itwikisource-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
itwikisource-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
itwikivoyage-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
itwikivoyage-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
itwiktionary-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
itwiktionary-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
ja-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ja-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
ka-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ka-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
kaa-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
kaa-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
ko-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ko-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
lbe-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
lbe-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
ln-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ln-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
mw-expt-tests.wmflabs.org                        http://172.16.1.159:80
mzn-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
mzn-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
nl-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
nl-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
pl-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
pl-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
pnb-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
pnb-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
pt-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
pt-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
ru-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ru-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
sv-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
sv-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
uk-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
uk-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
uz-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
uz-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
zh-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
zh-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
bd808 added a comment.Fri, Dec 7, 5:06 AM
$ ssh wikitextexp-base-1002.wikitextexp.eqiad.wmflabs
$ cd /srv/mediawiki-vagrant
$ vim puppet/hieradata/local.yaml
... added a mediawiki::multiwiki::base_domain setting ...
$ grep mediawiki::multiwiki::base_domain puppet/hieradata/local.yaml
mediawiki::multiwiki::base_domain: "-base-wikitextexp.wmflabs.org"
$ vagrant provision
...
Failed miserably
$ vagrant ssh
$ mysql
# manually fixed root and vagrant users to use unix_socket auth
$ vagrant provision
...
$ vagrant ssh -- sudo service apache2 restart

Repeated on wikitextexp-expt-1002.wikitextexp.eqiad.wmflabs. All the wikis that I spot checked from the list are working now. See https://tools.wmflabs.org/openstack-browser/project/wikitextexp for the full list.

Once @ssastry verifies that things are working and can switch his process over to using the new instances we can clean up the wildcard DNS records and the old instances.

Thanks!!

$ vagrant provision
...
Failed miserably
$ vagrant ssh
$ mysql
# manually fixed root and vagrant users to use unix_socket auth

Can you say more what this manual fixing was about? Was this because of something I did wrong (or a missing puppet class or something transient)? Or do we need to this always? Asking because I am putting together all the steps in one place for next time.

bd808 added a comment.Fri, Dec 7, 4:25 PM

Can you say more what this manual fixing was about? Was this because of something I did wrong (or a missing puppet class or something transient)? Or do we need to this always? Asking because I am putting together all the steps in one place for next time.

I think it was related to your dump and then load of all database tables from the older deployment. When we switched the MediaWiki-Vagrant base image from Jessie to Stretch one of the changes was in how the root and vagrant MariaDB users authenticate to the server. Inside your VMs the mysql.user table's grants were expecting them to auth using a password, but the Puppet code was expecting unix_socket auth. The fix I applied ultimately was to use mysqld_safe --skip-grant-tables to start the service in recovery mode, connect as the root user, and then drop and recreate the grants for the root and vagrant users. Something like this:

$ service mysql stop
$ /usr/bin/mysqld_safe --skip-grant-tables &
$ mysql
mysql> drop user root@localhost
mysql> CREATE USER 'root'@'localhost' IDENTIFIED VIA unix_socket; GRANT ALL PRIVILEGES ON *.* to 'root'@'localhost';
mysql> drop user vagrant@localhost;
mysql> CREATE USER 'vagrant'@'localhost' IDENTIFIED VIA unix_socket; GRANT ALL PRIVILEGES ON *.* to 'vagrant'@'localhost';
mysql> flush privileges;
mysql> quit
$ fg
^C
$ service mysql start

Can you say more what this manual fixing was about? Was this because of something I did wrong (or a missing puppet class or something transient)? Or do we need to this always? Asking because I am putting together all the steps in one place for next time.

I think it was related to your dump and then load of all database tables from the older deployment.

Aha .. so, I should just dump the specific wiki dbs instead of all dbs and this would any snafus like this if other aspects of mysql connections with mediawiki changes.

Change 478377 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] wmcs: Add a cli script for managing dynamicproxy entries

https://gerrit.wikimedia.org/r/478377