Proxies *and* floating IPs? The point of the proxy system is to avoid floating IPs. It looks like you just have DNS records for *.expt.wikitextexp.wmflabs.org and *.base.wikitextexp.wmflabs.org pointing at your floating IPs. Not being listed in horizon also suggests they do not exist. What makes you think proxies are involved here?

T132216: Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm is still open 2+ years later. Should that task have been closed at some point as working?

Assuming that the state from T132216 is the desired thing that we need to replicate with the new servers, here are relevant comments from the prior task:

In T132216#2191882, @bd808 wrote:

@Andrew and @yuvipanda allocated 2 floating IPs to the project and setup a wildcard DNS record for each: *.base.wikitextexp.wmflabs.org and *.expt.wikitextexp.wmflabs.org. The floating IPs were then assigned to the appropriate host via horizon.

Then we set the proper domain in /srv/mediawiki-vagrant/puppet/hieradata/local.yaml on each host:

mediawiki::multiwiki::base_domain: ".expt.wikitextexp.wmflabs.org"

and forced a puppet run to apply the change.

In T132216#2191962, @bd808 wrote:

Mediawiki-Vagrant in Labs is setup to expect a reverse proxy in front of it that maps port 80/443 to 8080 on the host. MediaWiki is configured to generate links that do not include a port number. Since we don't have a proxy this causes problems. We need to either (a) change the Puppet config to set the proper port number, or (b) add a port 80 -> port 8080 reverse proxy on each host. Option A would require a dirty diff of Vagrantfile in /srv/mediawiki-vagrant. Option B should be a pretty simple nginx config.

In T132216#2192078, @bd808 wrote:

Setup reverse proxy using nginx:

$ sudo apt-get install nginx-light
$ sudo vim /etc/nginx/sites-enabled/default

upstream wiki {
        server 127.0.0.1:8080;
}
server {
        listen 80 default_server;
        listen [::]:80 default_server ipv6only=on;
        location / {
                proxy_pass       http://wiki;
                proxy_set_header Host            $host;
                proxy_set_header X-Forwarded-For $remote_addr;
        }
}

Wikis on each vm can be seen via the MediaWiki-Vagrant error page:

• http://foo.base.wikitextexp.wmflabs.org/
• http://foo.expt.wikitextexp.wmflabs.org/

I think this means that we need to:

1. Add quota to allow 2 more floating IPs for the project
2. Attach floating ips to the new hosts
3. Ensure that the new hosts have the necessary locally installed nginx reverse proxy
4. Update the *.base.wikitextexp.wmflabs.org and *.expt.wikitextexp.wmflabs.org DNS entries to point to the new hosts
5. Verify that things work as expected
6. Release the old floating IPs
7. Reduce the floating IP quota back to 2
8. Delete the old vms
9. Profit!

In T211367#4804190, @bd808 wrote:

I think this means that we need to:

1. Add quota to allow 2 more floating IPs for the project
2. Attach floating ips to the new hosts
3. Ensure that the new hosts have the necessary locally installed nginx reverse proxy
4. Update the *.base.wikitextexp.wmflabs.org and *.expt.wikitextexp.wmflabs.org DNS entries to point to the new hosts
5. Verify that things work as expected
6. Release the old floating IPs
7. Reduce the floating IP quota back to 2
8. Delete the old vms
9. Profit!

Alternately, we could make this less of a special snowflake project by creating the 82 proxies via the project-proxy service. I think it would be possible to script this rather than making @ssastry click the buttons over and over in the Horizon UI. Then we could clean up the manually created wildcard DNS and also save a couple of floating IPs for another project.

If we go this direction then the proxied hostnames will be something like X-base-wikitextexp.wmflabs.org and X-expt-wikitextexp.wmflabs.org instead. Would that work @ssastry (assuming I help you get the config on the MediaWiki-Vagrant side working properly as well)? Do you have a list of the hostnames that need to be setup? I'm assuming it is a symmetrical set of X.base and X.expt entries for 41 language/project variations?

In T211367#4804218, @bd808 wrote:

In T211367#4804190, @bd808 wrote:

I think this means that we need to:

1. Add quota to allow 2 more floating IPs for the project
2. Attach floating ips to the new hosts
3. Ensure that the new hosts have the necessary locally installed nginx reverse proxy
4. Update the *.base.wikitextexp.wmflabs.org and *.expt.wikitextexp.wmflabs.org DNS entries to point to the new hosts
5. Verify that things work as expected
6. Release the old floating IPs
7. Reduce the floating IP quota back to 2
8. Delete the old vms
9. Profit!

Alternately, we could make this less of a special snowflake project

+1 to this.

by creating the 82 proxies via the project-proxy service. I think it would be possible to script this rather than making @ssastry click the buttons over and over in the Horizon UI. Then we could clean up the manually created wildcard DNS and also save a couple of floating IPs for another project.

If we go this direction then the proxied hostnames will be something like X-base-wikitextexp.wmflabs.org and X-expt-wikitextexp.wmflabs.org instead. Would that work @ssastry (assuming I help you get the config on the MediaWiki-Vagrant side working properly as well)?

This works. I would just need to update the config files on parsing-qa-01 and url bases on wiki which is easy to do.

Do you have a list of the hostnames that need to be setup? I'm assuming it is a symmetrical set of X.base and X.expt entries for 41 language/project variations?

Yes, symmetrical. Here is the list of 41 wikis.

ar
ckb
cu
cv
de
en
enwikisource
enwikivoyage
enwiktionary
es
eswikisource
eswikivoyage
eswiktionary
fr
frwikisource
frwikivoyage
frwiktionary
he
hi
hy
is
it
itwikisource
itwikivoyage
itwiktionary
ja
kaa
ka
ko
lbe
ln
mzn
nl
pl
pnb
pt
ru
sv
uk
uz
zh

Mentioned in SAL (#wikimedia-cloud) [2018-12-07T02:00:08Z] <bd808> Added BryanDavis (self) as project admin for T211367

Proxy entries created using a python cli tool I made to make it easier to automate the process:

$ for h in $(cat hosts); do
  ./webproxy.py --project=wikitextexp add ${h}-base-wikitextexp http://10.68.16.144:8080
  ./webproxy.py --project=wikitextexp add ${h}-expt-wikitextexp http://10.68.23.251:8080
done
$ ./webproxy.py --project=wikitextexp list
domain                                           backend
================================================ ========================
ar-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ar-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
ckb-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
ckb-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
cu-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
cu-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
cv-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
cv-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
de-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
de-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
en-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
en-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
enwikisource-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
enwikisource-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
enwikivoyage-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
enwikivoyage-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
enwiktionary-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
enwiktionary-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
es-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
es-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
eswikisource-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
eswikisource-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
eswikivoyage-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
eswikivoyage-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
eswiktionary-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
eswiktionary-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
fr-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
fr-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
frwikisource-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
frwikisource-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
frwikivoyage-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
frwikivoyage-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
frwiktionary-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
frwiktionary-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
he-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
he-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
hi-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
hi-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
hy-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
hy-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
is-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
is-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
it-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
it-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
itwikisource-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
itwikisource-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
itwikivoyage-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
itwikivoyage-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
itwiktionary-base-wikitextexp.wmflabs.org        http://10.68.16.144:8080
itwiktionary-expt-wikitextexp.wmflabs.org        http://10.68.23.251:8080
ja-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ja-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
ka-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ka-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
kaa-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
kaa-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
ko-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ko-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
lbe-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
lbe-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
ln-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ln-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
mw-expt-tests.wmflabs.org                        http://172.16.1.159:80
mzn-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
mzn-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
nl-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
nl-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
pl-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
pl-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
pnb-base-wikitextexp.wmflabs.org                 http://10.68.16.144:8080
pnb-expt-wikitextexp.wmflabs.org                 http://10.68.23.251:8080
pt-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
pt-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
ru-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
ru-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
sv-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
sv-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
uk-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
uk-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
uz-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
uz-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080
zh-base-wikitextexp.wmflabs.org                  http://10.68.16.144:8080
zh-expt-wikitextexp.wmflabs.org                  http://10.68.23.251:8080

$ ssh wikitextexp-base-1002.wikitextexp.eqiad.wmflabs
$ cd /srv/mediawiki-vagrant
$ vim puppet/hieradata/local.yaml
... added a mediawiki::multiwiki::base_domain setting ...
$ grep mediawiki::multiwiki::base_domain puppet/hieradata/local.yaml
mediawiki::multiwiki::base_domain: "-base-wikitextexp.wmflabs.org"
$ vagrant provision
...
Failed miserably
$ vagrant ssh
$ mysql
# manually fixed root and vagrant users to use unix_socket auth
$ vagrant provision
...
$ vagrant ssh -- sudo service apache2 restart

Repeated on wikitextexp-expt-1002.wikitextexp.eqiad.wmflabs. All the wikis that I spot checked from the list are working now. See https://tools.wmflabs.org/openstack-browser/project/wikitextexp for the full list.

Once @ssastry verifies that things are working and can switch his process over to using the new instances we can clean up the wildcard DNS records and the old instances.

Thanks!!

In T211367#4805119, @bd808 wrote:

$ vagrant provision
...
Failed miserably
$ vagrant ssh
$ mysql
# manually fixed root and vagrant users to use unix_socket auth

Can you say more what this manual fixing was about? Was this because of something I did wrong (or a missing puppet class or something transient)? Or do we need to this always? Asking because I am putting together all the steps in one place for next time.

In T211367#4806270, @ssastry wrote:

Can you say more what this manual fixing was about? Was this because of something I did wrong (or a missing puppet class or something transient)? Or do we need to this always? Asking because I am putting together all the steps in one place for next time.

I think it was related to your dump and then load of all database tables from the older deployment. When we switched the MediaWiki-Vagrant base image from Jessie to Stretch one of the changes was in how the root and vagrant MariaDB users authenticate to the server. Inside your VMs the mysql.user table's grants were expecting them to auth using a password, but the Puppet code was expecting unix_socket auth. The fix I applied ultimately was to use mysqld_safe --skip-grant-tables to start the service in recovery mode, connect as the root user, and then drop and recreate the grants for the root and vagrant users. Something like this:

$ service mysql stop
$ /usr/bin/mysqld_safe --skip-grant-tables &
$ mysql
mysql> drop user root@localhost
mysql> CREATE USER 'root'@'localhost' IDENTIFIED VIA unix_socket; GRANT ALL PRIVILEGES ON *.* to 'root'@'localhost';
mysql> drop user vagrant@localhost;
mysql> CREATE USER 'vagrant'@'localhost' IDENTIFIED VIA unix_socket; GRANT ALL PRIVILEGES ON *.* to 'vagrant'@'localhost';
mysql> flush privileges;
mysql> quit
$ fg
^C
$ service mysql start

In T211367#4806297, @bd808 wrote:

In T211367#4806270, @ssastry wrote:

Can you say more what this manual fixing was about? Was this because of something I did wrong (or a missing puppet class or something transient)? Or do we need to this always? Asking because I am putting together all the steps in one place for next time.

I think it was related to your dump and then load of all database tables from the older deployment.

Aha .. so, I should just dump the specific wiki dbs instead of all dbs and this would any snafus like this if other aspects of mysql connections with mediawiki changes.

Change 478377 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] wmcs: Add a cli script for managing dynamicproxy entries

https://gerrit.wikimedia.org/r/478377

Change 478377 merged by Andrew Bogott:
[operations/puppet@production] wmcs: Add a cli script for managing dynamicproxy entries

https://gerrit.wikimedia.org/r/478377