Page MenuHomePhabricator

maintenance/mysql.php does not support ssl
Open, HighPublic

Description

This means that, e.g., someone executing sql --wiki=enwiki --write on mwmaint2002 will send the credentials in the clear over the WAN to eqiad.

A CR to enforce ssl (672728) on the mysql client won't work as mysql.php spawns mysql with the IP of the destination host, instead of the FQDN, which causes ssl to fail.

Event Timeline

You can test this by adding this to ~/.my.cnf in your mwmaint* account:

[client]
ssl-ca=/etc/ssl/certs/Puppet_Internal_CA.pem
ssl-verify-server-cert
kormat@mwmaint1002:~(0:0)$ sql --wiki=enwiki
mysql --defaults-extra-file=/tmp/mw-mysql70e1f6214dea.ini --user=wikiadmin --database=enwiki --host=10.64.16.186
ERROR 2026 (HY000): SSL connection error: SSL certificate validation failure

(That's failing to validate the cert for db1164.eqiad.wmnet)

This example should talk to codfw servers rather than to eqiad. Anyway, we need to make mysql.pho better.

Right it runs sth like mysql --defaults-extra-file=/tmp/mw-mysql1a9ba1df9b68.ini --user=wikiadmin --database=cswiki --host=10.64.0.99. @Kormat What would be the right command? Would just using host name work?

@Urbanecm: If the command uses the FQDN of the host, then it'll work fine. So in your example, that would be --host=db1129.eqiad.wmnet.

@Urbanecm: If the command uses the FQDN of the host, then it'll work fine. So in your example, that would be --host=db1129.eqiad.wmnet.

Would only db1129 work as well?

@Urbanecm: If the command uses the FQDN of the host, then it'll work fine. So in your example, that would be --host=db1129.eqiad.wmnet.

Would only db1129 work as well?

No, needs to be the fqdn, as that's what the cert is issued for.

@Urbanecm: If the command uses the FQDN of the host, then it'll work fine. So in your example, that would be --host=db1129.eqiad.wmnet.

Would only db1129 work as well?

No, needs to be the fqdn, as that's what the cert is issued for.

Thanks, makes sense.

Urbanecm added a project: User-Urbanecm.

Change 672763 had a related patch set uploaded (by Urbanecm; owner: Urbanecm):
[mediawiki/core@master] mysql.php: Use hostname rather than IP addresses

https://gerrit.wikimedia.org/r/672763

I looked into what we need to do. It seems using the hostname is really easy, however, mediawiki uses whatever etcd provides (a live mirror seems to be on https://noc.wikimedia.org/dbconfig/eqiad.json), which is not FQDN.

I think we should change it, or issue certs to have both names, as both are unique in our infrastructure.

Using hostnames in mediawiki-config is not really an option.

I looked into what we need to do. It seems using the hostname is really easy, however, mediawiki uses whatever etcd provides (a live mirror seems to be on https://noc.wikimedia.org/dbconfig/eqiad.json), which is not FQDN.

I think we should change it, or issue certs to have both names, as both are unique in our infrastructure.

Why not have the script do a PTR lookup so that it can use the hostname, based on a config variable instead? That's by far the easiest thing to do at the moment.

Because most third party wikis probably won't have PTR for their DB server(s). The script is not Wikimedia specific, it is in MediaWiki core, and it should be as generic as possible.

Using hostnames in mediawiki-config is not really an option.

Why not, please? There already are hostnames, they are just not FQDN, see https://noc.wikimedia.org/dbconfig/eqiad.json.

Is this actually specific to sql and mwscript mysql.php? It seems like it might be an early sign that this is really just an issue for MediaWiki more broadly.

If I understand correctly, neither MW in general, nor mwscript mysql.php are using a .my.cnf file for credentials or destinations. Rather, the maintenane script gets the IP address and password credentials from MediaWiki's main database configuration.

Under what circumstances do MW processes currently make cross-datacenter mysql connections? I thought we prevented this by having a split db config. incl a read-only local master. At https://noc.wikimedia.org/db.php?dc=codfw I don't see any eqiad entries.

krinkle at mwmaint2002.codfw.wmnet in ~
$ sql dewiki --write
…
wikiadmin@10.192.16.12(dewiki)> exit;
Bye
$ host 10.192.16.12
12.16.192.10.in-addr.arpa domain name pointer db2123.codfw.wmnet.

After T134809 is addressed, we will likely allow MW to know about the primary DC master, but not before that I think. At that would, we'd presumably have mysql.php use the same kind of write connections as the ones we'd use for MW more generally. No need for them to be different, right?

Is this actually specific to sql and mwscript mysql.php? It seems like it might be an early sign that this is really just an issue for MediaWiki more broadly.

Not sure, might be, or maybe also not.

If I understand correctly, neither MW in general, nor mwscript mysql.php are using a .my.cnf file for credentials or destinations. Rather, the maintenane script gets the IP address and password credentials from MediaWiki's main database configuration.

I think this is wrong. mysql.php runs proc_open, executing native system mysql. Hence, it does anything the standard mysql binary does, incl. reading standard mysql config files (probably also .my.cnf, through I'm not sure whether it would look for them in www-data's home, or running user's home). See https://github.com/wikimedia/mediawiki/blob/master/maintenance/mysql.php#L109 for the responsible code.

In what circumstances do MW processes currently make cross-datacenter mysql connections? I thought we prevented this by having a split db config (incl a read-only local master). After T134809 is addressed, we will likely allow MW to know about the primary DC master, but not before that I think.

Change 672763 abandoned by Urbanecm:

[mediawiki/core@master] mysql.php: Prefer to connect via hostname than via IP addresses

Reason:

https://gerrit.wikimedia.org/r/672763