Probably caused by https://gerrit.wikimedia.org/r/#/c/196961/ ?

We have two problems here: First of all (as noted above) it's not possible to open a tcp connection to silver on that port (mysql -h silver is failing at connect(3, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr("208.80.154.136")}, 16) = -1 ETIMEDOUT (Connection timed out)).

On top of that, the wikiadmin/wikiuser logins wont work from anything but silver itself:

mysql:root@localhost [(none)]> SELECT Host, User FROM mysql.user WHERE User = 'wikiadmin' OR User = 'wikiuser';
+----------------+-----------+
| Host           | User      |
+----------------+-----------+
| 208.80.154.136 | wikiadmin |
| 208.80.154.136 | wikiuser  |
+----------------+-----------+
2 rows in set (0.01 sec)

Maybe dumps of labswiki are failing, too. Can you confirm it @ArielGlenn?

I just saw :real_connect(): (HY000/2003): Can't connect to MySQL server on '208.80.154.136' (4) (labswiki from snapshot1003) on the error logs.

This also prevents us from being able to add silver to the list of DB hosts which get QueryPages like Special:Wantedpages automatically updated

The title says "from tin, terbium, etc". Could you specify the "etc" part of that?

So we may want it to be accessible from certain other DB hosts (T89548), maybe snapshot hosts too (for dumps)?
I think this would technically grant the restricted group access to the DB. I don't know if you guys consider that an issue or not.

Change 232529 had a related patch set uploaded (by Dzahn):
silver/wikitech: allow mysql connection from tin

https://gerrit.wikimedia.org/r/232529

There are 2 steps here: a) allow connection per firewall rules b) grant permission in mysql itself

a) needs an "srange". this would ideally be a list of all servers that need to be able to connect or it can just be $INTERNAL which allows all of 10.0.0.0/8 Here's an example to allow just tin: https://gerrit.wikimedia.org/r/#/c/232529/1/manifests/role/mariadb.pp

b) i'd leave that to the DBA to decide

b) Depends on knowing a), as users in MySQL have an srange, too. I suppose you want the standard production users.

Change 232529 merged by Dzahn:
silver/wikitech: allow mysql connection from tin

https://gerrit.wikimedia.org/r/232529

merged the change above. watched on silver. can now do this from tin:

[tin:~] $ mysql -u wikiadmin -h silver.wikimedia.org labswiki
ERROR 2003 (HY000): Can't connect to MySQL server on 'silver.wikimedia.org' (110)

normal, because of the GRANTs, just to show the firewall is open for it now.

ACCEPT     tcp  --  db1011.eqiad.wmnet   anywhere             tcp dpt:mysql
ACCEPT     tcp  --  tin.eqiad.wmnet      anywhere             tcp dpt:mysql

@jcrespo let's do 'wikiadmin'@'tin.eqiad.wmnet' and 'wikiuser'@'tin.eqiad.wmnet' for now?

I'm not sure wikiuser is needed

me neither. i said it only because T98682#1285179 shows there is an existing grant for it to connect from silver itself.

let's do the minimal approach then and only allow wikiadmin and see

I will add the user on puppet. Just for the record- on our configuration, users with hosts using dns entries are ignored, ips have to be used.

Change 232750 had a related patch set uploaded (by Jcrespo):
Add extra grants for labswiki from tin

https://gerrit.wikimedia.org/r/232750

There are several databases on silver. I hope I am granting to the right one...

Yep, "labswiki" is the exact name of the correct database. Nothing else is necessary.

Change 232750 merged by Jcrespo:
Add extra grants for labswiki from tin

https://gerrit.wikimedia.org/r/232750

Change 232763 had a related patch set uploaded (by Dzahn):
silver/wikitech: also allow mysql from terbium

https://gerrit.wikimedia.org/r/232763

Change 232763 merged by Dzahn:
silver/wikitech: also allow mysql from terbium

https://gerrit.wikimedia.org/r/232763

also opened firewall to allow connections from terbium, in addition to tin