Page MenuHomePhabricator

Access denied to Toolforge DB since last Friday
Closed, ResolvedPublic

Description

Hi,

Since last Friday I can no longer access the Toolforge database using my username and password (which worked before). I get a message saying "User access denied using password:YES". I tested the access from DataGrip and from my Python code and in both cases I get access denied.

I have seen a related task https://phabricator.wikimedia.org/T183644 and followed the instructions at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Debugging_bad_mysql_credentials but when I test that I get a message saying "ERROR 2003 (HY000): Can't connect to MySQL server on 'm5-master.eqiad.wmnet' (110 "Connection timed out")".

What can I do to fix this and have access to the DB again? Thanks a lot in advance.

Event Timeline

Same as T235382. @bd808 can you help and with this as for previous task?

bd808 removed a project: Wikidata.
bd808 moved this task from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.

@Criscod can you clarify which replica.my.cnf credentials need repair? Giving the full path to the file here would be one way to do that.

@Criscod can you clarify which replica.my.cnf credentials need repair? Giving the full path to the file here would be one way to do that.

This is the full path: /mnt/nfs/labstore-secondary-tools-home/cristinasarasua/replica.my.cnf

bd808 assigned this task to JHedden.
bd808 added a subscriber: JHedden.
$ ldap uid=cristinasarasua uidNumber
dn: uid=cristinasarasua,ou=people,dc=wikimedia,dc=org
uidNumber: 20891
$ mysql -h m5-master.eqiad.wmnet -u labsdbaccounts -p
(labsdbaccounts@m5-master.eqiad.wmnet) [(none)]> use labsdbaccounts;
(labsdbaccounts@m5-master.eqiad.wmnet) [labsdbaccounts]> select a.id, a.mysql_username, a.username, ah.hostname, ah.status from account a left join account_host ah on a.id = ah.account_id where a.mysql_username='u20891'\G
*************************** 1. row ***************************
            id: 19164
mysql_username: u20891
      username: cristinasarasua
      hostname: 172.16.7.153
        status: present
*************************** 2. row ***************************
            id: 19164
mysql_username: u20891
      username: cristinasarasua
      hostname: labsdb1009.eqiad.wmnet
        status: present
*************************** 3. row ***************************
            id: 19164
mysql_username: u20891
      username: cristinasarasua
      hostname: labsdb1010.eqiad.wmnet
        status: present
*************************** 4. row ***************************
            id: 19164
mysql_username: u20891
      username: cristinasarasua
      hostname: labsdb1011.eqiad.wmnet
        status: present
4 rows in set (0.00 sec)
$ sudo journalctl -u maintain-dbusers --no-pager | grep cristinasarasua
Oct 16 21:52:17 labstore1004 /usr/local/sbin/maintain-dbusers[32862]: Created account in labsdb1011.eqiad.wmnet for user cristinasarasua

Fixed by @JHedden as a side effect of his work on T235697: labsdb1011 missing tools user accounts that looks to have fixed this issue for multiple accounts that have gone missing over time due to database reimports.

Thanks a lot - it works again!