Page MenuHomePhabricator
Create Task
Maniphest T351410

[toolsdb] Can't authenticate with Toolsdb
Closed, ResolvedPublicBUG REPORT
Actions

Description

Steps to replicate the issue (include links if applicable):

  • ssh to toolforge
  • become sample-ruby-rails-buildpack-app
  • try logging in to Toolsdb with mariadb --defaults-file=$HOME/replica.my.cnf -h tools.db.svc.wikimedia.cloud or sql tools

What happens?:
ERROR 1045 (28000): Access denied for user 's55624'@'172.16.6.95' (using password: YES)

What should have happened instead?:
I should have got a mariadb prompt

Additional info
toolforge envvars list reveals there is a discrepancy between the password contained in replica.my.cnf and the one configured in the k8s environment. However, neither password works.

My other tools works fine.

Event Timeline

Slst2020 created this task.Nov 16 2023, 2:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2023, 2:14 PM
JJMC89 edited projects, added Data-Services; removed Toolforge.Nov 16 2023, 2:55 PM
JJMC89 moved this task from Backlog to ToolsDB on the Data-Services board.Nov 28 2023, 5:54 PM
taavi closed this task as Resolved.Jan 20 2024, 4:49 PM
taavi claimed this task.
taavi subscribed.
taavi@cloudcontrol1005 ~ $ sudo /usr/local/sbin/maintain-dbusers delete tools.sample-ruby-rails-buildpack-app --account-type=tool --debug 
INFO [root.delete_account:1084] Deleted tool account in 185.15.56.15:3306 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1013.eqiad.wmnet:3311 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1017.eqiad.wmnet:3311 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1021.eqiad.wmnet:3311 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1014.eqiad.wmnet:3312 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1021.eqiad.wmnet:3312 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1018.eqiad.wmnet:3312 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1013.eqiad.wmnet:3313 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1017.eqiad.wmnet:3313 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1021.eqiad.wmnet:3313 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1015.eqiad.wmnet:3314 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1021.eqiad.wmnet:3314 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1019.eqiad.wmnet:3314 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1016.eqiad.wmnet:3315 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1020.eqiad.wmnet:3315 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1021.eqiad.wmnet:3315 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1015.eqiad.wmnet:3316 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1021.eqiad.wmnet:3316 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1019.eqiad.wmnet:3316 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1014.eqiad.wmnet:3317 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1021.eqiad.wmnet:3317 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1018.eqiad.wmnet:3317 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1016.eqiad.wmnet:3318 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1020.eqiad.wmnet:3318 for tools.sample-ruby-rails-buildpack-app
INFO [root.delete_account:1084] Deleted tool account in clouddb1021.eqiad.wmnet:3318 for tools.sample-ruby-rails-buildpack-app
DEBUG [urllib3.connectionpool._new_conn:1006] Starting new HTTPS connection (1): nfs-tools.wmcloud.org:443
DEBUG [urllib3.connectionpool._make_request:457] https://nfs-tools.wmcloud.org:443 "POST /v1/delete-replica-cnf HTTP/1.1" 400 167
ERROR [root.inner:160] Request to delete replica.my.cnf file for for account_type tool and account_id tools.sample-ruby-rails-buildpack-app failed. Reason: 400 Client Error: BAD REQUEST for url: https://nfs-tools.wmcloud.org/v1/delete-replica-cnf

Made https://gerrit.wikimedia.org/r/c/operations/puppet/+/991924/ for the account deletion failure.

root@tools-nfs-2:/srv/tools/project/sample-ruby-rails-buildpack-app# chattr -i replica.my.cnf 
root@tools-nfs-2:/srv/tools/project/sample-ruby-rails-buildpack-app# mv replica.my.cnf replica.my.cnf.bak
Jan 20 16:40:32 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._populate_new_account:697] Wrote replica.my.cnf for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in 185.15.56.15:3306 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1013.eqiad.wmnet:3311 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1017.eqiad.wmnet:3311 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1021.eqiad.wmnet:3311 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1014.eqiad.wmnet:3312 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1021.eqiad.wmnet:3312 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1018.eqiad.wmnet:3312 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1013.eqiad.wmnet:3313 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1017.eqiad.wmnet:3313 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1021.eqiad.wmnet:3313 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1015.eqiad.wmnet:3314 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1021.eqiad.wmnet:3314 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1019.eqiad.wmnet:3314 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1016.eqiad.wmnet:3315 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1020.eqiad.wmnet:3315 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1021.eqiad.wmnet:3315 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1015.eqiad.wmnet:3316 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1021.eqiad.wmnet:3316 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1019.eqiad.wmnet:3316 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1014.eqiad.wmnet:3317 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1021.eqiad.wmnet:3317 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1018.eqiad.wmnet:3317 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1016.eqiad.wmnet:3318 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:38 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1020.eqiad.wmnet:3318 for tool tools.sample-ruby-rails-buildpack-app
Jan 20 16:40:39 cloudcontrol1005 maintain-dbusers[3746023]: INFO [root._create_accounts_on_host:953] Created account in clouddb1021.eqiad.wmnet:3318 for tool tools.sample-ruby-rails-buildpack-app
tools.sample-ruby-rails-buildpack-app@tools-sgebastion-11:~$ sql tools
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 215733189
Server version: 10.4.29-MariaDB-log MariaDB Server
Slst2020 added a comment.Jan 22 2024, 5:58 AM

Thank you @taavi

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL