I created the wdqsbe tool yesterday. I can become the tool, and the replica file is there. However, I can't connect to toolsdb:
tools.wdqsbe@tools-sgebastion-10:~$ sql local ERROR 1045 (28000): Access denied for user 's55360'@'172.16.6.95' (using password: YES)
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| maintain_dbusers: don't skip the whole clouddb if one user fails | operations/puppet | production | +30 -20 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T272395 Cloud: reduce NAT exceptions from cloud to production | |||
| Resolved | Andrew | T291405 [NFS] Reduce or eliminate bare-metal NFS servers | |||
| Resolved | Andrew | T301280 Move project-specific NFS mounts onto project-local NFS servers | |||
| Resolved | dcaro | T303663 Split maintain-dbusers.py into two parts, one to run on cloudcontrol nodes and one to run on an NFS server VM | |||
| Resolved | BUG REPORT | dcaro | T332762 New tool not allowed to connect to toolsdb | ||
| Invalid | BUG REPORT | dcaro | T332789 [maintain-dbusers] when filtering by a tool we use inconsistent filters | ||
| Resolved | BUG REPORT | dcaro | T332798 [maintain-dbusers] When creating accounts, the script bails out processing other accounts if one of them fails in an unexpected way | ||
| Resolved | dcaro | T332955 [maintain-dbusers] Generate prometheus metrics |
Mar 22 09:24:15 cloudcontrol1005 /usr/local/sbin/maintain-dbusers[1248476]: WARNING [root.create_accounts:940] Could not connect to 185.15.56.15 due to (1396, "Operation CREATE USER failed for 's55337'@'%'"). Skipping.
Looking at the grant difference for the changes in T303663: Split maintain-dbusers.py into two parts, one to run on cloudcontrol nodes and one to run on an NFS server VM:
-- Grants for 'labsdbadmin'@'208.80.154.85' GRANT CREATE USER ON *.* TO 'labsdbadmin'@'208.80.154.85' IDENTIFIED BY PASSWORD '[redacted]' WITH GRANT OPTION; GRANT ALL PRIVILEGES ON `%\_\_%`.* TO 'labsdbadmin'@'208.80.154.85' WITH GRANT OPTION; GRANT SELECT ON `mysql`.* TO 'labsdbadmin'@'208.80.154.85'; GRANT SELECT, SHOW VIEW ON `%\_p`.* TO 'labsdbadmin'@'208.80.154.85' WITH GRANT OPTION; -- Grants for 'labsdbadmin'@'10.64.37.19' GRANT CREATE USER ON *.* TO 'labsdbadmin'@'10.64.37.19' IDENTIFIED BY PASSWORD '[redacted]'; GRANT ALL PRIVILEGES ON `%\_\_%`.* TO 'labsdbadmin'@'10.64.37.19' WITH GRANT OPTION; GRANT SELECT ON `mysql`.* TO 'labsdbadmin'@'10.64.37.19'; GRANT SELECT, SHOW VIEW ON `%\_p`.* TO 'labsdbadmin'@'10.64.37.19' WITH GRANT OPTION;
No obvious causes there - there's an extra WITH GRANT OPTION on the new(!) grants but otherwise those look identical to me.
I did this:
root@cloudcontrol1005:~# /usr/bin/python3 /usr/local/sbin/maintain-dbusers delete tools.wdqsbebut it's still failing :(
Mar 22 10:52:31 cloudcontrol1005 /usr/local/sbin/maintain-dbusers[1693288]: WARNING [root.create_accounts:940] Could not connect to 185.15.56.15 due to (1396, "Operation CREATE USER failed for 's55337'@'%'"). Skipping. Mar 22 10:53:31 cloudcontrol1005 /usr/local/sbin/maintain-dbusers[1693288]: INFO [root._populate_new_account:710] Wrote replica.my.cnf for tool tools.wdqsbe
yep, that error is a specific test tool from andrew iirc, I remember seeing it and trying to fix it when we did the migration, but being only one test tool failing I decided it was not urgent (that's why the WITH GRANTS is there).
The user does not exist:
mysql:labsdbadmin@185.15.56.15 [mysql]> select User, Host from user where User like 's55360%'; Empty set (0.009 sec)
And there's no databases there for that user either :/, looking
I think I found the issue, when the maintain_dbusers gets to the point of trying to create the account that fails, it stops processing the rest of users, and it turns out that the new account is listed after the failing one, so it never gets to be created. I'll open a subtask to work on it.
For now I have manually created the account, you should be able to log in now:
tools.wdqsbe@tools-sgebastion-10:~$ sql local Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 71361172 Server version: 10.1.44-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> Bye
Thanks for the report! (and patience)
Thanks, better, but not quite there yet:
tools.wdqsbe@tools-sgebastion-10:~$ sql local Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 73249977 Server version: 10.1.44-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> create database s55360_wdqsbe_p; ERROR 1044 (42000): Access denied for user 's55360'@'%' to database 's55360_wdqsbe_p'
Now there is the same problem on my new tool: iabot4nswiki.
ERROR 1045 (28000): Access denied for user 's55375'@'172.16.6.70' (using password: YES) tools.iabot4nswiki@tools-sgebastion-11:~$ sql tools ERROR 1045 (28000): Access denied for user 's55375'@'172.16.6.70' (using password: YES) tools.iabot4nswiki@tools-sgebastion-11:~$ vim replica.my.cnf tools.iabot4nswiki@tools-sgebastion-11:~$ sql local ERROR 1045 (28000): Access denied for user 's55375'@'172.16.6.70' (using password: YES)
The last read-only issue appeared to be caused by an ongoing maintenance, but it has now been reverted. Unfortunately, the database is still inaccessible, while one of my previous tools accounts and my own account can be accessed normally, suspected to encounter the same problem.
Perhaps this should be fixed soon?
This was the same issue yes, the patches to fix it are not yet in, so I fixed manually for now (I regenerated your credentials):
tools.iabot4nswiki@tools-sgebastion-10:~$ sql local Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 127319119 Server version: 10.1.44-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> Bye
Let me know if you find more issues!
I'm experiencing the same issue with my new tools, toolhunt-staging and toolhunt-api-staging:
tools.toolhunt-staging@tools-sgebastion-10:~$ sql tools ERROR 1045 (28000): Access denied for user 's55383'@'172.16.6.95' (using password: YES)
tools.toolhunt-api-staging@tools-sgebastion-10:~$ sql tools ERROR 1045 (28000): Access denied for user 's55384'@'172.16.6.95' (using password: YES)
Change 905630 had a related patch set uploaded (by David Caro; author: David Caro):
[operations/puppet@production] maintain_dbusers: don't skip the whole clouddb if one user fails
Change 905630 merged by David Caro:
[operations/puppet@production] maintain_dbusers: don't skip the whole clouddb if one user fails
I'll close this for now as it should not happen again, but please open a new task if you have a new issue, cheers!