Page MenuHomePhabricator

Several users and tools have invalid credentials in replica.my.cnf
Closed, ResolvedPublic

Description

(At least) the user danmichaelo (u2238) and the tools botomatik (s52307), boty (s51984), enbbsb (s51482), hasteurbot (s51307), wmve (s52526), and xxn-tools (s52403) have invalid credentials in their respective replica.my.cnfs; for example:

danmichaelo@tools-bastion-03:~$ mysql --defaults-file=replica.my.cnf -henwiki.labsdb enwiki_p
ERROR 1045 (28000): Access denied for user 'u2238'@'10.68.23.58' (using password: YES)
danmichaelo@tools-bastion-03:~$
(ENV)tools.danmicholobot@tools-bastion-03:~$ mysql --defaults-file=replica.my.cnf -henwiki.labsdb enwiki_p
ERROR 1045 (28000): Access denied for user 's51488'@'10.68.23.58' (using password: YES)
(ENV)tools.danmicholobot@tools-bastion-03:~$

All the tool accounts had the value for user set to ('s12345',). After changing that to s12345, the error persists.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
scfc updated the task description. (Show Details)

(user was ('s51488',) in /data/project/danmicholobot/replica.my.cnf; I fixed that to s51488, but it does not change the access problem.)

The grants are there...

[root@labsdb1001 03:48 /root]
# pt-show-grants  | grep s51488
-- Grants for 's51488'@'%'
<snip>
GRANT SELECT, SHOW VIEW ON `%\_p`.* TO 's51488'@'%';
scfc renamed this task from Credentials in replica.my.cnf gives Access Denied to Several users and tools have invalid credentials in replica.my.cnf.Jan 11 2017, 5:35 PM
scfc updated the task description. (Show Details)
scfc updated the task description. (Show Details)

Some more and command to list them:

scfc@tools-bastion-03:~$ sudo find -H /home /data/project -mindepth 2 -maxdepth 2 -type f -name replica.my.cnf -not -exec mysql --defaults-file={} -h enwiki.labsdb enwiki_p \; -print < /dev/null 
ERROR 1045 (28000): Access denied for user 'u3180'@'10.68.23.58' (using password: YES)
/home/inkowik/replica.my.cnf
ERROR 1045 (28000): Access denied for user 'u4507'@'10.68.23.58' (using password: YES)
/home/mnemonicflow/replica.my.cnf
ERROR 1045 (28000): Access denied for user 'u5024'@'10.68.23.58' (using password: YES)
/home/shbibalsubaie13/replica.my.cnf
ERROR 1045 (28000): Access denied for user 'u12396'@'10.68.23.58' (using password: NO)
/home/db111/replica.my.cnf
ERROR 1045 (28000): Access denied for user 'u10457'@'10.68.23.58' (using password: YES)
/home/botomatik/replica.my.cnf
ERROR 1045 (28000): Access denied for user 'u4720'@'10.68.23.58' (using password: YES)
/home/xxn/replica.my.cnf
ERROR 1045 (28000): Access denied for user 'u3128'@'10.68.23.58' (using password: YES)
/home/whitemaster/replica.my.cnf
ERROR 1045 (28000): Access denied for user 'u12400'@'10.68.23.58' (using password: NO)
/home/elee/replica.my.cnf
ERROR 1045 (28000): Access denied for user 'u3533'@'10.68.23.58' (using password: YES)
/home/hasteur/replica.my.cnf
ERROR 1045 (28000): Access denied for user 'u2238'@'10.68.23.58' (using password: YES)
/home/danmichaelo/replica.my.cnf
ERROR 1226 (42000): User 's51053' has exceeded the 'max_user_connections' resource (current value: 1)
/data/project/jackbot/replica.my.cnf
ERROR 1045 (28000): Access denied for user 's51488'@'10.68.23.58' (using password: YES)
/data/project/danmicholobot/replica.my.cnf
ERROR 1045 (28000): Access denied for user 's51307'@'10.68.23.58' (using password: YES)
/data/project/hasteurbot/replica.my.cnf
ERROR 1045 (28000): Access denied for user 's52477'@'10.68.23.58' (using password: YES)
/data/project/reportsbot/replica.my.cnf
ERROR 1045 (28000): Access denied for user 's51482'@'10.68.23.58' (using password: YES)
/data/project/enbbsb/replica.my.cnf
ERROR 1045 (28000): Access denied for user 's52307'@'10.68.23.58' (using password: YES)
/data/project/botomatik/replica.my.cnf
ERROR 1045 (28000): Access denied for user 's52697'@'10.68.23.58' (using password: YES)
/data/project/hgztools/replica.my.cnf
ERROR 1045 (28000): Access denied for user 's52403'@'10.68.23.58' (using password: YES)
/data/project/xxn-tools/replica.my.cnf
ERROR 1045 (28000): Access denied for user 's51481'@'10.68.23.58' (using password: YES)
/data/project/newwebtest/replica.my.cnf
ERROR 1045 (28000): Access denied for user 's52426'@'10.68.23.58' (using password: YES)
/data/project/xtools-articleinfo/replica.my.cnf
mysql: unknown variable 'test=asd'
/data/project/alex/replica.my.cnf
ERROR 1045 (28000): Access denied for user 's51984'@'10.68.23.58' (using password: YES)
/data/project/boty/replica.my.cnf
ERROR 1045 (28000): Access denied for user 's52526'@'10.68.23.58' (using password: YES)
/data/project/wmve/replica.my.cnf
scfc@tools-bastion-03:~$

Do you happen to know to which of the labsdb servers are you connecting to?

That's probably 10.64.4.11:

scfc@tools-bastion-03:~$ host enwiki.labsdb
enwiki.labsdb has address 10.64.4.11
scfc@tools-bastion-03:~$

Known broken AFAIK:

tools.jackbot
tools.danmicholobot
tools.hasteurbot
tools.reportsbot
tools.enbbsb
tools.botomatik
tools.hgztools
tools.xxn-tools
tools.newwebtest
tools.xtools-articleinfo
tools.alex
tools.boty
tools.wmve

I ran:

/usr/local/sbin/maintain-dbusers delete tools.hasteurbot --debug
/usr/local/sbin/maintain-dbusers maintain --debug

to fix hasteurbot and equiv for hgztools which now seems ok.

I am going to do the rest and see what we get. But first @marostgeui is going to pull the current grants for two users to see what end state changes happen during:

ERROR 1045 (28000): Access denied for user 's52526'@'10.68.23.58' (using password: YES)

/data/project/wmve/replica.my.cnf

ERROR 1045 (28000): Access denied for user 's51984'@'10.68.23.58' (using password: YES)

/data/project/boty/replica.my.cnf

That's probably 10.64.4.11:

scfc@tools-bastion-03:~$ host enwiki.labsdb
enwiki.labsdb has address 10.64.4.11
scfc@tools-bastion-03:~$

Thanks - that is labsdb1001, the host I was checking.
I have been talking to @chasemp and we are going to dig into something he's seen - we will keep you posted.

root@labstore1004:~# nano /tmp/bad-tool-labsdb
root@labstore1004:~# for tool in `cat /tmp/bad-tool-labsdb`; do /usr/local/sbin/maintain-dbusers delete $tool --debug; done
Deleted account in labsdb1001.eqiad.wmnet for tools.jackbot
Deleted account in labsdb1003.eqiad.wmnet for tools.jackbot
Deleted account in labsdb1010.eqiad.wmnet for tools.jackbot
Deleted account in labsdb1011.eqiad.wmnet for tools.jackbot
Deleted account in labsdb1005.eqiad.wmnet for tools.jackbot
Deleted account in labsdb1009.eqiad.wmnet for tools.jackbot
Deleted /srv/tools/shared/tools/project/jackbot/replica.my.cnf
Deleted account in labsdb1010.eqiad.wmnet for tools.danmicholobot
Deleted account in labsdb1003.eqiad.wmnet for tools.danmicholobot
Deleted account in labsdb1009.eqiad.wmnet for tools.danmicholobot
Deleted account in labsdb1001.eqiad.wmnet for tools.danmicholobot
Deleted account in labsdb1005.eqiad.wmnet for tools.danmicholobot
Deleted account in labsdb1011.eqiad.wmnet for tools.danmicholobot
Deleted /srv/tools/shared/tools/project/danmicholobot/replica.my.cnf
Deleted account in labsdb1005.eqiad.wmnet for tools.hasteurbot
Deleted account in labsdb1003.eqiad.wmnet for tools.hasteurbot
Deleted account in labsdb1001.eqiad.wmnet for tools.hasteurbot
Deleted account in labsdb1009.eqiad.wmnet for tools.hasteurbot
Deleted account in labsdb1011.eqiad.wmnet for tools.hasteurbot
Deleted account in labsdb1010.eqiad.wmnet for tools.hasteurbot
Deleted /srv/tools/shared/tools/project/hasteurbot/replica.my.cnf
Deleted account in labsdb1003.eqiad.wmnet for tools.reportsbot
Deleted account in labsdb1009.eqiad.wmnet for tools.reportsbot
Deleted account in labsdb1011.eqiad.wmnet for tools.reportsbot
Deleted account in labsdb1001.eqiad.wmnet for tools.reportsbot
Deleted account in labsdb1005.eqiad.wmnet for tools.reportsbot
Deleted account in labsdb1010.eqiad.wmnet for tools.reportsbot
Deleted /srv/tools/shared/tools/project/reportsbot/replica.my.cnf
Deleted account in labsdb1010.eqiad.wmnet for tools.enbbsb
Deleted account in labsdb1001.eqiad.wmnet for tools.enbbsb
Deleted account in labsdb1005.eqiad.wmnet for tools.enbbsb
Deleted account in labsdb1003.eqiad.wmnet for tools.enbbsb
Deleted account in labsdb1011.eqiad.wmnet for tools.enbbsb
Deleted account in labsdb1009.eqiad.wmnet for tools.enbbsb
Deleted /srv/tools/shared/tools/project/enbbsb/replica.my.cnf
Deleted account in labsdb1005.eqiad.wmnet for tools.botomatik
Deleted account in labsdb1010.eqiad.wmnet for tools.botomatik
Deleted account in labsdb1001.eqiad.wmnet for tools.botomatik
Deleted account in labsdb1003.eqiad.wmnet for tools.botomatik
Deleted account in labsdb1009.eqiad.wmnet for tools.botomatik
Deleted account in labsdb1011.eqiad.wmnet for tools.botomatik
Deleted /srv/tools/shared/tools/project/botomatik/replica.my.cnf
Deleted account in labsdb1011.eqiad.wmnet for tools.hgztools
Deleted account in labsdb1005.eqiad.wmnet for tools.hgztools
Deleted account in labsdb1009.eqiad.wmnet for tools.hgztools
Deleted account in labsdb1010.eqiad.wmnet for tools.hgztools
Deleted account in labsdb1003.eqiad.wmnet for tools.hgztools
Deleted account in labsdb1001.eqiad.wmnet for tools.hgztools
Deleted /srv/tools/shared/tools/project/hgztools/replica.my.cnf
Deleted account in labsdb1001.eqiad.wmnet for tools.xxn-tools
Deleted account in labsdb1005.eqiad.wmnet for tools.xxn-tools
Deleted account in labsdb1010.eqiad.wmnet for tools.xxn-tools
Deleted account in labsdb1003.eqiad.wmnet for tools.xxn-tools
Deleted account in labsdb1011.eqiad.wmnet for tools.xxn-tools
Deleted account in labsdb1009.eqiad.wmnet for tools.xxn-tools
Deleted /srv/tools/shared/tools/project/xxn-tools/replica.my.cnf
Deleted account in labsdb1010.eqiad.wmnet for tools.newwebtest
Deleted account in labsdb1005.eqiad.wmnet for tools.newwebtest
Deleted account in labsdb1001.eqiad.wmnet for tools.newwebtest
Deleted account in labsdb1009.eqiad.wmnet for tools.newwebtest
Deleted account in labsdb1003.eqiad.wmnet for tools.newwebtest
Deleted account in labsdb1011.eqiad.wmnet for tools.newwebtest
Deleted /srv/tools/shared/tools/project/newwebtest/replica.my.cnf
Deleted account in labsdb1005.eqiad.wmnet for tools.xtools-articleinfo
Deleted account in labsdb1010.eqiad.wmnet for tools.xtools-articleinfo
Deleted account in labsdb1009.eqiad.wmnet for tools.xtools-articleinfo
Deleted account in labsdb1001.eqiad.wmnet for tools.xtools-articleinfo
Deleted account in labsdb1003.eqiad.wmnet for tools.xtools-articleinfo
Deleted account in labsdb1011.eqiad.wmnet for tools.xtools-articleinfo
Deleted /srv/tools/shared/tools/project/xtools-articleinfo/replica.my.cnf
Deleted account in labsdb1001.eqiad.wmnet for tools.alex
Deleted account in labsdb1009.eqiad.wmnet for tools.alex
Deleted account in labsdb1011.eqiad.wmnet for tools.alex
Deleted account in labsdb1003.eqiad.wmnet for tools.alex
Deleted account in labsdb1005.eqiad.wmnet for tools.alex
Deleted account in labsdb1010.eqiad.wmnet for tools.alex
Deleted /srv/tools/shared/tools/project/alex/replica.my.cnf
Deleted account in labsdb1005.eqiad.wmnet for tools.boty
Deleted account in labsdb1009.eqiad.wmnet for tools.boty
Deleted account in labsdb1001.eqiad.wmnet for tools.boty
Deleted account in labsdb1003.eqiad.wmnet for tools.boty
Deleted account in labsdb1011.eqiad.wmnet for tools.boty
Deleted account in labsdb1010.eqiad.wmnet for tools.boty
Deleted /srv/tools/shared/tools/project/boty/replica.my.cnf
Deleted account in labsdb1011.eqiad.wmnet for tools.wmve
Deleted account in labsdb1003.eqiad.wmnet for tools.wmve
Deleted account in labsdb1001.eqiad.wmnet for tools.wmve
Deleted account in labsdb1010.eqiad.wmnet for tools.wmve
Deleted account in labsdb1009.eqiad.wmnet for tools.wmve
Deleted account in labsdb1005.eqiad.wmnet for tools.wmve
Deleted /srv/tools/shared/tools/project/wmve/replica.my.cnf
root@labstore1004:~# /usr/local/sbin/maintain-dbusers maintain --debug
Found 13 new tools: tools.jackbot, tools.hasteurbot, tools.newwebtest, tools.enbbsb, tools.danmicholobot, tools.boty, tools.alex, tools.botomatik, tools.xxn-tools, tools.xtools-articleinfo, tools.reportsbot, tools.wmve, tools.hgztools
Wrote replica.my.cnf for tools.jackbot
Wrote replica.my.cnf for tools.hasteurbot
Wrote replica.my.cnf for tools.newwebtest
Wrote replica.my.cnf for tools.enbbsb
Wrote replica.my.cnf for tools.danmicholobot
Wrote replica.my.cnf for tools.boty
Wrote replica.my.cnf for tools.alex
Wrote replica.my.cnf for tools.botomatik
Wrote replica.my.cnf for tools.xxn-tools
Wrote replica.my.cnf for tools.xtools-articleinfo
Wrote replica.my.cnf for tools.reportsbot
Wrote replica.my.cnf for tools.wmve
Wrote replica.my.cnf for tools.hgztools
Created account in labsdb1010.eqiad.wmnet for tools.jackbot
Created account in labsdb1010.eqiad.wmnet for tools.hasteurbot
Created account in labsdb1010.eqiad.wmnet for tools.newwebtest
Created account in labsdb1010.eqiad.wmnet for tools.enbbsb
Created account in labsdb1010.eqiad.wmnet for tools.danmicholobot
Created account in labsdb1010.eqiad.wmnet for tools.boty
Created account in labsdb1010.eqiad.wmnet for tools.alex
Created account in labsdb1010.eqiad.wmnet for tools.botomatik
Created account in labsdb1010.eqiad.wmnet for tools.xxn-tools
Created account in labsdb1010.eqiad.wmnet for tools.xtools-articleinfo
Created account in labsdb1010.eqiad.wmnet for tools.reportsbot
Created account in labsdb1010.eqiad.wmnet for tools.wmve
Created account in labsdb1010.eqiad.wmnet for tools.hgztools
Created account in labsdb1005.eqiad.wmnet for tools.jackbot
Created account in labsdb1005.eqiad.wmnet for tools.hasteurbot
Created account in labsdb1005.eqiad.wmnet for tools.newwebtest
Created account in labsdb1005.eqiad.wmnet for tools.enbbsb
Created account in labsdb1005.eqiad.wmnet for tools.danmicholobot
Created account in labsdb1005.eqiad.wmnet for tools.boty
Created account in labsdb1005.eqiad.wmnet for tools.alex
Created account in labsdb1005.eqiad.wmnet for tools.botomatik
Created account in labsdb1005.eqiad.wmnet for tools.xxn-tools
Created account in labsdb1005.eqiad.wmnet for tools.xtools-articleinfo
Created account in labsdb1005.eqiad.wmnet for tools.reportsbot
Created account in labsdb1005.eqiad.wmnet for tools.wmve
Created account in labsdb1005.eqiad.wmnet for tools.hgztools
Created account in labsdb1009.eqiad.wmnet for tools.jackbot
Created account in labsdb1009.eqiad.wmnet for tools.hasteurbot
Created account in labsdb1009.eqiad.wmnet for tools.newwebtest
Created account in labsdb1009.eqiad.wmnet for tools.enbbsb
Created account in labsdb1009.eqiad.wmnet for tools.danmicholobot
Created account in labsdb1009.eqiad.wmnet for tools.boty
Created account in labsdb1009.eqiad.wmnet for tools.alex
Created account in labsdb1009.eqiad.wmnet for tools.botomatik
Created account in labsdb1009.eqiad.wmnet for tools.xxn-tools
Created account in labsdb1009.eqiad.wmnet for tools.xtools-articleinfo
Created account in labsdb1009.eqiad.wmnet for tools.reportsbot
Created account in labsdb1009.eqiad.wmnet for tools.wmve
Created account in labsdb1009.eqiad.wmnet for tools.hgztools
Created account in labsdb1001.eqiad.wmnet for tools.jackbot
Created account in labsdb1001.eqiad.wmnet for tools.hasteurbot
Created account in labsdb1001.eqiad.wmnet for tools.newwebtest
Created account in labsdb1001.eqiad.wmnet for tools.enbbsb
Created account in labsdb1001.eqiad.wmnet for tools.danmicholobot
Created account in labsdb1001.eqiad.wmnet for tools.boty
Created account in labsdb1001.eqiad.wmnet for tools.alex
Created account in labsdb1001.eqiad.wmnet for tools.botomatik
Created account in labsdb1001.eqiad.wmnet for tools.xxn-tools
Created account in labsdb1001.eqiad.wmnet for tools.xtools-articleinfo
Created account in labsdb1001.eqiad.wmnet for tools.reportsbot
Created account in labsdb1001.eqiad.wmnet for tools.wmve
Created account in labsdb1001.eqiad.wmnet for tools.hgztools
Created account in labsdb1003.eqiad.wmnet for tools.jackbot
Created account in labsdb1003.eqiad.wmnet for tools.hasteurbot
Created account in labsdb1003.eqiad.wmnet for tools.newwebtest
Created account in labsdb1003.eqiad.wmnet for tools.enbbsb
Created account in labsdb1003.eqiad.wmnet for tools.danmicholobot
Created account in labsdb1003.eqiad.wmnet for tools.boty
Created account in labsdb1003.eqiad.wmnet for tools.alex
Created account in labsdb1003.eqiad.wmnet for tools.botomatik
Created account in labsdb1003.eqiad.wmnet for tools.xxn-tools
Created account in labsdb1003.eqiad.wmnet for tools.xtools-articleinfo
Created account in labsdb1003.eqiad.wmnet for tools.reportsbot
Created account in labsdb1003.eqiad.wmnet for tools.wmve
Created account in labsdb1003.eqiad.wmnet for tools.hgztools
Created account in labsdb1011.eqiad.wmnet for tools.jackbot
Created account in labsdb1011.eqiad.wmnet for tools.hasteurbot
Created account in labsdb1011.eqiad.wmnet for tools.newwebtest
Created account in labsdb1011.eqiad.wmnet for tools.enbbsb
Created account in labsdb1011.eqiad.wmnet for tools.danmicholobot
Created account in labsdb1011.eqiad.wmnet for tools.boty
Created account in labsdb1011.eqiad.wmnet for tools.alex
Created account in labsdb1011.eqiad.wmnet for tools.botomatik
Created account in labsdb1011.eqiad.wmnet for tools.xxn-tools
Created account in labsdb1011.eqiad.wmnet for tools.xtools-articleinfo
Created account in labsdb1011.eqiad.wmnet for tools.reportsbot
Created account in labsdb1011.eqiad.wmnet for tools.wmve
Created account in labsdb1011.eqiad.wmnet for tools.hgztools

@scfc can you try those ones that Chase recreated and see if they work?

The credentials for the 13 tools now work:

scfc@tools-bastion-03:~$ for tool in alex botomatik boty danmicholobot enbbsb hasteurbot hgztools jackbot newwebtest reportsbot wmve xtools-articleinfo xxn-tools; do sudo become "$tool" sql enwiki 'SELECT 1 FROM page WHERE page_id = 15580374;'; done                                                                                       
1                                                                                                                                                                       
1                                                                                                                                                                       
1                                                                                                                                                                       
1
1
1
✔ Adding pywikibot to PYTHONPATH
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
scfc@tools-bastion-03:~$

But danmichaelo (u2238) is still locked out:

danmichaelo@tools-bastion-03:~$ sql enwiki
ERROR 1045 (28000): Access denied for user 'u2238'@'10.68.23.58' (using password: YES)
danmichaelo@tools-bastion-03:~$

The user danmichaelo isn't in the list that @chasemp iterated over the first time to test if that would fix the issue.
Looks like recreating that one too will likely solve the issue as it did with the other ones he just did.

I am going to remove the DBA tag from this ticket as it doesn't look related to a DB problem at the moment (if you guys feel it needs to be, feel free to add it again!). I am subscribed to this ticket though, so I will keep an eye just in case I am needed.

But danmichaelo (u2238) is still locked out:

danmichaelo@tools-bastion-03:~$ sql enwiki
ERROR 1045 (28000): Access denied for user 'u2238'@'10.68.23.58' (using password: YES)
danmichaelo@tools-bastion-03:~$

would like @yuvipanda to weigh in on the question here as I'm not sure what the status of generating per user vs per tool DB credentials actually is. I know when I attempt to delete this I see odd output:

sudo /usr/local/sbin/maintain-dbusers delete uu2238 --debug
/usr/bin/chattr: No such file or directory while trying to stat /srv/tools/shared/tools/project/replica.my.cnf
Traceback (most recent call last):
  File "/usr/local/sbin/maintain-dbusers", line 518, in <module>
    delete_account(config, args.extra_args)
  File "/usr/local/sbin/maintain-dbusers", line 414, in delete_account
    subprocess.check_output(['/usr/bin/chattr', '-i', replica_file_path])
  File "/usr/lib/python3.4/subprocess.py", line 620, in check_output
    raise CalledProcessError(retcode, process.args, output=output)
subprocess.CalledProcessError: Command '['/usr/bin/chattr', '-i', '/srv/tools/shared/tools/project/replica.my.cnf']' returned non-zero exit status 1

Which makes me think possibly we don't account for the use case.

[…]

But danmichaelo (u2238) is still locked out:

[…]

sudo /usr/local/sbin/maintain-dbusers delete uu2238 --debug
[…]

Have you tried u2238?

[…]

But danmichaelo (u2238) is still locked out:

[…]

sudo /usr/local/sbin/maintain-dbusers delete uu2238 --debug
[…]

Have you tried u2238?

heh, good question but yes I did. Same result. Mispaste of early typo there :)

I responded on the wrong ticket, whoops. We don't have support for user accounts in the new maintain-dbusers script yet.

scfc triaged this task as Low priority.Feb 16 2017, 11:12 PM
scfc moved this task from Backlog to Ready to be worked on on the Toolforge board.
madhuvishy claimed this task.
madhuvishy subscribed.

Closing this - all user accounts should have valid replica.my.cnfs now. @Danmichaelo you should be able to use your replica.my.cnf from your tools home directory (u2238).