Page MenuHomePhabricator

Lost access to x1-analytics-slave
Closed, ResolvedPublic

Description

Logging to stat1006.eqiad.wmnet s1-analytics-slave or s3 is fine. Logging to x1-analytics-slave displays 'Access denied'. I used to log in to x1 without any problems.

etonkovidova@AdministorsAir2:~/Documents/gerrit/vagrant/mediawiki$ ssh stat1006.eqiad.wmnet
Linux stat1006 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64
Debian GNU/Linux 9.1 (stretch)
stat1006 is a Statistics general compute node (non private data) (statistics::cruncher)
The last Puppet run was at Thu Sep 14 23:11:36 UTC 2017 (2 minutes ago). 
Debian GNU/Linux 9 auto-installed on Wed Jul 12 13:47:00 UTC 2017.
Last login: Thu Sep 14 18:27:36 2017 from 198.35.26.5

etonkovidova@stat1006:~$ mysql -h s1-analytics-slave 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 47557758
Server version: 10.0.22-MariaDB-log MariaDB Server

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql:research@s1-analytics-slave [(none)]> exit
Bye
etonkovidova@stat1006:~$ mysql -h x1-analytics-slave 
ERROR 1045 (28000): Access denied for user 'research'@'10.64.53.31' (using password: YES)

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a project: Operations. · View Herald TranscriptSep 14 2017, 11:34 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
jcrespo edited projects, added DBA; removed SRE-Access-Requests.Sep 15 2017, 7:49 AM
jcrespo moved this task from Triage to Done on the DBA board.
elukey added a subscriber: jcrespo.Sep 15 2017, 7:53 AM

After a chat with @jcrespo we decided to point the x1-analytics-slave domain to dbstore1002 (analytics-slave) - https://gerrit.wikimedia.org/r/#/c/378211

db1029 is a production host and it would be better not to expose it to research queries, so we have also removed the research user from there. From what we know dbstore1002 should be enough but please let us know if any replication that you need is missing.

Thanks for the report!

@elukey Thanks! I can access
I need an access to echo_ tables and I do not see them replicated there (or the acccess is not allowed?).

@Etonkovidova Can you provide more information about how you plan to use that data? It was not initially included on analytics data because it was both more difficult technologically and not needed at the time. If you need a one-time access, I can provide you access to a host temporarilly quickly. If it is an ongoing project (long term), I will import those tables into dbstore1002, which may take a few days.

@jcrespo Echo notifications is a project that Global Collaboration team is responsible for (I am a QA for that team). From time to time, following users' complaints or requests, or our own testing we need to verify that notifications were, in fact, delivered and their type, a sender, and other parameters have been recorded correctly to verify it against what was displayed in UI.

Right now, I was trying to investigate T174873: Ping system failed to notify mention on hi.wikiversity and hiwikivoyage and needed to track down in db whether my testing mentions in hiwikiversity, in fact, were recorded in db or not (and how many).

Jayprakash12345 added a comment.EditedSep 16 2017, 4:12 AM

Also Sir When I use http://tools.wmflabs.org/pageviews and type hi.wikiversity.org in project. It is showing an error. See Below.

hi.wikiversity.org is not a valid project or is currently unsupported.

@Jayprakash12345 Unless I am wrong, that is a different issue, nor related to the analytics db servers- please file a separate ticket so @Analytics ops can have a look at it (it is probably not related to mysql).

@Etonkovidova I will try to setup something quickly and then import the echo tables and other missing pieces ASAP. Please give me some days to figure it out.

@jcrespo Thank you very much!

jcrespo added a subscriber: Catrope.
jcrespo added a subscriber: Marostegui.

CC @Marostegui Maybe we can setup an unpuppetized copy of x1 from dbstore2002 on dbstore1002?

CC @Marostegui Maybe we can setup an unpuppetized copy of x1 from dbstore2002 on dbstore1002?

Yeah, dbstore1002 now has plenty of space (1.9T available) and x1 is around 250G...

Should we do it on the same instance, is a mysqloader fast enough for x1, do you think?

jcrespo claimed this task.Oct 20 2017, 6:34 AM

Should we do it on the same instance, is a mysqloader fast enough for x1, do you think?

I would do it on the same instance, otherwise we will need to run very ugly non-puppetized stuff I think.
Keep in mind that myloader/mydumper will need to be run remotely as dbstore1002 is jessie and we have seen what happened to me two days ago with the jessie version. But maybe for such a small amount of data mysqldump will be fast enough?

Change 385334 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Add all x1 tables, including ones on %wik% schemas on dbstores

https://gerrit.wikimedia.org/r/385334

I am reimporting x1 on dbstore1002 (including the echo tables), hopefully, that will work and will not break anything else :-/

Change 385334 merged by Jcrespo:
[operations/puppet@production] mariadb: Add all x1 tables, including ones on %wik% schemas on dbstores

https://gerrit.wikimedia.org/r/385334

jcrespo closed this task as Resolved.Oct 20 2017, 10:31 AM

@Etonkovidova, @Catrope sorry for the delay. We how this access to echo tables on dbstore1002 will be relatively reliable. I added grants to all x1 tables- note that, being on %wiki% databases, x1 and s* tables will be mixed.

Otherwise, this should be solved.