⚓ T193187 Prepare and check storage layer for idwikimedia

	Project	Branch	Lines +/-	Subject
	operations/puppet	production	+3 -3	wiki replicas: place grant before create in maintain-views script

		Status	Subtype	Assigned	Task
		Resolved		Reedy	T192726 Create a Wikimedia hosted wiki site for Wikimedia Indonesia
		Resolved		Bstorm	T193187 Prepare and check storage layer for idwikimedia

Urbanecm created this task.Apr 26 2018, 4:45 PM

Restricted Application added a project: User-Urbanecm. · View Herald TranscriptApr 26 2018, 4:45 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Urbanecm moved this task from Backlog to Created on the User-Urbanecm board.Apr 26 2018, 4:45 PM

Let us know when this is created so we can sanitize it

Mentioned in SAL (#wikimedia-operations) [2018-06-07T14:53:25Z] <marostegui> Sanitize wikis on db1095 (old sanitarium) - T196362 T196358 T196359 T195008 T193187

Mentioned in SAL (#wikimedia-operations) [2018-06-07T15:08:29Z] <marostegui> Sanitize wikis on db1124 (current sanitarium for s3) - T196362 T196358 T196359 T195008 T193187

This wiki has been sanitized. Ready for the views creation cc @bd808 @Bstorm. We have to double check the script for the views works correctly before going for it as a default after changing how the grants are deployed.

I'll try this one as our first test case.

Surprisingly, the script has failed on the rights to create _p the database (could not execute the CREATE DATABASE statement. I could have sworn that wasn't a problem before, but I have heard that it was encountered before. The new code won't be triggered unless the script creates the _p database. I am working on labsdb1010 first. Can we fix that on that server at least @Marostegui ?

In T193187#4264531, @Bstorm wrote:

Surprisingly, the script has failed on the rights to create _p the database (could not execute the CREATE DATABASE statement. I could have sworn that wasn't a problem before, but I have heard that it was encountered before. The new code won't be triggered unless the script creates the _p database. I am working on labsdb1010 first. Can we fix that on that server at least @Marostegui ?

Which error did you get?

I got it from executing what you were executing:

root@labsdb1010:/home/bstorm# sudo /usr/local/sbin/maintain-views --databases idwikimedia
Traceback (most recent call last):
  File "/usr/local/sbin/maintain-views", line 594, in <module>
    main()
  File "/usr/local/sbin/maintain-views", line 575, in main
    ops.execute(fullviews, customviews)
  File "/usr/local/sbin/maintain-views", line 361, in execute
    "CREATE DATABASE `{}`;".format(self.db_p)

So this is the one that gets solved with the grants creation that we discussed a few weeks ago (and I thought your script was issuing it):

GRANT SELECT, SHOW VIEW ON `idwikimedia\_p`.* to labsdbuser;

Once that is run, I was able to create the views.

mysql:root@localhost [(none)]> pager grep idwikimedia; show grants for labsdbuser;
PAGER set to 'grep idwikimedia'
| GRANT SELECT, SHOW VIEW ON `idwikimedia\_p`.* TO 'labsdbuser'            |
891 rows in set (0.00 sec)

And looks like I can see it:

MariaDB [enwiki_p]> select @@hostname;
+------------+
| @@hostname |
+------------+
| labsdb1010 |
+------------+
1 row in set (0.00 sec)

MariaDB [enwiki_p]> use idwikimedia_p
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

show tables;
Database changed
MariaDB [idwikimedia_p]> show tables;
+-------------------------+
| Tables_in_idwikimedia_p |
+-------------------------+
| abuse_filter            |
| abuse_filter_action     |
| abuse_filter_log        |
| archive                 |
| archive_userindex       |
| babel                   |
| category                |
| categorylinks           |
| change_tag              |
| comment                 |
| content                 |
| content_models          |
| externallinks           |
| filearchive             |
| filearchive_userindex   |
| geo_tags                |
| global_block_whitelist  |
| globalblocks            |
| image                   |
| imagelinks              |
| interwiki               |
| ip_changes              |
| ipblocks                |
| ipblocks_ipindex        |
| iwlinks                 |
| l10n_cache              |
| langlinks               |
| linter                  |
| logging                 |
| logging_logindex        |
| logging_userindex       |
| module_deps             |
| oldimage                |
| oldimage_userindex      |
| page                    |
| page_props              |
| page_restrictions       |
| pagelinks               |
| protected_titles        |
| recentchanges           |
| recentchanges_userindex |
| redirect                |
| revision                |
| revision_userindex      |
| site_identifiers        |
| site_stats              |
| sites                   |
| slot_roles              |
| slots                   |
| tag_summary             |
| templatelinks           |
| transcode               |
| updatelog               |
| user                    |
| user_former_groups      |
| user_groups             |
| user_properties         |
| valid_tag               |
| wbc_entity_usage        |
+-------------------------+
59 rows in set (0.02 sec)

Note that I have only run sudo /usr/local/sbin/maintain-views --databases idwikimedia I have not run any of the other usual scripts involved in the views handling/creation

• Dustinapril2018 added a subscriber: • Dustinapril2018.Jun 8 2018, 9:44 AM

This is a user that does use that role. The maintainviews user doesn't have rights to create a database of that name (which is surprising to me). The maintainviews user can manage the grants as I understand just fine, but I could have sworn I used it in the past to create _p databases during the run.

It failed with permission denied, operational error on CREATE DATABASE idwikimedia_p as maintainviews.localhost

I have run this script without this error (maybe some other action created the DB?), so I find it strange. This happens just *before* the grant.

*edit* perhaps that's the problem. I think the grant should come before the create? I'm going to try that.

In T193187#4267569, @Bstorm wrote:

*edit* perhaps that's the problem. I think the grant should come before the create? I'm going to try that.

Yep, it needs to come before the CREATE.
Once you've amended the script, try this same thing on either 1009 or 1011 to see if you get the same thing.

Change 438280 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] wiki replicas: place grant before create in maintain-views script

https://gerrit.wikimedia.org/r/438280

gerritbot added a project: Patch-For-Review.Jun 8 2018, 5:44 PM

Change 438280 merged by Bstorm:
[operations/puppet@production] wiki replicas: place grant before create in maintain-views script

https://gerrit.wikimedia.org/r/438280

Ok, that did not work. If anything is run that will create the idwikimedia_p database, the script runs fine. However, it cannot run it's own grant or the create. I find this baffling because I thought, for sure, that the script did this just fine before (and it completed everything without the GRANT being run so that it needed to be run by hand at the end).

So the user this runs as lacks the GRANT OPTION (confirmed that). However, the creation not succeeding just seems strange to me because I did not encounter that error before. This does have "ALL PRIVILEGES" on %\_p.* Is that enough to allow creating the DBs? I looked around and realized that it doesn't have the grant option and probably needs it to do this.

The reason the manual actions allowed it to succeed strike me as something created the database, which would have it skip the CREATE and GRANT statements entirely.

What have changed within the script lately? I am asking because within labs grants noting has changed.

In order to debug further, can you explain a bit what is the workflow you have and which users and SQL operations are involved per users?

This is what I did for idwikimedia that apparently worked, let's see what is different from your workflow:

sudo /usr/local/sbin/maintain-views --databases idwikimedia which failed

As root:

GRANT SELECT, SHOW VIEW ON `idwikimedia\_p`.* to labsdbuser;

sudo /usr/local/sbin/maintain-views --databases idwikimedia and the script went thru.

In T193187#4267930, @Bstorm wrote:

So the user this runs as lacks the GRANT OPTION (confirmed that). However, the creation not succeeding just seems strange to me because I did not encounter that error before. This does have "ALL PRIVILEGES" on %\_p.* Is that enough to allow creating the DBs? I looked around and realized that it doesn't have the grant option and probably needs it to do this.

I tested it to confirm it should be fine:

root@labsdb1010:~# mysql -umaintainviews -p --skip-ssl
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3926331
Server version: 10.1.33-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql:maintainviews@localhost [(none)]> set session sql_log_bin=0;
Query OK, 0 rows affected (0.00 sec)

mysql:maintainviews@localhost [(none)]> create database this_is_a_test_p;
Query OK, 1 row affected (0.00 sec)

mysql:maintainviews@localhost [(none)]> use this_is_a_test_p;
Database changed
mysql:maintainviews@localhost [this_is_a_test_p]> drop database this_is_a_test_p;
Query OK, 0 rows affected (0.00 sec)

I think I know what's up. I think I misread the stack trace, and it may be failing only on the grant. If so, it could explain a lot. I'll test when I get a chance. That user should be able to do all needed except the grant itself, which is a permission it is missing.

If the scripts fails on the grant creation is the only thing that fails, it is most likely related to the missing GRANT option.
I have tried to reproduce it on my lab:

mysql -umaintainviews
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 67
Server version: 10.1.33-MariaDB-1~xenial mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show grants for 'maintainviews'@'localhost';
<snip>
| GRANT ALL PRIVILEGES ON `%\_p`.* TO 'maintainviews'@'localhost'             |
| GRANT ALL PRIVILEGES ON `%wik%\_p`.* TO 'maintainviews'@'localhost'         |
<snip>


MariaDB [(none)]> create database test20_p;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> GRANT SELECT, SHOW VIEW ON `test20\_p`.* to test_user;
ERROR 1044 (42000): Access denied for user 'maintainviews'@'localhost' to database 'test20\_p'
MariaDB [(none)]> Ctrl-C -- exit!

And now I grant that privilege to my maintainviews user with root:

MariaDB [(none)]> GRANT GRANT OPTION ON `%\_p`.* TO 'maintainviews'@'localhost';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> show grants for 'maintainviews'@'localhost';
+-----------------------------------------------------------------------------------+
| Grants for maintainviews@localhost                                                |
+-----------------------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON `%\_p`.* TO 'maintainviews'@'localhost' WITH GRANT OPTION |

Let's try again with the maintainviews user:

 mysql -umaintainviews
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 69
Server version: 10.1.33-MariaDB-1~xenial mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> GRANT SELECT, SHOW VIEW ON `test20\_p`.* to test_user;
Query OK, 0 rows affected (0.00 sec)

I will grant GRANT for maintainviews to %\_p and %wik%\_p on Monday only to labsdb1010 so you can test again.

@Bstorm please try again on labsdb1010, I have added the GRANT grant

| GRANT ALL PRIVILEGES ON `%wik%\_p`.* TO 'maintainviews'@'localhost' WITH GRANT OPTION                                |
| GRANT ALL PRIVILEGES ON `%\_p`.* TO 'maintainviews'@'localhost' WITH GRANT OPTION

I removed the idwikimedia_p database on labsdb1010, then I ran the script with debug mode on (so I am damned sure where it dies next time), and it worked great! Script completed. Do you want to verify what it did to grants before I run it on the other servers, @Marostegui ?

In T193187#4272778, @Bstorm wrote:

I removed the idwikimedia_p database on labsdb1010, then I ran the script with debug mode on (so I am damned sure where it dies next time), and it worked great! Script completed. Do you want to verify what it did to grants before I run it on the other servers, @Marostegui ?

Yeah, let me check the grants first.
Can you run the script for another of the pending views to be created on labsdb1010 first to confirm the script works fine from scratch? Remember I did some manual steps on: T193187#4266857 so let's try with one of the pending wikis on this host

I certainly can. Running it for T196362

Grants for idwikimedia_p look fine as I created that one manually, let's try with another wiki only on labsdb1010.

*sigh* It failed on CREATE. I have no idea how or why:

2018-06-11 16:13:01,612 DEBUG Removing 0 dbs as sensitive
2018-06-11 16:13:01,617 DEBUG SQL: GRANT SELECT, SHOW VIEW ON `sahwikiquote\_p`.* TO 'labsdbuser';
2018-06-11 16:13:01,620 DEBUG SQL: CREATE DATABASE `sahwikiquote_p`;
Traceback (most recent call last):
  File "/usr/local/sbin/maintain-views", line 594, in <module>
    main()
  File "/usr/local/sbin/maintain-views", line 575, in main
    ops.execute(fullviews, customviews)
  File "/usr/local/sbin/maintain-views", line 365, in execute
    "CREATE DATABASE `{}`;".format(self.db_p)
  File "/usr/local/sbin/maintain-views", line 54, in write_execute
    self.cursor.execute(query)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 166, in execute
    result = self._query(query)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 322, in _query
    conn.query(q)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 852, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1053, in _read_query_result
    result.read()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1336, in read
    first_packet = self.connection._read_packet()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1010, in _read_packet
    packet.check_error()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 393, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python3/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1044, "Access denied for user 'maintainviews'@'localhost' to database 'sahwikiquote_p'")

It should have created the grants for it, though :)

The grants look good:

| GRANT SELECT, SHOW VIEW ON `sahwikiquote\_p`.* TO 'labsdbuser'

I did a FLUSH PRIVILEGES just in case, can you try again the script?

Fascinating! It died on the GRANT this time (possibly since the grant is already in place?). The additional backslash should just be python string display whatnot.

2018-06-11 16:17:00,109 DEBUG Removing 0 dbs as sensitive
2018-06-11 16:17:00,111 DEBUG SQL: GRANT SELECT, SHOW VIEW ON `sahwikiquote\_p`.* TO 'labsdbuser';
Traceback (most recent call last):
  File "/usr/local/sbin/maintain-views", line 594, in <module>
    main()
  File "/usr/local/sbin/maintain-views", line 575, in main
    ops.execute(fullviews, customviews)
  File "/usr/local/sbin/maintain-views", line 362, in execute
    self.db_p.replace('_', '\\_'))
  File "/usr/local/sbin/maintain-views", line 54, in write_execute
    self.cursor.execute(query)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 166, in execute
    result = self._query(query)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 322, in _query
    conn.query(q)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 852, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1053, in _read_query_result
    result.read()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1336, in read
    first_packet = self.connection._read_packet()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1010, in _read_packet
    packet.check_error()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 393, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python3/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1044, "Access denied for user 'maintainviews'@'localhost' to database 'sahwikiquote\\_p'")

Yeah, I was wondering how you'd handle the fact that the grant is already added.
No other grants were added, so can you try to skip that part now and see if it creates the DB this time? I guess you'd need to write a check to see if the GRANT is there and then skip that step

Yeah, that's easy enough to add. Let me try that.

@Bstorm and myself are trying to debug the issue and it is all very weird sometimes commands work, sometimes they don't, this is an example:

mysql:root@localhost [(none)]> revoke SELECT, SHOW VIEW ON `sahwikiquote\_p`.* FROM 'labsdbuser';
Query OK, 0 rows affected (0.00 sec)

mysql:root@localhost [(none)]> pager grep sahwiki; show grants for labsdbuser;
PAGER set to 'grep sahwiki'
| GRANT SELECT, SHOW VIEW ON `sahwikisource\_p`.* TO 'labsdbuser'          |
| GRANT SELECT, SHOW VIEW ON `sahwiki\_p`.* TO 'labsdbuser'                |
891 rows in set (0.00 sec)



root@labsdb1010:~# mysql -umaintainviews -p --skip-ssl
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7119399
Server version: 10.1.33-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql:maintainviews@localhost [(none)]> set session sql_log_bin=0;
Query OK, 0 rows affected (0.00 sec)

mysql:maintainviews@localhost [(none)]> GRANT SELECT, SHOW VIEW ON `sahwikiquote\_p`.* TO 'labsdbuser';
Query OK, 0 rows affected (0.00 sec)

mysql:maintainviews@localhost [(none)]> create database sahwikiquote_p;
Query OK, 1 row affected (0.00 sec)

A few minutes later after revoking those roles and trying to create them again:

mysql:maintainviews@localhost [(none)]> GRANT SELECT, SHOW VIEW ON `sahwikiquote\_p`.* TO 'labsdbuser';
ERROR 1044 (42000): Access denied for user 'maintainviews'@'localhost' to database 'sahwikiquote\_p'

More food for thought:

Create the grant as root
connect as maintainviews and play with the role (revoking+granting) works
disconnect, and connect again as maintaviews and then I cannot do anything else

MariaDB bug?

root@labsdb1010:~# mysql -uroot --skip-ssl
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7151174
Server version: 10.1.33-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql:root@localhost [(none)]> set session sql_log_bin=0;
Query OK, 0 rows affected (0.00 sec)

mysql:root@localhost [(none)]> GRANT SELECT, SHOW VIEW ON `sahwikiquote\_p`.* TO 'labsdbuser';
Query OK, 0 rows affected (0.00 sec)

mysql:root@localhost [(none)]> Ctrl-C -- exit!
Aborted
root@labsdb1010:~# mysql -umaintainviews -p --skip-ssl
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7151515
Server version: 10.1.33-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql:maintainviews@localhost [(none)]> GRANT SELECT, SHOW VIEW ON `sahwikiquote\_p`.* TO 'labsdbuser';
Query OK, 0 rows affected (0.00 sec)

mysql:maintainviews@localhost [(none)]> revoke SELECT, SHOW VIEW ON `sahwikiquote\_p`.* FROM 'labsdbuser';
Query OK, 0 rows affected (0.00 sec)

mysql:maintainviews@localhost [(none)]> GRANT SELECT, SHOW VIEW ON `sahwikiquote\_p`.* TO 'labsdbuser';
Query OK, 0 rows affected (0.00 sec)

mysql:maintainviews@localhost [(none)]> Ctrl-C -- exit!
Aborted
root@labsdb1010:~# mysql -umaintainviews -p --skip-ssl
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7152593
Server version: 10.1.33-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql:maintainviews@localhost [(none)]> revoke SELECT, SHOW VIEW ON `sahwikiquote\_p`.* FROM 'labsdbuser';
ERROR 1044 (42000): Access denied for user 'maintainviews'@'localhost' to database 'sahwikiquote\_p'
mysql:maintainviews@localhost [(none)]> GRANT SELECT, SHOW VIEW ON `sahwikiquote\_p`.* TO 'labsdbuser';
ERROR 1044 (42000): Access denied for user 'maintainviews'@'localhost' to database 'sahwikiquote\_p'

Looks like a bug?:

root@labsdb1010:~# mysql --skip-ssl -uroot
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7156483
Server version: 10.1.33-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql:root@localhost [(none)]> revoke SELECT, SHOW VIEW ON `sahwikiquote\_p`.* FROM 'labsdbuser';
ERROR 1141 (42000): There is no such grant defined for user 'labsdbuser' on host ''
mysql:root@localhost [(none)]> GRANT SELECT, SHOW VIEW ON `sahwikiquote\_p`.* TO 'labsdbuser';
Query OK, 0 rows affected (0.00 sec)

mysql:root@localhost [(none)]> Ctrl-C -- exit!
Aborted
root@labsdb1010:~# mysql -umaintainviews -p --skip-ssl
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7156617
Server version: 10.1.33-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql:maintainviews@localhost [(none)]> revoke SELECT, SHOW VIEW ON `sahwikiquote\_p`.* FROM 'labsdbuser';
Query OK, 0 rows affected (0.00 sec)

mysql:maintainviews@localhost [(none)]> GRANT SELECT, SHOW VIEW ON `sahwikiquote\_p`.* TO 'labsdbuser';
Query OK, 0 rows affected (0.00 sec)

mysql:maintainviews@localhost [(none)]> Ctrl-C -- exit!
Aborted
root@labsdb1010:~# mysql -umaintainviews -p --skip-ssl
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7156684
Server version: 10.1.33-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql:maintainviews@localhost [(none)]> revoke SELECT, SHOW VIEW ON `sahwikiquote\_p`.* FROM 'labsdbuser';
ERROR 1044 (42000): Access denied for user 'maintainviews'@'localhost' to database 'sahwikiquote\_p'
mysql:maintainviews@localhost [(none)]> Ctrl-C -- exit!
Aborted

I have been doing more tests with this and it really looks like something weird is going on, so I have filed a MariaDB bug: https://jira.mariadb.org/browse/MDEV-16466

@Bstorm in order not to block this on the issues we are having with the grant, I have manually run the database and grant creation on all the hosts.
Please feel free to proceed with the views creation

Creation is finished. Waiting on dns scripts before close.

Seems it is working.

urbanecm@tools-bastion-02 ~ 
$ sql --cluster=analytics idwikimedia
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7747812
Server version: 10.1.33-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [idwikimedia_p]> Bye
urbanecm@tools-bastion-02 ~ 
$ sql --cluster=web idwikimedia
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 13226172
Server version: 10.1.33-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [idwikimedia_p]> Bye
urbanecm@tools-bastion-02 ~ 
$ host idwikimedia.analytics.db.svc.eqiad.wmflabs
idwikimedia.analytics.db.svc.eqiad.wmflabs is an alias for s3.analytics.db.svc.eqiad.wmflabs.
s3.analytics.db.svc.eqiad.wmflabs has address 10.64.37.14
urbanecm@tools-bastion-02 ~ 
$ host idwikimedia.web.db.svc.eqiad.wmflabs                                                                                                                                                                 
idwikimedia.web.db.svc.eqiad.wmflabs is an alias for s3.web.db.svc.eqiad.wmflabs.
s3.web.db.svc.eqiad.wmflabs has address 10.64.37.15
urbanecm@tools-bastion-02 ~ 
$

• Vvjjkkii renamed this task from Prepare and check storage layer for idwikimedia to 04daaaaaaa.Jul 1 2018, 1:13 AM

• Vvjjkkii reopened this task as Open.

• Vvjjkkii removed Bstorm as the assignee of this task.

• Vvjjkkii raised the priority of this task from Medium to High.