Let us know when this is created so we can sanitize it

Mentioned in SAL (#wikimedia-operations) [2018-08-02T12:19:08Z] <Reedy> Wikis created T196748 T198401 T199599

Mentioned in SAL (#wikimedia-operations) [2018-08-02T12:58:48Z] <marostegui> Sanitize zhwikiversity satwiki T199599 T198401

I have run sudo /usr/local/sbin/maintain-replica-indexes --debug --database zhwikiversity on all 3 wiki replica hosts.

@Bstorm I can't finish this one due to pymysql.err.OperationalError: (1044, "Access denied for user 'maintainviews'@'localhost' to database 'satwiki\\_p'") on all 3 hosts. Your root powers are required.

I assume you meant zhwikiversity, I have granted that on all hosts. Try re-running the script.

for i in labsdb1009 labsdb1010 labsdb1011; do echo $i; mysql.py -h$i -e " show grants for 'labsdbuser';" | grep -i zhwikiversity;done
labsdb1009
GRANT SELECT, SHOW VIEW ON `zhwikiversity\\_p`.* TO 'labsdbuser'
labsdb1010
GRANT SELECT, SHOW VIEW ON `zhwikiversity\\_p`.* TO 'labsdbuser'
labsdb1011
GRANT SELECT, SHOW VIEW ON `zhwikiversity\\_p`.* TO 'labsdbuser'

I think the failure I'm seeing is the same upstream wildcard grant bug that we have seen before. The maintain-views script connects to the databsase server as the maintainviews user. That user is being denied the ability to operate on the zhwikiversity_p database.

2018-08-03 18:48:11,762 DEBUG SQL: GRANT SELECT, SHOW VIEW ON `zhwikiversity\_p`.* TO 'labsdbuser';
Traceback (most recent call last):
  File "/usr/local/sbin/maintain-views", line 594, in <module>
    main()
  File "/usr/local/sbin/maintain-views", line 575, in main
    ops.execute(fullviews, customviews)
  File "/usr/local/sbin/maintain-views", line 362, in execute
    self.db_p.replace('_', '\\_'))
  File "/usr/local/sbin/maintain-views", line 54, in write_execute
    self.cursor.execute(query)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 166, in execute
    result = self._query(query)
  File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 322, in _query
    conn.query(q)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 852, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1053, in _read_query_result
    result.read()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1336, in read
    first_packet = self.connection._read_packet()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1010, in _read_packet
    packet.check_error()
  File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 393, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python3/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1044, "Access denied for user 'maintainviews'@'localhost' to database 'zhwikiversity\\_p'")

The bug is weird. The grant is already there if @Marostegui ran it, so the script doesn't need to do it. It only tries to do it if the database with the _p doesn't exist. It also often will randomly fail to create the DB. I think right now, it tries the grant first. If we create the DB, it'll run. It won't attempt to do the grant itself if the DB exists.

Yup, that did it. I created the DB manually, and it ran fine.

2018-08-03T19:51:57Z mwopenstackclients.DnsManager WARNING : Creating id_internalwikimedia.analytics.db.svc.eqiad.wmflabs.
2018-08-03T19:53:37Z mwopenstackclients.DnsManager WARNING : Creating satwiki.analytics.db.svc.eqiad.wmflabs.
2018-08-03T19:54:44Z mwopenstackclients.DnsManager WARNING : Creating wikimaniawiki.analytics.db.svc.eqiad.wmflabs.
2018-08-03T19:54:55Z mwopenstackclients.DnsManager WARNING : Creating zhwikiversity.analytics.db.svc.eqiad.wmflabs.
2018-08-03T19:56:55Z mwopenstackclients.DnsManager WARNING : Creating id_internalwikimedia.web.db.svc.eqiad.wmflabs.
2018-08-03T19:58:36Z mwopenstackclients.DnsManager WARNING : Creating satwiki.web.db.svc.eqiad.wmflabs.
2018-08-03T19:59:38Z mwopenstackclients.DnsManager WARNING : Creating wikimaniawiki.web.db.svc.eqiad.wmflabs.
2018-08-03T19:59:48Z mwopenstackclients.DnsManager WARNING : Creating zhwikiversity.web.db.svc.eqiad.wmflabs.

The dns step set up all the new wiki db names.