Page MenuHomePhabricator

Prepare and check storage layer for fixcopyright.wikimedia.org
Closed, ResolvedPublic

Description

The new wiki is going to be a fishbowl wiki.

Event Timeline

Urbanecm updated the task description. (Show Details)Aug 26 2018, 1:09 PM
Urbanecm added a subscriber: Urbanecm.
Marostegui added a subscriber: Marostegui.

Let us know when this wiki is created so we can sanitize it and give cloud-services-team the green light to create the views.

Marostegui triaged this task as Medium priority.Aug 27 2018, 5:21 AM
Reedy added a subscriber: Reedy.Aug 30 2018, 12:30 PM

Wiki is created

jcrespo claimed this task.Aug 30 2018, 12:31 PM
jcrespo added a subscriber: jcrespo.

Will sanitize for replication before communicating to cloud.

Mentioned in SAL (#wikimedia-operations) [2018-08-30T13:08:33Z] <jynus> sanitizing fixcopyrightwiki on db1124 and children T202820

Mentioned in SAL (#wikimedia-operations) [2018-08-30T13:10:42Z] <jynus> sanitizing fixcopyrightwiki on db2094 and children T202820

jcrespo changed the task status from Open to Stalled.Aug 30 2018, 3:31 PM
jcrespo added a subscriber: Bstorm.

This is technically done, but I would like some users being filtered before handling it to cloud (there is no rows at the moment). Once it has some rows (and check that the filters work- only then) we will handle it to cloud team so the views can be created. CC @Bstorm

Stalled until that happens.

There is one row now there (I just checked the user table) and it is filtered correctly.
I would, however, as Jaime said, wait a bit more until we have a few more rows to make sure they are definitely being sanitized correctly. I tried to create an user but as it is a fishbowl, I could not do so.

Marostegui removed jcrespo as the assignee of this task.Sep 3 2018, 1:23 PM
Marostegui moved this task from Blocked external/Not db team to Done on the DBA board.

I talked to Jaime about that existing user, and when he first sanitized it, the user wasn't there, so it means it was created after the triggers got in place and they worked fine, because the user is sanitized, that means that this is ready for cloud-services-team to create the views.

Marostegui changed the task status from Stalled to Open.Sep 4 2018, 1:59 PM

In order to bypass the access denied typical error due to the mariadb bug, I have created the database (_p) and granted the labsdb user access to that db.

for i in labsdb1009 labsdb1010 labsdb1011; do echo $i; mysql.py -BN -h$i -e "show databases like 'fixcopyrightwiki_p'" ;  done
labsdb1009
fixcopyrightwiki_p
labsdb1010
fixcopyrightwiki_p
labsdb1011
fixcopyrightwiki_p

On all labsdb hosts:

GRANT SELECT, SHOW VIEW ON `fixcopyrightwiki\_p`.* TO 'labsdbuser'

@Bstorm you are good to go! Thanks!

Bstorm added a comment.Sep 4 2018, 4:24 PM

@Marostegui, on all three replicas, I just got:

pymysql.err.InternalError: (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement')

For creating indexes by script. User is maintainindexes
Apparently it doesn't have the super option on these servers?

We haven't changed anything from our side as far as I know - anything changed from your side?

Bstorm added a comment.Sep 4 2018, 4:27 PM

Nope. It could have just been missed in the blur of GRANT changes? I thought we'd done one of these since, though.

Bstorm added a comment.Sep 4 2018, 4:29 PM

SHOW GRANTS doesn't show the SUPER option on that user, though. Huh.

Yeah, not sure what happened, it is supposed to have it: https://github.com/wikimedia/puppet/blob/production/modules/role/templates/mariadb/grants/wiki-replicas.sql#L33
I have granted it, please try again :-)

Bstorm added a comment.Sep 4 2018, 4:33 PM

Works! views and indexes are in place. Setting up DNS and so forth.

Bstorm added a comment.EditedSep 4 2018, 5:13 PM

Noted for my reference:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='cloudservices1003.wikimedia.org', port=9001): Max retries exceeded with url: /v2/zones?name=analytics.db.svc.eqiad.wmflabs. (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f43d5e359d0>: Failed to establish a new connection: [Errno 110] Connection timed out',))

I've never seen that before from the DNS script. I think I might need to fix something there. We haven't tried it since the change to cloudservices1003.

Bstorm closed this task as Resolved.Sep 4 2018, 5:39 PM
Bstorm claimed this task.

Updated docs to get around the above problem. All steps are now done for cloud services.
I am able to connect from tools.