Page MenuHomePhabricator
Create Task
Maniphest T202588

Do the big Quarry migration
Closed, ResolvedPublic
Actions

Description

Quarry will be subject to server maintenance on Wednesday, September 12 from 7pm UTC. The site will be read-only for a few hours, but should remain online.

Ordered to-do list:

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
killer.py: fix import since py3analytics/quarry/webmaster+1 -1
quarry::database: Use mariadb instead of mysql moduleoperations/puppetproduction+12 -7
connections.py: Explicitly enable MULTI_STATEMENTS for replicaanalytics/quarry/webmaster+2 -1
Port to Python3analytics/quarry/webmaster+66 -83
Update dependenciesanalytics/quarry/webmaster+413 -37
quarry: Move the install into a venv and upgrade to Python 3operations/puppetproduction+33 -411
app.py: Load Redis for session from Connectionsanalytics/quarry/webmaster+2 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Framawiki created this task.Aug 22 2018, 10:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 22 2018, 10:13 PM
Framawiki triaged this task as High priority.EditedAug 22 2018, 10:14 PM
Framawiki added subtasks: T194691: Upgrade Quarry main server from Trusty, T192698: Upgrade Quarry to python 3, T181205: Let quarry use the mariadb module.

@zhuyifei1999 I purpose this ordered migration to-do list, feel free to edit it.
Normally we should have only a read-only period, no downtime :)

Framawiki updated the task description. (Show Details)Aug 22 2018, 10:22 PM
zhuyifei1999 awarded a token.Aug 23 2018, 4:19 AM
zhuyifei1999 added a comment.EditedAug 23 2018, 8:38 AM

Backup sql db and resultset folder of legacy live main instance

I'm pretty sure the results live on NFS, so there isn't really a need to back this up (I think).

zhuyifei1999 added a comment.EditedAug 24 2018, 7:52 PM

The current NFS implementation of resultsets is flawed. The runners have user 'quarry' with UID of 998, but main has it as 997. Currently it's not too bad, just that puppet keeps changing the ownership of /data/project/quarry/results/ back and forth, but if the two runners have different UIDs between them, it'll go really bad.

We should probably do T178520: Find somewhere else (not NFS) to store Quarry's resultsets after this task, if possible.

gerritbot subscribed.Sep 5 2018, 7:50 PM

Change 454481 had a related patch set uploaded (by Dzahn; owner: Zhuyifei1999):
[operations/puppet@production] quarry::database: Use mariadb instead of mysql module

https://gerrit.wikimedia.org/r/454481

gerritbot added a project: Patch-For-Review.Sep 5 2018, 7:50 PM
Dzahn mentioned this in T162070: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases.Sep 5 2018, 7:51 PM
Dzahn mentioned this in T165625: Evaluate future of wmf puppet module "mysql".
gerritbot added a comment.Sep 6 2018, 4:04 AM

Change 458351 had a related patch set uploaded (by Zhuyifei1999; owner: Zhuyifei1999):
[analytics/quarry/web@master] app.py: Load Redis for session from Connections

https://gerrit.wikimedia.org/r/458351

zhuyifei1999 added a comment.Sep 6 2018, 4:15 AM

@Framawiki I think we can proceed with the migration (the test instance at https://quarry-dev.wmflabs.org/ shares NFS with the production instance so I'm too afraid to do any real testing there). When do you have time? I usually have time from 9PM to 3AM UTC (the next day) from Tuesday to Thursday, and 3PM to 3AM UTC (the next day) on weekends. Considering we have a few puppet patches, might be better to merge on a weekday.

zhuyifei1999 added a comment.Sep 6 2018, 8:25 PM

Scheduled for 7pm UTC next Wednesday

Stashbot subscribed.Sep 7 2018, 6:11 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-07T18:11:07Z] <framawiki> deployed 769cace to quarry-main-01 (T202588)

Framawiki added a comment.Sep 7 2018, 6:13 PM

I've set the following maintenance message:

Quarry will be subject to server maintenance on Wednesday, September 12 from 7pm UTC. The site will be read-only for a few hours, but should remain online.

Stashbot added a comment.Sep 7 2018, 7:56 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-07T19:56:53Z] <framawiki> deployed 501695f to quarry-main-01 (T202588)

Framawiki removed a project: Patch-For-Review.Sep 7 2018, 8:15 PM
Framawiki updated the task description. (Show Details)
Framawiki updated the task description. (Show Details)Sep 7 2018, 8:17 PM
gerritbot added a comment.Sep 7 2018, 8:24 PM

Change 458351 merged by jenkins-bot:
[analytics/quarry/web@master] app.py: Load Redis for session from Connections

https://gerrit.wikimedia.org/r/458351

Framawiki updated the task description. (Show Details)Sep 7 2018, 9:21 PM
gerritbot added a comment.Sep 7 2018, 9:24 PM

Change 451698 had a related patch set uploaded (by Zhuyifei1999; owner: Zhuyifei1999):
[operations/puppet@production] quarry: Move the install into a venv and upgrade to Python 3

https://gerrit.wikimedia.org/r/451698

gerritbot added a project: Patch-For-Review.Sep 7 2018, 9:24 PM
Framawiki updated the task description. (Show Details)Sep 7 2018, 9:25 PM
Stashbot added a comment.Sep 11 2018, 5:14 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-11T17:14:22Z] <zhuyifei1999_> disabling puppet on quarry-main-01, quarry-runner-0{1,2} T202588

zhuyifei1999 updated the task description. (Show Details)Sep 11 2018, 5:16 PM
zhuyifei1999 updated the task description. (Show Details)
zhuyifei1999 updated the task description. (Show Details)Sep 11 2018, 5:18 PM
Stashbot added a comment.Sep 11 2018, 5:22 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-11T17:22:48Z] <zhuyifei1999_> doing another backup of main db: sudo mysqldump quarry | sudo tee /data/project/dump-$(date '+%Y-%m-%d').sql > /dev/null T202588

zhuyifei1999 updated the task description. (Show Details)Sep 11 2018, 5:23 PM
zhuyifei1999 updated the task description. (Show Details)Sep 11 2018, 5:26 PM
gerritbot added a comment.Sep 12 2018, 5:09 PM

Change 454481 merged by Bstorm:
[operations/puppet@production] quarry::database: Use mariadb instead of mysql module

https://gerrit.wikimedia.org/r/454481

gerritbot added a comment.Sep 12 2018, 5:18 PM

Change 451698 merged by Bstorm:
[operations/puppet@production] quarry: Move the install into a venv and upgrade to Python 3

https://gerrit.wikimedia.org/r/451698

gerritbot added a comment.Sep 12 2018, 5:21 PM

Change 428140 had a related patch set uploaded (by Zhuyifei1999; owner: Framawiki):
[analytics/quarry/web@master] Update dependencies

https://gerrit.wikimedia.org/r/428140

gerritbot added a comment.Sep 12 2018, 5:22 PM

Change 440007 had a related patch set uploaded (by Zhuyifei1999; owner: Framawiki):
[analytics/quarry/web@master] Port to Python3

https://gerrit.wikimedia.org/r/440007

Framawiki updated the task description. (Show Details)Sep 12 2018, 6:02 PM
gerritbot added a comment.Sep 12 2018, 7:06 PM

Change 428140 merged by jenkins-bot:
[analytics/quarry/web@master] Update dependencies

https://gerrit.wikimedia.org/r/428140

gerritbot added a comment.Sep 12 2018, 7:07 PM

Change 440007 merged by jenkins-bot:
[analytics/quarry/web@master] Port to Python3

https://gerrit.wikimedia.org/r/440007

Framawiki updated the task description. (Show Details)Sep 12 2018, 7:13 PM
Framawiki added a comment.EditedSep 12 2018, 7:16 PM

For the record

New instances:
quarry-web-01.quarry.eqiad.wmflabs

quarry-db-01.quarry.eqiad.wmflabs

quarry-runner-01.quarry.eqiad.wmflabs

quarry-runner-02.quarry.eqiad.wmflabs

Old ones:
quarry-main-01.quarry.eqiad.wmflabs
role::labs::quarry::killer
role::labs::quarry::web
role::labs::quarry::redis
role::labs::lvm::srv
role::labs::quarry::database

quarry-worker-01.quarry.eqiad.wmflabs
role::labs::quarry::celeryrunner

quarry-worker-02.quarry.eqiad.wmflabs
role::labs::quarry::celeryrunner
role::labs::quarry::killer

Stashbot added a comment.Sep 12 2018, 7:45 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-12T19:45:34Z] <zhuyifei1999_> created new quarry database and user in quarry-db-01.quarry.eqiad.wmflabs T202588

Stashbot added a comment.Sep 12 2018, 8:02 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-12T20:02:04Z] <zhuyifei1999_> stopped celery-quarry-worker on quarry-runner-0{1,2} T202588

Stashbot added a comment.Sep 12 2018, 8:03 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-12T20:03:54Z] <zhuyifei1999_> set quarry-main-01 mariadb read-only T202588

zhuyifei1999 updated the task description. (Show Details)Sep 12 2018, 8:04 PM
Framawiki removed a project: Patch-For-Review.Sep 12 2018, 8:21 PM
Stashbot added a comment.Sep 12 2018, 8:27 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-12T20:27:11Z] <zhuyifei1999_> backed up old db to /data/project/dump-2018-09-12.sql and restoring to new server T202588

zhuyifei1999 updated the task description. (Show Details)Sep 12 2018, 8:33 PM
Framawiki mentioned this in T204161: "Recent queries" page badly shows time.Sep 12 2018, 8:38 PM
Stashbot added a comment.Sep 12 2018, 8:42 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-12T20:42:34Z] <zhuyifei1999_> unset read-only on new database T202588

Framawiki added a comment.EditedSep 12 2018, 8:44 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-12T20:41] <framawiki> switched quarry.wmflabs.org proxy to new quarry-web-01.quarry.eqiad.wmflabs

Framawiki updated the task description. (Show Details)Sep 12 2018, 8:47 PM
Stashbot added a comment.Sep 12 2018, 9:13 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-12T21:13:27Z] <zhuyifei1999_> set read-only again on new database because new quarry's UID is 498 T202588

Stashbot added a comment.Sep 12 2018, 9:15 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-12T21:15:40Z] <zhuyifei1999_> sudo chown quarry:quarry /data/project/quarry/ -Rv T202588

gerritbot added a comment.Sep 12 2018, 9:42 PM

Change 460111 had a related patch set uploaded (by Zhuyifei1999; owner: Zhuyifei1999):
[analytics/quarry/web@master] connections.py: Explicitly enable MULTI_STATEMENTS for replica

https://gerrit.wikimedia.org/r/460111

gerritbot added a project: Patch-For-Review.Sep 12 2018, 9:43 PM

Change 460111 merged by jenkins-bot:
[analytics/quarry/web@master] connections.py: Explicitly enable MULTI_STATEMENTS for replica

https://gerrit.wikimedia.org/r/460111

Framawiki updated the task description. (Show Details)Sep 12 2018, 10:22 PM
zhuyifei1999 closed subtask T192698: Upgrade Quarry to python 3 as Resolved.Sep 12 2018, 11:17 PM
zhuyifei1999 closed subtask T194691: Upgrade Quarry main server from Trusty as Resolved.
Stashbot added a comment.Sep 13 2018, 7:10 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-13T19:10:07Z] <framawiki> copy /var/log/nginx from legacy main-01 to /data/project/nginx-logs-legacy-20180913-framawiki for further analysis T202588 T197256

Stashbot mentioned this in T197256: GoogleDocs bot has download 125 000 csv exports in the last month.Sep 13 2018, 7:10 PM
Stashbot added a comment.Sep 13 2018, 7:19 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-13T19:19:09Z] <framawiki> deleted legacy instances quarry-main-01 and quarry-runner-0{1,2}, migration is over T202588

Framawiki closed this task as Resolved.Sep 13 2018, 7:20 PM
Framawiki claimed this task.

The migration is complete ! \o/

zhuyifei1999 closed subtask T181205: Let quarry use the mariadb module as Resolved.Sep 13 2018, 9:11 PM
zhuyifei1999 mentioned this in T136315: puppet disabled on quarry-main-01.Sep 13 2018, 10:02 PM
Dzahn added a subtask: T153615: Remove dependency on labs_debrepo.Sep 13 2018, 10:04 PM
zhuyifei1999 closed subtask T153615: Remove dependency on labs_debrepo as Resolved.Sep 13 2018, 10:10 PM
gerritbot added a comment.Sep 23 2018, 2:29 PM

Change 462295 had a related patch set uploaded (by Framawiki; owner: Framawiki):
[analytics/quarry/web@master] killer.py: fix import since py3

https://gerrit.wikimedia.org/r/462295

gerritbot added a comment.Sep 23 2018, 4:24 PM

Change 462295 merged by jenkins-bot:
[analytics/quarry/web@master] killer.py: fix import since py3

https://gerrit.wikimedia.org/r/462295

Stashbot added a comment.Sep 23 2018, 4:56 PM

Mentioned in SAL (#wikimedia-cloud) [2018-09-23T16:56:55Z] <zhuyifei1999_> deployed till e74f575 on -web-01, T202588 T205153

Stashbot mentioned this in T205153: Add full date as tooltip in Recent queries "Last run" column.Sep 23 2018, 4:56 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits