Page MenuHomePhabricator
Create Task
Maniphest T153058

LabsDB infrastructure pending work
Closed, ResolvedPublic
Actions

Description

[WIP] Meta ticket too recap all the stuff pending and planning regarding the labs infra (we can always create more detailed subtasks)

  • Repoint labsdb1001 and labsdb1003 to db1095 (T166546)
  • Add and sanitize s2, s4, s5, s6 and s7 (T153743)
  • Puppetize summary tables such as watchlist_count (maybe others?)
  • implement GTID (requires production changes) (only on sanitariums for now, done)
  • Implement a periodic check_private_data.py run (T153680)

Feel free to edit the description and add stuff

Details

SubjectRepoBranchLines +/-
sanitarium_multiinstance: Keep less the binlogsoperations/puppetproduction+1 -1
db-eqiad.php: Depool db1072operations/mediawiki-configmaster+4 -4
db1072.yaml: Change binlog to ROWoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenJdrewniakT128546 [Recurring Task] Update Wikipedia and sister projects portals statistics
 ResolvedKrenairT134017 Create Wikipedia Jamaican
 ResolvedjcrespoT140788 Labs databases rearchitecture (tracking)
 ResolvedjcrespoT153058 LabsDB infrastructure pending work
 ResolvedMarosteguiT153680 Create a cronjob/check to run check_private_data data script and report back
 ResolvedMarosteguiT149418 Deploy gtid_domain_id flag in our mysql hosts
 StalledNoneT59617 Make watchlist table available as curated foo_p.watchlist_count on labsdb
 DuplicateNoneT93887 Replicate watchlist to labs
 DuplicateNoneT106089 Replicate sanitized watchlist table
 ResolvedoriT135029 adywiki and jamwiki are missing the associated *_p databases with appropriate views
Restricted Task
 Resolved• chasempT138450 maintain-replicas.pl unmaintained, unmaintainable
 ResolvedMarosteguiT153743 Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts
 ResolvedMarosteguiT157931 s5: db1070 not using file per table
 ResolvedMarosteguiT168021 setup dewiki and wikidatawiki on the labsdb1009, 1010 and 1011
 DeclinedNoneT156869 Design a method for keeping user-created tables in sync across labsDBs
 ResolvedMarosteguiT172169 Compress InnoDB on db1102
 ResolvedjcrespoT166546 Point labsdb1001 and labsdb1003 to db1095 and db1102
 ResolvedMarosteguiT172996 Migrate s4 from db1095 to db1102
 ResolvedjcrespoT182450 db1102 (sanitarium) filling up (WAS: Clean up old binlogs from db1102 (sanitarium multi-instance))

Event Timeline

Marostegui created this task.Dec 13 2016, 10:58 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 13 2016, 10:58 AM
Marostegui moved this task from Triage to Backlog on the DBA board.Dec 13 2016, 10:58 AM
jcrespo moved this task from Backlog to Meta/Epic on the DBA board.Dec 13 2016, 10:58 AM
jcrespo updated the task description. (Show Details)Dec 13 2016, 2:28 PM
Krenair subscribed.Dec 13 2016, 6:35 PM
jcrespo updated the task description. (Show Details)Dec 20 2016, 9:54 AM
Marostegui updated the task description. (Show Details)Dec 20 2016, 9:55 AM
jcrespo renamed this task from Pending things in the labs infra to LabsDB infrastructure pending work.Dec 20 2016, 9:55 AM
jcrespo updated the task description. (Show Details)
jcrespo updated the task description. (Show Details)Dec 20 2016, 10:00 AM
jcrespo added subtasks: T153680: Create a cronjob/check to run check_private_data data script and report back, T149418: Deploy gtid_domain_id flag in our mysql hosts, T59617: Make watchlist table available as curated foo_p.watchlist_count on labsdb.
Marostegui updated the task description. (Show Details)Dec 20 2016, 10:10 AM
Marostegui added a subtask: T153743: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts.
jcrespo added a parent task: T140788: Labs databases rearchitecture (tracking).Dec 21 2016, 5:35 PM
Halfak created subtask T156869: Design a method for keeping user-created tables in sync across labsDBs.Jan 31 2017, 11:00 PM
jcrespo changed the status of subtask T59617: Make watchlist table available as curated foo_p.watchlist_count on labsdb from Open to Stalled.Mar 3 2017, 9:51 AM
Marostegui closed subtask T149418: Deploy gtid_domain_id flag in our mysql hosts as Resolved.Mar 7 2017, 12:50 PM
Marostegui closed subtask T153680: Create a cronjob/check to run check_private_data data script and report back as Resolved.Apr 27 2017, 8:55 AM
jcrespo mentioned this in T138967: Labs database replica drift.May 24 2017, 3:51 PM
• MZMcBride subscribed.Jun 21 2017, 12:41 AM
Ankry subscribed.Jun 21 2017, 5:53 AM
Marostegui created subtask T172169: Compress InnoDB on db1102.Aug 1 2017, 5:32 AM
bd808 added a project: Data-Services.Aug 7 2017, 5:25 AM
bd808 moved this task from Backlog to Wiki replicas on the Data-Services board.
Marostegui closed subtask T172169: Compress InnoDB on db1102 as Resolved.Aug 7 2017, 6:23 AM
Marostegui closed subtask T153743: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts as Resolved.Aug 7 2017, 11:37 AM
jcrespo added a subtask: T166546: Point labsdb1001 and labsdb1003 to db1095 and db1102.Aug 7 2017, 11:50 AM
jcrespo updated the task description. (Show Details)
jcrespo updated the task description. (Show Details)Aug 7 2017, 3:59 PM
jcrespo closed subtask T166546: Point labsdb1001 and labsdb1003 to db1095 and db1102 as Resolved.Aug 9 2017, 7:39 AM
Stashbot subscribed.Aug 10 2017, 1:43 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-10T13:43:08Z] <marostegui> Compress cebwiki on db1095 - T153058

jcrespo added a subtask: T172996: Migrate s4 from db1095 to db1102.Aug 11 2017, 4:14 PM
Marostegui closed subtask T172996: Migrate s4 from db1095 to db1102 as Resolved.Aug 22 2017, 8:49 AM
jcrespo triaged this task as Low priority.Aug 22 2017, 8:56 AM
bd808 closed subtask T156869: Design a method for keeping user-created tables in sync across labsDBs as Declined.Aug 30 2017, 3:41 PM
gerritbot subscribed.Nov 23 2017, 7:56 AM

Change 393027 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1072.yaml: Change binlog to ROW

https://gerrit.wikimedia.org/r/393027

gerritbot added a project: Patch-For-Review.Nov 23 2017, 7:56 AM
gerritbot added a comment.Nov 23 2017, 7:58 AM

Change 393028 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1072

https://gerrit.wikimedia.org/r/393028

gerritbot added a comment.Nov 23 2017, 8:02 AM

Change 393027 abandoned by Marostegui:
db1072.yaml: Change binlog to ROW

Reason:
Already running ROW

https://gerrit.wikimedia.org/r/393027

gerritbot added a comment.Nov 23 2017, 8:02 AM

Change 393028 abandoned by Marostegui:
db-eqiad.php: Depool db1072

Reason:
Not needed - already running ROW

https://gerrit.wikimedia.org/r/393028

gerritbot added a comment.Nov 23 2017, 8:16 AM

Change 393027 restored by Marostegui:
db1072.yaml: Change binlog to ROW

https://gerrit.wikimedia.org/r/393027

gerritbot added a comment.Nov 23 2017, 8:16 AM

Change 393028 restored by Marostegui:
db-eqiad.php: Depool db1072

https://gerrit.wikimedia.org/r/393028

gerritbot added a comment.Nov 23 2017, 8:20 AM

Change 393028 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1072

https://gerrit.wikimedia.org/r/393028

gerritbot added a comment.Nov 23 2017, 8:22 AM

Change 393027 merged by Marostegui:
[operations/puppet@production] db1072.yaml: Change binlog to ROW

https://gerrit.wikimedia.org/r/393027

gerritbot added a comment.Dec 8 2017, 5:25 PM

Change 396435 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] sanitarium_multiinstance: Keep less the binlogs

https://gerrit.wikimedia.org/r/396435

gerritbot added a comment.Dec 8 2017, 5:31 PM

Change 396435 merged by Marostegui:
[operations/puppet@production] sanitarium_multiinstance: Keep less the binlogs

https://gerrit.wikimedia.org/r/396435

jcrespo closed subtask T182450: db1102 (sanitarium) filling up (WAS: Clean up old binlogs from db1102 (sanitarium multi-instance)) as Resolved.Dec 11 2017, 9:29 AM
Marostegui reopened subtask T182450: db1102 (sanitarium) filling up (WAS: Clean up old binlogs from db1102 (sanitarium multi-instance)) as Open.Jan 21 2018, 12:00 PM
Marostegui closed subtask T182450: db1102 (sanitarium) filling up (WAS: Clean up old binlogs from db1102 (sanitarium multi-instance)) as Resolved.Jan 23 2018, 12:33 PM
jcrespo added a comment.Mar 26 2018, 5:46 PM

Except for labsdb multi-instance/GTID and custom tables this is feature-complete, so this and T140788 can be closed and use individual subtickets for the few pending tasks.

jcrespo closed this task as Resolved.Mar 26 2018, 7:51 PM
jcrespo claimed this task.

Everthing except T59617 is done.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL