Page MenuHomePhabricator

Compress new Wikibase tables
Closed, ResolvedPublic

Description

This and maybe other newly created tables are in innodb uncompressed format, and probably will benefit a lot from it:

| wbt_item_terms           | InnoDB |      10 | Compact    |   79850814 |             38 |   3068133376 |               
| wbt_property_terms       | InnoDB |      10 | Compact    |     261980 |             34 |      8929280 |               
| wbt_term_in_lang         | InnoDB |      10 | Compact    |   45801912 |             35 |   1633681408 |               
| wbt_text                 | InnoDB |      10 | Compact    |   17176409 |             60 |   1044381696 |               
| wbt_text_in_lang         | InnoDB |      10 | Compact    |   45872021 |             36 |   1680867328 |               
| wbt_type                 | InnoDB |      10 | Compact    |          2 |           8192 |        16384 |

Progress:

  • db1109
  • db1087
  • db1092
  • db1099:3318
  • db1101:3318
  • db1104
  • db1116:3318
  • db1124:3318
  • db1126
  • db2079
  • db2080
  • db2081
  • db2082
  • db2083
  • db2085:3318
  • db2086:3318
  • db2094:3318
  • db2100:3318
  • dbstore1005:3318
  • labsdb1009
  • labsdb1010
  • labsdb1011
  • labsdb1012

Also the new wikibase tables created in s4 need compression:

  • labsdb1012
  • labsdb1011
  • labsdb1010
  • labsdb1009
  • dbstore1004:3314
  • db2119
  • db2110
  • db2106
  • db2099
  • db2095
  • db2091
  • db2090
  • db2084
  • db2073
  • db1138
  • db1125
  • db1121
  • db1103
  • db1102
  • db1097
  • db1091
  • db1084
  • db1081

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Marostegui updated the task description. (Show Details)Feb 7 2020, 6:21 AM

Change 570790 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1126: Enable notifications

https://gerrit.wikimedia.org/r/570790

Change 570790 merged by Marostegui:
[operations/puppet@production] db1126: Enable notifications

https://gerrit.wikimedia.org/r/570790

Mentioned in SAL (#wikimedia-operations) [2020-02-07T06:25:02Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1126 T232446', diff saved to https://phabricator.wikimedia.org/P10330 and previous config saved to /var/cache/conftool/dbconfig/20200207-062502-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-02-07T06:38:31Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1126 T232446', diff saved to https://phabricator.wikimedia.org/P10333 and previous config saved to /var/cache/conftool/dbconfig/20200207-063831-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-02-07T07:30:26Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1126 T232446', diff saved to https://phabricator.wikimedia.org/P10334 and previous config saved to /var/cache/conftool/dbconfig/20200207-073026-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-02-07T07:45:12Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fullyy repool db1126 T232446', diff saved to https://phabricator.wikimedia.org/P10338 and previous config saved to /var/cache/conftool/dbconfig/20200207-074511-marostegui.json

Marostegui updated the task description. (Show Details)Feb 7 2020, 3:25 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-10T06:16:57Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10355 and previous config saved to /var/cache/conftool/dbconfig/20200210-061656-marostegui.json

Change 571114 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1091: Enable notifications

https://gerrit.wikimedia.org/r/571114

Change 571114 merged by Marostegui:
[operations/puppet@production] db1091: Enable notifications

https://gerrit.wikimedia.org/r/571114

Mentioned in SAL (#wikimedia-operations) [2020-02-10T06:37:16Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10358 and previous config saved to /var/cache/conftool/dbconfig/20200210-063716-marostegui.json

Change 571116 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1124: Disable notifications

https://gerrit.wikimedia.org/r/571116

Mentioned in SAL (#wikimedia-operations) [2020-02-10T06:39:09Z] <marostegui> Compress db1124:3318 - this will generate lag on s8 wiki replicas - T232446

Change 571116 merged by Marostegui:
[operations/puppet@production] db1124: Disable notifications

https://gerrit.wikimedia.org/r/571116

Mentioned in SAL (#wikimedia-operations) [2020-02-10T06:44:59Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10359 and previous config saved to /var/cache/conftool/dbconfig/20200210-064458-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-02-10T06:51:36Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10361 and previous config saved to /var/cache/conftool/dbconfig/20200210-065135-marostegui.json

Marostegui updated the task description. (Show Details)Feb 11 2020, 6:09 AM

Change 571417 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1124: Enable notifications

https://gerrit.wikimedia.org/r/571417

Change 571417 merged by Marostegui:
[operations/puppet@production] db1124: Enable notifications

https://gerrit.wikimedia.org/r/571417

Change 571864 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1087: Disable notifications

https://gerrit.wikimedia.org/r/571864

Change 571864 merged by Marostegui:
[operations/puppet@production] db1087: Disable notifications

https://gerrit.wikimedia.org/r/571864

Mentioned in SAL (#wikimedia-operations) [2020-02-13T06:35:35Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1087 for compression - T232446', diff saved to https://phabricator.wikimedia.org/P10401 and previous config saved to /var/cache/conftool/dbconfig/20200213-063535-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-02-13T06:36:42Z] <marostegui> Upgrade and compress db1087, this will generate lag on s8 on the wiki replicas - T232446

db1087 has all its tables compressed but wb_terms.
I am going to wait for labs to catch up a bit before moving them under codfw's sanitarium before starting the compression on that table, which I guess it will take around 4-5 days.

Change 573436 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1087: Enable notifications

https://gerrit.wikimedia.org/r/573436

Change 573436 merged by Marostegui:
[operations/puppet@production] db1087: Enable notifications

https://gerrit.wikimedia.org/r/573436

Mentioned in SAL (#wikimedia-operations) [2020-02-24T10:10:30Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1087 for compression and place db1101:3318 into vslow,dump - T232446', diff saved to https://phabricator.wikimedia.org/P10493 and previous config saved to /var/cache/conftool/dbconfig/20200224-101030-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-02-24T10:12:34Z] <marostegui> Stop db1087 and db2079 in sync - T232446

Mentioned in SAL (#wikimedia-operations) [2020-02-24T11:02:45Z] <marostegui> Move labsdb1009, labsdb1011 and labsdb1012 (labsdb1010 is currently delayed, will be done later) to replicate under codfw for a few days while we alter wb_terms on db1087 - T232446

Mentioned in SAL (#wikimedia-operations) [2020-02-24T14:42:33Z] <marostegui> Compress innodb on wb_terms on db1087 - T232446

Mentioned in SAL (#wikimedia-operations) [2020-02-25T06:02:05Z] <marostegui> Move labsdb1010 under db2094:3318 - T232446

db1087 finished and it is now catching up

Marostegui updated the task description. (Show Details)Feb 26 2020, 9:06 PM
jcrespo added a comment.EditedFeb 27 2020, 5:07 PM

I've given db1087 a bit of weight (weight 100, percentage: 20) so it can get some small amount of traffic, even if that means to depool it back down again for replication stop.

Mentioned in SAL (#wikimedia-operations) [2020-02-27T17:30:17Z] <jynus@cumin1001> dbctl commit (dc=all): 'Repool db1087 at 20% T232446', diff saved to https://phabricator.wikimedia.org/P10547 and previous config saved to /var/cache/conftool/dbconfig/20200227-173017-jynus.json

Change 575477 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1087: Enable notifications

https://gerrit.wikimedia.org/r/575477

Change 575477 merged by Marostegui:
[operations/puppet@production] db1087: Enable notifications

https://gerrit.wikimedia.org/r/575477

Due to the load issues we are experimenting on Wikidata I am not compressing the remaining hosts till the load has decreased or till we've done the DC switchover and eqiad becomes passive for a few weeks. Whatever comes first cc @Addshore

Per T249565: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views DBA should check whether wb_items_per_site table is compressed in each DB server.

It was compressed everywhere but on db1109, db1104, db1092. The new table that will be reimported is already compressed.

Mentioned in SAL (#wikimedia-operations) [2020-04-13T07:39:40Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1092 T232446', diff saved to https://phabricator.wikimedia.org/P10963 and previous config saved to /var/cache/conftool/dbconfig/20200413-073939-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-04-13T07:43:23Z] <marostegui> Compress db1092 T232446

Marostegui updated the task description. (Show Details)Apr 16 2020, 6:59 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-27T05:53:20Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1104 for defragmentation - T232446', diff saved to https://phabricator.wikimedia.org/P11039 and previous config saved to /var/cache/conftool/dbconfig/20200427-055320-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-04-27T05:56:50Z] <marostegui> Compress tables on db1104 - T232446

Marostegui changed the task status from Open to Stalled.May 1 2020, 5:12 AM
Marostegui updated the task description. (Show Details)
Marostegui moved this task from In progress to Blocked external/Not db team on the DBA board.

Only the primary database masters are pending.
To do those we'd either need a DC switchover or switch them individually.

Mentioned in SAL (#wikimedia-operations) [2020-05-01T13:28:04Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly warm up db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11107 and previous config saved to /var/cache/conftool/dbconfig/20200501-132804-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-05-01T13:47:07Z] <marostegui@cumin1001> dbctl commit (dc=all): 'More traffic to db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11108 and previous config saved to /var/cache/conftool/dbconfig/20200501-134707-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-05-01T14:06:06Z] <marostegui@cumin1001> dbctl commit (dc=all): 'More traffic to db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11109 and previous config saved to /var/cache/conftool/dbconfig/20200501-140603-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-05-01T14:23:54Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1104 - T232446', diff saved to https://phabricator.wikimedia.org/P11110 and previous config saved to /var/cache/conftool/dbconfig/20200501-142354-marostegui.json

db1138 is no longer s4 master, so this can be done as well there

Mentioned in SAL (#wikimedia-operations) [2020-05-29T10:02:37Z] <marostegui> Compress InnoDB on db1138 T232446

Marostegui updated the task description. (Show Details)Jun 1 2020, 8:19 AM
Marostegui changed the task status from Stalled to Open.Sep 7 2020, 5:07 AM
Marostegui moved this task from Blocked external/Not db team to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2020-09-07T08:03:39Z] <marostegui> Compress InnoDB on s8 eqiad master (db1109) - T232446

Marostegui closed this task as Resolved.Sep 14 2020, 5:13 AM
Marostegui updated the task description. (Show Details)

This is all done