Page MenuHomePhabricator

Reload metawiki.content table on s7 hosts
Closed, ResolvedPublic

Description

There were two identical crashes on the metawiki.content table on 10.4 hosts (T261869). Let's rebuild that table on codfw 10.4 hosts:

  • db2086:3317
  • db2087:3317
  • db2120 (this was was already fully rebuilt at T261869)
  • db2121
  • db2122

I would also do the same for 10.4 hosts on eqiad, as it is a small table, let's do it preventively

  • dbstore1003:3317
  • db1127
  • db1101:3317
  • db1098:3317
  • db1094
  • db1090:3317

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, Sep 3, 7:07 AM
Marostegui triaged this task as High priority.Thu, Sep 3, 7:07 AM
Marostegui moved this task from Triage to In progress on the DBA board.

db2121 done. Analyze table also looks good after the reload.

Mentioned in SAL (#wikimedia-operations) [2020-09-03T07:48:28Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2086:3317 T261917', diff saved to https://phabricator.wikimedia.org/P12450 and previous config saved to /var/cache/conftool/dbconfig/20200903-074827-marostegui.json

db1094 done - analyze looking good. Host upgraded too.

db2086:3317 done - analyze looking good. Host upgraded too.

db1127 done - analyze looking good. Host upgraded too.

db1101 done - analyze looking good. Host upgraded too.

Marostegui updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2020-09-03T08:29:59Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2122 T261917', diff saved to https://phabricator.wikimedia.org/P12461 and previous config saved to /var/cache/conftool/dbconfig/20200903-082956-marostegui.json

db2122 done - analyze looking good. Host upgraded too.

dbstore1003:3317 done - analyze looking good.

db1090:3317 done - analyze looking good. Host upgraded too.

db1098:3317 done - analyze looking good. Host upgraded too.

Mentioned in SAL (#wikimedia-operations) [2020-09-03T09:20:29Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2087:3316 db2087:3317 T261917', diff saved to https://phabricator.wikimedia.org/P12472 and previous config saved to /var/cache/conftool/dbconfig/20200903-092028-marostegui.json

db2087:3317 done - analyze looking good. Host upgraded too.

Change 623981 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software@master] mariadb: Add content to the list of tables to check with db-compare

https://gerrit.wikimedia.org/r/623981

Marostegui closed this task as Resolved.Thu, Sep 3, 9:55 AM

This is all done and hosts repooled. Also, the hosts that needed a reboot for the kernel upgrade for T261389, were rebooted.

Change 623981 merged by jenkins-bot:
[operations/software@master] mariadb: Add content to the list of tables to check with db-compare

https://gerrit.wikimedia.org/r/623981