Page MenuHomePhabricator

move db1114 to s8
Closed, ResolvedPublic

Description

s8 has been overloading lately, another host would be helpful

Event Timeline

Marostegui moved this task from Triage to In progress on the DBA board.

Change 588831 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1114 to s8

https://gerrit.wikimedia.org/r/588831

Change 588831 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1114 to s8

https://gerrit.wikimedia.org/r/588831

Change 588832 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Allow reimage db1114

https://gerrit.wikimedia.org/r/588832

Mentioned in SAL (#wikimedia-operations) [2020-04-15T05:07:29Z] <marostegui> Remove db1114 from tendril - T250224

Change 588832 merged by Marostegui:
[operations/puppet@production] install_server: Allow reimage db1114

https://gerrit.wikimedia.org/r/588832

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1114.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202004150514_marostegui_23354.log.

Mentioned in SAL (#wikimedia-operations) [2020-04-15T05:21:24Z] <marostegui> Remove db1114 from tendril and zarcillo T250224

Completed auto-reimage of hosts:

['db1114.eqiad.wmnet']

and were ALL successful.

I am populating this host from s8's snapshot from yesterday (600GB) already transferred.

Change 588949 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1114: Enable notifications

https://gerrit.wikimedia.org/r/588949

Change 588949 merged by Marostegui:
[operations/puppet@production] db1114: Enable notifications

https://gerrit.wikimedia.org/r/588949

Mentioned in SAL (#wikimedia-operations) [2020-04-15T08:14:21Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1114 on s8 with low weight T250224', diff saved to https://phabricator.wikimedia.org/P10985 and previous config saved to /var/cache/conftool/dbconfig/20200415-081421-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-04-15T08:54:33Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase db1114's weight T250224', diff saved to https://phabricator.wikimedia.org/P10986 and previous config saved to /var/cache/conftool/dbconfig/20200415-085432-kormat.json

db1114 added to the Wikidata CPU dashboard

Mentioned in SAL (#wikimedia-operations) [2020-04-15T09:43:05Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase db1114's weight some more T250224', diff saved to https://phabricator.wikimedia.org/P10988 and previous config saved to /var/cache/conftool/dbconfig/20200415-094305-kormat.json

Mentioned in SAL (#wikimedia-operations) [2020-04-15T12:49:31Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase db1114's weight to 50% of target T250224', diff saved to https://phabricator.wikimedia.org/P10989 and previous config saved to /var/cache/conftool/dbconfig/20200415-124931-kormat.json

Mentioned in SAL (#wikimedia-operations) [2020-04-15T13:23:12Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase db1114's weight to 100% of target, and reduce db1104 slightly T250224', diff saved to https://phabricator.wikimedia.org/P10990 and previous config saved to /var/cache/conftool/dbconfig/20200415-132310-kormat.json

Kormat subscribed.

db1114 is now fully pooled with weight 300.

db1104 was also reduced from 350 to 300 to match.

Thank you so much for the help!
This should help s8 indeed. I expect also to repool db1092 tomorrow once the ALTERs are done.
Later today I will also decrease the weight for the RC slaves, which we increased yesterday night to help out in main traffic.