s8 has been overloading lately, another host would be helpful
Description
Details
Related Objects
- Mentioned Here
- P10990 dbctl commit (dc=all): 'Increase db1114's weight to 100% of target, and reduce db1104 slightly T250224'
P10989 dbctl commit (dc=all): 'Increase db1114's weight to 50% of target T250224'
P10988 dbctl commit (dc=all): 'Increase db1114's weight some more T250224'
P10986 dbctl commit (dc=all): 'Increase db1114's weight T250224'
P10985 dbctl commit (dc=all): 'Pool db1114 on s8 with low weight T250224'
Event Timeline
Change 588831 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1114 to s8
Change 588831 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1114 to s8
Change 588832 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Allow reimage db1114
Mentioned in SAL (#wikimedia-operations) [2020-04-15T05:07:29Z] <marostegui> Remove db1114 from tendril - T250224
Change 588832 merged by Marostegui:
[operations/puppet@production] install_server: Allow reimage db1114
Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:
['db1114.eqiad.wmnet']
The log can be found in /var/log/wmf-auto-reimage/202004150514_marostegui_23354.log.
Mentioned in SAL (#wikimedia-operations) [2020-04-15T05:21:24Z] <marostegui> Remove db1114 from tendril and zarcillo T250224
I am populating this host from s8's snapshot from yesterday (600GB) already transferred.
Change 588949 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1114: Enable notifications
Change 588949 merged by Marostegui:
[operations/puppet@production] db1114: Enable notifications
Mentioned in SAL (#wikimedia-operations) [2020-04-15T08:14:21Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1114 on s8 with low weight T250224', diff saved to https://phabricator.wikimedia.org/P10985 and previous config saved to /var/cache/conftool/dbconfig/20200415-081421-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2020-04-15T08:54:33Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase db1114's weight T250224', diff saved to https://phabricator.wikimedia.org/P10986 and previous config saved to /var/cache/conftool/dbconfig/20200415-085432-kormat.json
Mentioned in SAL (#wikimedia-operations) [2020-04-15T09:43:05Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase db1114's weight some more T250224', diff saved to https://phabricator.wikimedia.org/P10988 and previous config saved to /var/cache/conftool/dbconfig/20200415-094305-kormat.json
Mentioned in SAL (#wikimedia-operations) [2020-04-15T12:49:31Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase db1114's weight to 50% of target T250224', diff saved to https://phabricator.wikimedia.org/P10989 and previous config saved to /var/cache/conftool/dbconfig/20200415-124931-kormat.json
Mentioned in SAL (#wikimedia-operations) [2020-04-15T13:23:12Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase db1114's weight to 100% of target, and reduce db1104 slightly T250224', diff saved to https://phabricator.wikimedia.org/P10990 and previous config saved to /var/cache/conftool/dbconfig/20200415-132310-kormat.json
db1114 is now fully pooled with weight 300.
db1104 was also reduced from 350 to 300 to match.
Thank you so much for the help!
This should help s8 indeed. I expect also to repool db1092 tomorrow once the ALTERs are done.
Later today I will also decrease the weight for the RC slaves, which we increased yesterday night to help out in main traffic.