As discussed with Amir in the meeting - let's see if adding one host to x1 can help with connection spikes a bit.
I will use db1183 (T394507) which was pending to be decommissioned, but we can use here for a bit and get some more ideas if this can prevent the circuit breaker to kick in.
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| db1183: Enable notifications | operations/puppet | production | +0 -1 | |
| mariadb: Move db1183 to x1 | operations/puppet | production | +3 -4 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | PRODUCTION ERROR | Ladsgroup | T393513 Fatal exception of type "DBUnexpectedError: Database servers in extension1 are overloaded." affecting page views | ||
| Resolved | Marostegui | T394661 Move db1183 to x1 |
Event Timeline
Comment Actions
Change #1147759 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] mariadb: Move db1183 to x1
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2025-05-19T12:43:03Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1179 T394661', diff saved to https://phabricator.wikimedia.org/P76301 and previous config saved to /var/cache/conftool/dbconfig/20250519-124302-marostegui.json
Comment Actions
Change #1147759 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1183 to x1
Comment Actions
Start pool of db1179 gradually with 4 steps - Pool db1179.eqiad.wmnet in after cloning - marostegui@cumin1002
Comment Actions
Completed pool of db1179 gradually with 4 steps - Pool db1179.eqiad.wmnet in after cloning - marostegui@cumin1002
Comment Actions
Change #1147920 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1183: Enable notifications
Comment Actions
Change #1147920 merged by Marostegui:
[operations/puppet@production] db1183: Enable notifications