Description
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Marostegui | T383398 Reorganize and clean existing pc1-pc5 sections | |||
| Resolved | Ladsgroup | T373037 Make ParserCache more like a ring | |||
| Resolved | Ladsgroup | T383137 Allow full depool of pc sections in dbctl | |||
| Resolved | Marostegui | T383234 Introduce pc6 and move one spare per dc to it | |||
| Resolved | Marostegui | T383235 Introduce pc7 and move one spare per dc to it |
Event Timeline
Change #1109702 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] mariadb: Update pc3 situation
Change #1109702 merged by Marostegui:
[operations/puppet@production] mariadb: Update pc3 situation
Mentioned in SAL (#wikimedia-operations) [2025-01-10T13:30:28Z] <marostegui> Move pc1013 to pc3 dbmaint eqiad - T383398
Change #1109703 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] pc2014: Move it to pc4
Change #1109703 merged by Marostegui:
[operations/puppet@production] pc2014: Move it to pc4
Mentioned in SAL (#wikimedia-operations) [2025-01-10T13:38:27Z] <marostegui> Move pc2013 to pc4 dbmaint codfw - T383398
Mentioned in SAL (#wikimedia-operations) [2025-01-13T10:05:55Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool pc3 T383398', diff saved to https://phabricator.wikimedia.org/P71989 and previous config saved to /var/cache/conftool/dbconfig/20250113-100554-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2025-01-13T10:07:44Z] <marostegui> Upgrade pc2013 pc1015 pc3 dbmaint eqiad codfw T383398
Change #1110729 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] pc1015: Move to pc5
Change #1110729 merged by Marostegui:
[operations/puppet@production] pc1015: Move to pc5
Change #1110731 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] pc1013: Make it pc3 master
Change #1110731 merged by Marostegui:
[operations/puppet@production] pc1013: Make it pc3 master
Mentioned in SAL (#wikimedia-operations) [2025-01-13T10:21:53Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Make pc1013 master in pc3 T383398', diff saved to https://phabricator.wikimedia.org/P71992 and previous config saved to /var/cache/conftool/dbconfig/20250113-102152-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2025-01-13T10:23:44Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Repool pc3 T383398', diff saved to https://phabricator.wikimedia.org/P71993 and previous config saved to /var/cache/conftool/dbconfig/20250113-102343-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2025-01-14T09:43:51Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool pc4 T383398', diff saved to https://phabricator.wikimedia.org/P72028 and previous config saved to /var/cache/conftool/dbconfig/20250114-094350-marostegui.json
Change #1111174 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] mariadb: Reorganize pc4
Mentioned in SAL (#wikimedia-operations) [2025-01-14T09:53:21Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Promote pc2014 to codfw pc4 master dbmaint T383398', diff saved to https://phabricator.wikimedia.org/P72031 and previous config saved to /var/cache/conftool/dbconfig/20250114-095320-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2025-01-14T09:54:04Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Repool pc4 T383398', diff saved to https://phabricator.wikimedia.org/P72032 and previous config saved to /var/cache/conftool/dbconfig/20250114-095404-marostegui.json
pc2014 has been promoted to master in pc4.
pc1016 is still serving in pc4, which it shouldn't so the next move is to move pc1015 to master in pc5 and move pc1014 from pc5 to pc4.
Also next is: move pc2015 to pc5
Change #1111174 merged by Marostegui:
[operations/puppet@production] mariadb: Reorganize pc4
Change #1111179 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] mariadb: Reorganize pc5
Change #1111179 merged by Marostegui:
[operations/puppet@production] mariadb: Reorganize pc5
Zarcillo has been reorganized even if the final picture isn't ready:
+----------+---------+ | instance | section | +----------+---------+ | pc1011 | pc1 | | pc2011 | pc1 | | pc2012 | pc2 | | pc1012 | pc2 | | pc2013 | pc3 | | pc1013 | pc3 | | pc1014 | pc4 | | pc2014 | pc4 | | pc1015 | pc5 | | pc2015 | pc5 | | pc1016 | pc6 | | pc2016 | pc6 | | pc1017 | pc7 | | pc2017 | pc7 | +----------+---------+ 14 rows in set (0.001 sec)
Change #1111183 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] site.pp: Reorganize pc sections
Change #1111183 merged by Marostegui:
[operations/puppet@production] site.pp: Reorganize pc sections
Change #1111189 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/dns@master] wmnet: Update pc4-master CNAME
Change #1111189 merged by Marostegui:
[operations/dns@master] wmnet: Update pc4-master CNAME
Mentioned in SAL (#wikimedia-operations) [2025-01-14T12:02:35Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool pc5 eqiad codfw dbmaint T383398', diff saved to https://phabricator.wikimedia.org/P72043 and previous config saved to /var/cache/conftool/dbconfig/20250114-120234-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2025-01-14T12:08:05Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Repool pc5 T383398', diff saved to https://phabricator.wikimedia.org/P72044 and previous config saved to /var/cache/conftool/dbconfig/20250114-120804-marostegui.json
pc1015 promoted to eqiad pc5 master. Now pc1014 can be moved to pc4 to start getting warmed up.
Change #1111205 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] pc1014: Move it to pc4
Change #1111205 merged by Marostegui:
[operations/puppet@production] pc1014: Move it to pc4
Change #1111207 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] site.pp: Reorganize pc4 and pc5
Change #1111207 merged by Marostegui:
[operations/puppet@production] site.pp: Reorganize pc4 and pc5
Mentioned in SAL (#wikimedia-operations) [2025-01-16T09:44:40Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool pc5 eqiad and codfw dbmaint T383398', diff saved to https://phabricator.wikimedia.org/P72104 and previous config saved to /var/cache/conftool/dbconfig/20250116-094439-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2025-01-16T09:54:32Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Promote pc2015 to pc5 codfw master dbmaint and enable pc5 back in eqiad and codfw T383398', diff saved to https://phabricator.wikimedia.org/P72107 and previous config saved to /var/cache/conftool/dbconfig/20250116-095431-marostegui.json
Change #1111931 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] site.pp: Reorganize pc5
Change #1111931 merged by Marostegui:
[operations/puppet@production] site.pp: Reorganize pc5
Mentioned in SAL (#wikimedia-operations) [2025-01-16T12:31:30Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool pc4 eqiad and codfw dbmaint T383398', diff saved to https://phabricator.wikimedia.org/P72117 and previous config saved to /var/cache/conftool/dbconfig/20250116-123129-marostegui.json
Mentioned in SAL (#wikimedia-operations) [2025-01-16T12:36:57Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Promote pc1014 to pc4 eqiad master dbmaint and enable pc4 back in eqiad and codfw T383398', diff saved to https://phabricator.wikimedia.org/P72118 and previous config saved to /var/cache/conftool/dbconfig/20250116-123656-marostegui.json
Change #1111967 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/dns@master] wmnet: Update pc5-master CNAME
Change #1111967 merged by Marostegui:
[operations/dns@master] wmnet: Update pc5-master CNAME