Page MenuHomePhabricator

Reorganize and clean existing pc1-pc5 sections
Closed, ResolvedPublic

Description

After the issues faced in Oct the existing parsercache sections are a bit messy in terms of hosts and replication.
Let's leave it "clean" before adding pc6 and pc7 T383234 T383235

Aiming for:

  • pc1
    • pc1011
    • pc2011
  • pc2
    • pc1012
    • pc2012
  • pc3
    • pc1013
    • pc2013
  • pc4
    • pc1014
    • pc2014
  • pc5
    • pc1015
    • pc2015

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change #1109702 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Update pc3 situation

https://gerrit.wikimedia.org/r/1109702

Change #1109702 merged by Marostegui:

[operations/puppet@production] mariadb: Update pc3 situation

https://gerrit.wikimedia.org/r/1109702

Mentioned in SAL (#wikimedia-operations) [2025-01-10T13:30:28Z] <marostegui> Move pc1013 to pc3 dbmaint eqiad - T383398

Change #1109703 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] pc2014: Move it to pc4

https://gerrit.wikimedia.org/r/1109703

Change #1109703 merged by Marostegui:

[operations/puppet@production] pc2014: Move it to pc4

https://gerrit.wikimedia.org/r/1109703

Mentioned in SAL (#wikimedia-operations) [2025-01-10T13:38:27Z] <marostegui> Move pc2013 to pc4 dbmaint codfw - T383398

Mentioned in SAL (#wikimedia-operations) [2025-01-13T10:05:55Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool pc3 T383398', diff saved to https://phabricator.wikimedia.org/P71989 and previous config saved to /var/cache/conftool/dbconfig/20250113-100554-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2025-01-13T10:07:44Z] <marostegui> Upgrade pc2013 pc1015 pc3 dbmaint eqiad codfw T383398

Change #1110729 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] pc1015: Move to pc5

https://gerrit.wikimedia.org/r/1110729

Change #1110729 merged by Marostegui:

[operations/puppet@production] pc1015: Move to pc5

https://gerrit.wikimedia.org/r/1110729

Change #1110731 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] pc1013: Make it pc3 master

https://gerrit.wikimedia.org/r/1110731

Change #1110731 merged by Marostegui:

[operations/puppet@production] pc1013: Make it pc3 master

https://gerrit.wikimedia.org/r/1110731

Mentioned in SAL (#wikimedia-operations) [2025-01-13T10:21:53Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Make pc1013 master in pc3 T383398', diff saved to https://phabricator.wikimedia.org/P71992 and previous config saved to /var/cache/conftool/dbconfig/20250113-102152-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2025-01-13T10:23:44Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Repool pc3 T383398', diff saved to https://phabricator.wikimedia.org/P71993 and previous config saved to /var/cache/conftool/dbconfig/20250113-102343-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2025-01-14T09:43:51Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool pc4 T383398', diff saved to https://phabricator.wikimedia.org/P72028 and previous config saved to /var/cache/conftool/dbconfig/20250114-094350-marostegui.json

Change #1111174 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Reorganize pc4

https://gerrit.wikimedia.org/r/1111174

Mentioned in SAL (#wikimedia-operations) [2025-01-14T09:53:21Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Promote pc2014 to codfw pc4 master dbmaint T383398', diff saved to https://phabricator.wikimedia.org/P72031 and previous config saved to /var/cache/conftool/dbconfig/20250114-095320-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2025-01-14T09:54:04Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Repool pc4 T383398', diff saved to https://phabricator.wikimedia.org/P72032 and previous config saved to /var/cache/conftool/dbconfig/20250114-095404-marostegui.json

pc2014 has been promoted to master in pc4.
pc1016 is still serving in pc4, which it shouldn't so the next move is to move pc1015 to master in pc5 and move pc1014 from pc5 to pc4.

Also next is: move pc2015 to pc5

Change #1111174 merged by Marostegui:

[operations/puppet@production] mariadb: Reorganize pc4

https://gerrit.wikimedia.org/r/1111174

Change #1111179 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Reorganize pc5

https://gerrit.wikimedia.org/r/1111179

Change #1111179 merged by Marostegui:

[operations/puppet@production] mariadb: Reorganize pc5

https://gerrit.wikimedia.org/r/1111179

pc2015 moved to pc5 to start warming up

Zarcillo has been reorganized even if the final picture isn't ready:

+----------+---------+
| instance | section |
+----------+---------+
| pc1011   | pc1     |
| pc2011   | pc1     |
| pc2012   | pc2     |
| pc1012   | pc2     |
| pc2013   | pc3     |
| pc1013   | pc3     |
| pc1014   | pc4     |
| pc2014   | pc4     |
| pc1015   | pc5     |
| pc2015   | pc5     |
| pc1016   | pc6     |
| pc2016   | pc6     |
| pc1017   | pc7     |
| pc2017   | pc7     |
+----------+---------+
14 rows in set (0.001 sec)

Change #1111183 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: Reorganize pc sections

https://gerrit.wikimedia.org/r/1111183

Change #1111183 merged by Marostegui:

[operations/puppet@production] site.pp: Reorganize pc sections

https://gerrit.wikimedia.org/r/1111183

Change #1111189 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Update pc4-master CNAME

https://gerrit.wikimedia.org/r/1111189

Change #1111189 merged by Marostegui:

[operations/dns@master] wmnet: Update pc4-master CNAME

https://gerrit.wikimedia.org/r/1111189

Mentioned in SAL (#wikimedia-operations) [2025-01-14T12:02:35Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool pc5 eqiad codfw dbmaint T383398', diff saved to https://phabricator.wikimedia.org/P72043 and previous config saved to /var/cache/conftool/dbconfig/20250114-120234-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2025-01-14T12:08:05Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Repool pc5 T383398', diff saved to https://phabricator.wikimedia.org/P72044 and previous config saved to /var/cache/conftool/dbconfig/20250114-120804-marostegui.json

pc1015 promoted to eqiad pc5 master. Now pc1014 can be moved to pc4 to start getting warmed up.

Change #1111205 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] pc1014: Move it to pc4

https://gerrit.wikimedia.org/r/1111205

Change #1111205 merged by Marostegui:

[operations/puppet@production] pc1014: Move it to pc4

https://gerrit.wikimedia.org/r/1111205

Change #1111207 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: Reorganize pc4 and pc5

https://gerrit.wikimedia.org/r/1111207

Change #1111207 merged by Marostegui:

[operations/puppet@production] site.pp: Reorganize pc4 and pc5

https://gerrit.wikimedia.org/r/1111207

Mentioned in SAL (#wikimedia-operations) [2025-01-16T09:44:40Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool pc5 eqiad and codfw dbmaint T383398', diff saved to https://phabricator.wikimedia.org/P72104 and previous config saved to /var/cache/conftool/dbconfig/20250116-094439-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2025-01-16T09:54:32Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Promote pc2015 to pc5 codfw master dbmaint and enable pc5 back in eqiad and codfw T383398', diff saved to https://phabricator.wikimedia.org/P72107 and previous config saved to /var/cache/conftool/dbconfig/20250116-095431-marostegui.json

Change #1111931 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: Reorganize pc5

https://gerrit.wikimedia.org/r/1111931

pc5 has been set up with pc1015 and pc2015 as masters.

Change #1111931 merged by Marostegui:

[operations/puppet@production] site.pp: Reorganize pc5

https://gerrit.wikimedia.org/r/1111931

Mentioned in SAL (#wikimedia-operations) [2025-01-16T12:31:30Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool pc4 eqiad and codfw dbmaint T383398', diff saved to https://phabricator.wikimedia.org/P72117 and previous config saved to /var/cache/conftool/dbconfig/20250116-123129-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2025-01-16T12:36:57Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Promote pc1014 to pc4 eqiad master dbmaint and enable pc4 back in eqiad and codfw T383398', diff saved to https://phabricator.wikimedia.org/P72118 and previous config saved to /var/cache/conftool/dbconfig/20250116-123656-marostegui.json

Change #1111967 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Update pc5-master CNAME

https://gerrit.wikimedia.org/r/1111967

Change #1111967 merged by Marostegui:

[operations/dns@master] wmnet: Update pc5-master CNAME

https://gerrit.wikimedia.org/r/1111967