Page MenuHomePhabricator

Upgrade parsercache to Buster and MariaDB 10.4
Closed, ResolvedPublic

Description

pc2 have been running Buster and 10.4 for a few weeks now and the spare hosts (pc1010 and pc2010) have just been upgraded.
Let's upgrade pc1 and pc3 also so we can have the full pc infra running Buster and 10.4

This involves MW depooling.
My suggestion would be to upgrade:

pc1: first codfw, then eqiad
pc3: first codfw, then eqiad

Event Timeline

Marostegui triaged this task as Medium priority.
Marostegui moved this task from Triage to Pending comment on the DBA board.
Marostegui added a subscriber: Kormat.

Assigning to @Kormat as I think it can be a good exercise as it involves MW deployments.

jcrespo renamed this task from Upgrade parsercache to Buster and 10.4 to Upgrade parsercache to Buster and MariaDB 10.4.May 11 2020, 5:23 PM
jcrespo added a subscriber: jcrespo.

updating title to prevent confusion with Buster 10.4 release.

Correction: pc2010 was not upgraded.

Change 595901 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Allow reimage of pc2010

https://gerrit.wikimedia.org/r/595901

Change 595901 merged by Kormat:
[operations/puppet@production] install_server: Allow reimage of pc2010

https://gerrit.wikimedia.org/r/595901

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['pc2010.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202005121059_kormat_190456.log.

Completed auto-reimage of hosts:

['pc2010.codfw.wmnet']

and were ALL successful.

Change 595922 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Allow reimage of pc2007

https://gerrit.wikimedia.org/r/595922

Change 595922 merged by Kormat:
[operations/puppet@production] install_server: Allow reimage of pc2007

https://gerrit.wikimedia.org/r/595922

Change 595928 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch all remaining pc* hosts to buster.

https://gerrit.wikimedia.org/r/595928

Change 595928 merged by Kormat:
[operations/puppet@production] install_server: Switch all remaining pc* hosts to buster.

https://gerrit.wikimedia.org/r/595928

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['pc2007.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202005121339_kormat_212939.log.

Change 595945 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Fix netboot.cfg entry for pc2007

https://gerrit.wikimedia.org/r/595945

Change 595945 merged by Kormat:
[operations/puppet@production] install_server: Fix netboot.cfg entry for pc2007

https://gerrit.wikimedia.org/r/595945

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['pc2007.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202005121447_kormat_222215.log.

Completed auto-reimage of hosts:

['pc2007.codfw.wmnet']

Of which those FAILED:

['pc2007.codfw.wmnet']

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['pc2007.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202005121449_kormat_222380.log.

Completed auto-reimage of hosts:

['pc2007.codfw.wmnet']

and were ALL successful.

Change 595961 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] Revert "install_server: Allow reimage of pc2010"

https://gerrit.wikimedia.org/r/595961

Change 595961 merged by Kormat:
[operations/puppet@production] Revert "install_server: Allow reimage of pc2010"

https://gerrit.wikimedia.org/r/595961

Change 596146 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Disallow reimaging of pc2007

https://gerrit.wikimedia.org/r/596146

Change 596146 merged by Kormat:
[operations/puppet@production] install_server: Disallow reimaging of pc2007

https://gerrit.wikimedia.org/r/596146

Change 596152 had a related patch set uploaded (by Kormat; owner: Stephen Shirley):
[operations/mediawiki-config@master] db-eqiad.php: Pool pc1010 for pc1

https://gerrit.wikimedia.org/r/596152

Change 596152 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Pool pc1010 for pc1

https://gerrit.wikimedia.org/r/596152

Mentioned in SAL (#wikimedia-operations) [2020-05-13T08:52:09Z] <kormat@deploy1001> Synchronized wmf-config/db-eqiad.php: Pool pc1010 as pc1 master T252182 (duration: 01m 17s)

Change 596158 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Allow reimage of pc1007

https://gerrit.wikimedia.org/r/596158

Change 596158 merged by Kormat:
[operations/puppet@production] install_server: Allow reimage of pc1007

https://gerrit.wikimedia.org/r/596158

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['pc1007.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202005130924_kormat_89303.log.

Completed auto-reimage of hosts:

['pc1007.eqiad.wmnet']

and were ALL successful.

Change 596173 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/mediawiki-config@master] Revert "db-eqiad.php: Pool pc1010 for pc1"

https://gerrit.wikimedia.org/r/596173

Change 596173 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "db-eqiad.php: Pool pc1010 for pc1"

https://gerrit.wikimedia.org/r/596173

Mentioned in SAL (#wikimedia-operations) [2020-05-13T10:09:03Z] <kormat@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool pc1007 as pc1 master T252182 (duration: 01m 05s)

Change 596176 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] Revert "install_server: Allow reimage of pc1007"

https://gerrit.wikimedia.org/r/596176

Change 596176 merged by Kormat:
[operations/puppet@production] Revert "install_server: Allow reimage of pc1007"

https://gerrit.wikimedia.org/r/596176

Change 602014 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Allow reimage of pc2009

https://gerrit.wikimedia.org/r/602014

Change 602019 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/mediawiki-config@master] db-codfw.php: Replace pc2009 with pc2010 while reimaging

https://gerrit.wikimedia.org/r/602019

Change 602014 merged by Kormat:
[operations/puppet@production] install_server: Allow reimage of pc2009

https://gerrit.wikimedia.org/r/602014

Change 602019 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Replace pc2009 with pc2010 while reimaging

https://gerrit.wikimedia.org/r/602019

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['pc2009.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006030906_kormat_72120.log.

Completed auto-reimage of hosts:

['pc2009.codfw.wmnet']

and were ALL successful.

Change 602038 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Enable notifications for db2009

https://gerrit.wikimedia.org/r/602038

Change 602038 merged by Kormat:
[operations/puppet@production] mariadb: Enable notifications for db2009

https://gerrit.wikimedia.org/r/602038

Mentioned in SAL (#wikimedia-operations) [2020-06-03T13:13:36Z] <kormat@deploy1001> Synchronized wmf-config/db-codfw.php: Put pc2009 back into pc3 after reimaging T252182 (duration: 01m 05s)

Change 602081 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Allow reimage of pc1009

https://gerrit.wikimedia.org/r/602081

Mentioned in SAL (#wikimedia-operations) [2020-06-03T13:44:07Z] <kormat> reimaging pc1007 to buster, wish me luck T252182

Change 602081 merged by Kormat:
[operations/puppet@production] mariadb: Allow reimage of pc1009

https://gerrit.wikimedia.org/r/602081

Mentioned in SAL (#wikimedia-operations) [2020-06-03T13:47:34Z] <kormat> reimaging *pc1009 (promise) to buster T252182

Mentioned in SAL (#wikimedia-operations) [2020-06-03T13:59:56Z] <kormat@deploy1001> Synchronized wmf-config/db-eqiad.php: Replace pc1009 with pc1010 reimaging T252182 (duration: 01m 06s)

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['pc1009.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006031405_kormat_68270.log.

Completed auto-reimage of hosts:

['pc1009.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-06-03T14:50:40Z] <kormat@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool pc1009 in pc3 after reimaging T252182 (duration: 01m 06s)

pc3 is done now too. Let us never speak of this again.