It's most useful if effort is directed towards completing T237773, which will render this issue moot. In theory the DBAs are going to work on the first step of that sometime soon.

In T269004#6677107, @Andrew wrote:

It's most useful if effort is directed towards completing T237773, which will render this issue moot. In theory the DBAs are going to work on the first step of that sometime soon.

Unless T237773 realistically gets completed in January/Fabruary, we need to plan to upgrade the labweb* servers in their current config to Buster. We can't keep these two as outliers, since it would mean to continue to build all the PHP 7.2 packages for Stretch following future security updates. All the Puppet classes are adapted already, it's mostly a matter of reimaging at this point.

Change 650529 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudweb2001-dev: move from Stretch to Buster

https://gerrit.wikimedia.org/r/650529

Change 650529 merged by Andrew Bogott:
[operations/puppet@production] cloudweb2001-dev: move from Stretch to Buster

https://gerrit.wikimedia.org/r/650529

Change 650587 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add backend entry for labtesttoolsadmin

https://gerrit.wikimedia.org/r/650587

Change 650587 merged by Andrew Bogott:
[operations/puppet@production] Add backend entry for labtesttoolsadmin

https://gerrit.wikimedia.org/r/650587

Change 651204 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Labweb100[1-2] to Buster

https://gerrit.wikimedia.org/r/651204

Change 651204 merged by Andrew Bogott:
[operations/puppet@production] Labweb100[1-2] to Buster

https://gerrit.wikimedia.org/r/651204

I think I'm ready to do this whenever -- I upgraded our test host and it looks just fine.

@jijiki, my tests suggest that this upgrade will go smoothly. If you judge MW to be mostly ready for Buster then I'll go ahead and upgrade things this week; if not let's mark this stalled.

In T269004#6720008, @Andrew wrote:

@jijiki, my tests suggest that this upgrade will go smoothly. If you judge MW to be mostly ready for Buster then I'll go ahead and upgrade things this week; if not let's mark this stalled.

We have production systems running just fine on Buster (mwdebug1003 and mw1265) and the Puppet classes are adapted, this should be good to go. Feel free to ping me when you start, happy to have a look in case you run into anything anyway.

Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts:

labweb1002.wikimedia.org

The log can be found in /var/log/wmf-auto-reimage/202101061728_andrew_14527_labweb1002_wikimedia_org.log.

Change 654663 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Striker: replace requirement with libmariadb3

https://gerrit.wikimedia.org/r/654663

Change 654663 merged by Andrew Bogott:
[operations/puppet@production] Striker: replace requirement with libmariadb3

https://gerrit.wikimedia.org/r/654663

Completed auto-reimage of hosts:

['labweb1002.wikimedia.org']

and were ALL successful.

Change 654728 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: switch from python3.5 to python3.7

https://gerrit.wikimedia.org/r/654728

Change 654728 merged by Andrew Bogott:
[operations/puppet@production] Horizon: switch from python3.5 to python3.7

https://gerrit.wikimedia.org/r/654728

Change 654944 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[openstack/horizon/deploy@train-buster] Disable offline compression in train/buster

https://gerrit.wikimedia.org/r/654944

Change 654945 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: Disable offline compression in Train

https://gerrit.wikimedia.org/r/654945

Change 654945 merged by Andrew Bogott:
[operations/puppet@production] Horizon: Disable offline compression in Train

https://gerrit.wikimedia.org/r/654945

Change 654944 merged by Andrew Bogott:
[openstack/horizon/deploy@train-buster] Disable offline compression in train/buster

https://gerrit.wikimedia.org/r/654944

Script wmf-auto-reimage was launched by andrew on cumin1001.eqiad.wmnet for hosts:

['labweb1001.wikimedia.org']

The log can be found in /var/log/wmf-auto-reimage/202101081645_andrew_10780.log.

Completed auto-reimage of hosts:

['labweb1001.wikimedia.org']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2021-01-08T19:45:54Z] <andrew@deploy1001> Started deploy [striker/deploy@e4db843]: Striker deploy for T269004

Mentioned in SAL (#wikimedia-operations) [2021-01-08T19:48:05Z] <andrew@deploy1001> Finished deploy [striker/deploy@e4db843]: Striker deploy for T269004 (duration: 02m 11s)