deployment-fluorine02.deployment-prep.eqiad.wmflabs is a Jessie host. It should be replaced with a Buster host.
|Open||None||T197804 Puppet: forbid new Python2 code|
|Open||None||T218426 Upgrade various Cloud VPS Python 2 scripts to Python 3|
|Resolved||BUG REPORT||Bstorm||T218423 Add python 3 packages to openstack::clientpackages::common|
|Open||None||T232677 Remove support for Debian Jessie in Cloud Services|
|Duplicate||None||T236575 "deployment-prep" Cloud VPS project jessie deprecation|
|Resolved||None||T218729 Migrate deployment-prep away from Debian Jessie to Debian Stretch/Buster|
|Resolved||Majavah||T276419 Replace deployment-fluorine02 with a Buster host|
|Resolved||Majavah||T276421 Package udplog for Buster|
|Resolved||• hashar||T276422 Unarchive analytics/udplog repository|
I created deployment-mwlog01 as a Buster host. fluorine02 is a disk80 host but the log partition is fairly small:
/dev/mapper/vd-second--local--disk 60G 656M 56G 2% /srv
I made a 2G Cinder volume for that, given that the current logs are only about 650M it should be large enough. Also mounted it to /srv on the new VM.
I tried to apply the udp2log role to the new VM but it's failing because udplog package is not available for Buster yet. Since prod SRE is currently updating mwlog* hosts to buster (T224565: Migrate mwlog/udp2log servers to Buster) I'm hopeful that it will be fixed fairly soon. For now it does not have any roles to avoid puppet errors.
Now that udplog is available on Buster, next I'll try live hacking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/668338/, check that logs will be saved to deployment-mwlog01 and delivered to logstash-beta, and if it works get that patch properly merged
Mentioned in SAL (#wikimedia-releng) [2021-03-04T11:02:21Z] <Majavah> live hacking https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/668338/ on deployment-deploy01 to test new deployment-mwlog01 ref T276419
Live hacks reverted. I confirmed that mwlog01 receives events and logs are still flowing to logstash-beta.wmflabs.org. Scheduled the mediawiki-config patch for today's EU backport window.