But isn't it simper to just grep in the output of a single cookbook as opposed to grep the output of multiple tools?

serpens has been migrated to Bullseye, seaborgium to follow in a few days.

I think the reason the installation failed is because there is no entry in site.pp yet.

All insetup roles default to Puppet 7 these days (as does the kafka-main roler itself), so these should be installed with Puppet 7.

LGTM

LGTM

In T362681#9790329, @Jdforrester-WMF wrote:

In T362681#9782772, @Jdforrester-WMF wrote:

In T362681#9781064, @MoritzMuehlenhoff wrote:

I kicked off a build of the node20 image, it should hopefully appear in the registry soon.

Not there yet; did the build get stuck/break?

OK, after the weekend it's now shown up, and works great. Thank you!

@Dwisehaupt @Jgreen The kafka cert issued by the PKI is now getting deployed to /etc/fr-tech-kafka-client on cumin1002/cumin2002. Could you please sync it to fr-tech and test/deploy instead of the old cergen-issued cert? When this has been confirmed to work fine, we can add a systemd timer which sends a notification if the key renewal is forthcoming.

Redict is now packaged in Debian: https://tracker.debian.org/pkg/redict

Nice work!

I kicked off a build of the node20 image, it should hopefully appear in the registry soon.

In T279509#9776518, @hashar wrote:

Awesome, it is great to see repositories have been migrated. I am reopening this one though to uninstall the git-fat Debian package, it is based on Python 2 and that is blocking the migration out of Buster/Bullseye. We still have a bunch of reference to it in Puppet.

Ideally git-fat would be purged and then python2 removed, several Puppet roles have:

The remaining traces of git-fat have been removed from Puppet and git-fat uninstalled fleet-wide.

In T279509#9776518, @hashar wrote:

Awesome, it is great to see repositories have been migrated. I am reopening this one though to uninstall the git-fat Debian package, it is based on Python 2 and that is blocking the migration out of Buster/Bullseye. We still have a bunch of reference to it in Puppet.

Ideally git-fat would be purged and then python2 removed, several Puppet roles have:

# python 2 is required for git-fat
profile::base::remove_python2_on_bullseye: false

There is also all the bits in Archiva which were made to support git-fat then I know Archiva is no more actively maintained and it will be entirely phased out from our infrastructure (eg T358612 ). So I guess the supporting bits will be removed as Archiva is removed.

gerrit1003 is on Bullseye, not Buster. And Bullseye also provides OpenJDK17 in parallel to 11, so you can switch to 17 without any OS change, but a simple Hiera change in profile::java

+1 on becoming a CNA for Mediawiki core and extensions.

The two clusters (magru01 and magru02) are setup and initial VMs have been created already.

This certificate doesn't show up anywhere in certificate.manifests.d for cergen, though?

I've set the server back to "Active" in Netbox.

Looks good. Best to create it in group D

In T363452#9744332, @taavi wrote:

Those are running in containers, and the user does exist in the container:

In T361087#9745154, @jcrespo wrote:

In T361087#9744977, @cmooney wrote:

In T361087#9744384, @jcrespo wrote:

Booting failed (PXE):

PXELINUX 6.03 lwIP 20150819 Copyright (C) 1994-2014 H. Peter Anvin et al


Debian 12 (bookworm) amd64 (Wikimedia edition)

                                              boot: 
Loading debian-installer/amd64/linux... ok
Loading debian-installer/amd64/initrd.gz...
Boot failed: press a key to retry, or wait for reset...

Hmm. Not sure if we've seen this problem before. DHCP clearly worked as did the debian image download, but Linux failed to load for some reason.

@jcrespo the only difference was selecting bullseye rather than bookworm on the second attempt?

Yes. Check with @MoritzMuehlenhoff he did something to fix something, but not sure what, or if it applies here.

Will parsoidtest1001 be installed with Bullseye? scandium is currently running buster, but all the mediawiki manifests are compatible with bullseye (cloudweb already runs it), and so is the component/php74.

In T362981#9729490, @Jdforrester-WMF wrote:

In T362981#9729480, @MoritzMuehlenhoff wrote:

All the custom PHP extensions are already fully rebuilt for the component/php74 for bullseye! And being kept updated, e.g. the recent PHP security fixes were backported to both component/icu67 (buster) and component/php74 (bullseye).

In fact we're already using them on the cloudweb hosts (and the snapshot hosts using bullseye). So from that perspective the migration could happen any time, as long as we're okay with the remaining percentage of baremetal traffic running on Buster until it's fully shrunk to zero.

Hmm. Locally I get:

$ docker-pkg -c config.yaml --info build images/php --select '*/php7.4-cli:*'
…
2024-04-19 09:13:25 [docker-pkg-build] INFO - E: Unable to locate package php7.4-excimer
E: Couldn't find any package by glob 'php7.4-excimer'
E: Couldn't find any package by regex 'php7.4-excimer'

In T291916#9744868, @Dzahn wrote:

Ok, fair enough about the tracking task. But don't we still need some kind of task that someone can take to do the actual upgrade work? So all the subtasks without the tracking parent task?

In T291916#9742764, @Dzahn wrote:

@Muehlenhoff Where does deploy* (deployment_server role both prod and wmcs) fit in? Since we are still on buster there. But want bullseye deployment_servers in cloud VPS projects and production hasn't upgraded the role yet. A legit subtask for here?

In T284145#7218511, @Keegan wrote:

@jbond my utmost apologies for not replying to this earlier! These errors can be ignored, they will never go away AFAIK. These are the result of email addresses that used to be processed through OTRS/Znuny LTS, but were usurped by gsuite handling. When this occurs the old emails were invalidated and disabled from OTRS/Znuny LTS, but they cannot be deleted and so they will forever log errors unless/until the ability to delete unused emails is added as functionality.

In T362746#9740460, @ABran-WMF wrote:

@MoritzMuehlenhoff OK for me to reimage db1241 ?

Looks good. We can't disable DRBD on instance creation currently, simply add it as usual and then you can use the sre.ganeti.changedisk cookbook to switch to plain disks.

In T363125#9735541, @Andrew wrote:

I started out thinking that but I don't think it's so easy to switch to RO ldap. The ldap mw extension will still expect to be able to change things (email address, for example)

In T360439#9735349, @BTullis wrote:

I'll check to see if there is any code ready to deploy cfssl based certificates for nginx.

@MuratKaribay Can you please retry? I just tested changing the timezone to Almaty and Kostanai and it gave me a five hour offset.

In T363125#9734178, @taavi wrote:

B: Fishbowl wiki hosted on wikikube, accounts in ldap. This option could be a final state OR a temporary state on the way to the SUL option.
Con (long-term): Allowing r/w ldap access from wikitech/wikikube may continue to introduce surprising edge cases for product maintenance.

I don't think Wikitech will require r/w access after T359544: Disable SSH key management on Wikitech is done.

All the custom PHP extensions are already fully rebuilt for the component/php74 for bullseye! And being kept updated, e.g. the recent PHP security fixes were backported to both component/icu67 (buster) and component/php74 (bullseye).

That's not problem. We should just use the nodesource packages for this, we've been doing the same for "intermediate LTSes" before (e.g. node 16 or node 14) not covered by an intree Debian nodejs version. I'll work on this next week.

Looks good to me.

This is complete

In T362628#9722794, @akosiaris wrote:

• If a new image found to be okay, have some script/option/tool to promote the current staging image as the new main production images

Do we want this to be done forced, or should we rather rely on the Monday update automatically promoting ?

In T362852#9725716, @hashar wrote:

I am fine skipping this update.

Do you still want me to import the latest LTS release (like for non-security fixes) or shall we skip this update entirely?

Updated packages have been released and installed on all Wikimedia production systems. Cloud VPS instances are automatically updated via a nightly cron job, so should generally also be updated by now.

In T350179#9722934, @ssingh wrote:

@Papaul deserves a lot of love for fixing this persistent issue. The 21.x firmware (specifically, Network_Firmware_YK81Y_WN64_21.60.22.11_03) worked in the first attempt when reimaging cp1114. I think we can consider this closed given we have observed the fix on two hosts now.

Thanks, 1-800-Call-Papaul!

This isn't just limited to updating PHP, but also extends to the full OS stack underneath (libs used by PHP etc). When those currently get updated in Debian (via a security update or a point release), foe baremetal I check the context of how it's used, keep an eye on regressions and necessary restarts and usually roll out some canaries first. Typically there's always a few updates in flight at any point in time.

It's worth nothing that the stat hosts are on Bullseye/Debian 11, which being provides the following versions:

• Erlang 23.2.6
• Elixir 1.10.3
• cmake 3.18.4

In T362518#9719270, @Jdforrester-WMF wrote:

This has also broken building CI images. Will have to migrate them to bullseye immediately, I suppose.