In T284145#7218511, @Keegan wrote:

@jbond my utmost apologies for not replying to this earlier! These errors can be ignored, they will never go away AFAIK. These are the result of email addresses that used to be processed through OTRS/Znuny LTS, but were usurped by gsuite handling. When this occurs the old emails were invalidated and disabled from OTRS/Znuny LTS, but they cannot be deleted and so they will forever log errors unless/until the ability to delete unused emails is added as functionality.

In T362746#9740460, @ABran-WMF wrote:

@MoritzMuehlenhoff OK for me to reimage db1241 ?

Looks good. We can't disable DRBD on instance creation currently, simply add it as usual and then you can use the sre.ganeti.changedisk cookbook to switch to plain disks.

In T363125#9735541, @Andrew wrote:

I started out thinking that but I don't think it's so easy to switch to RO ldap. The ldap mw extension will still expect to be able to change things (email address, for example)

In T360439#9735349, @BTullis wrote:

I'll check to see if there is any code ready to deploy cfssl based certificates for nginx.

@MuratKaribay Can you please retry? I just tested changing the timezone to Almaty and Kostanai and it gave me a five hour offset.

In T363125#9734178, @taavi wrote:

B: Fishbowl wiki hosted on wikikube, accounts in ldap. This option could be a final state OR a temporary state on the way to the SUL option.
Con (long-term): Allowing r/w ldap access from wikitech/wikikube may continue to introduce surprising edge cases for product maintenance.

I don't think Wikitech will require r/w access after T359544: Disable SSH key management on Wikitech is done.

All the custom PHP extensions are already fully rebuilt for the component/php74 for bullseye! And being kept updated, e.g. the recent PHP security fixes were backported to both component/icu67 (buster) and component/php74 (bullseye).

That's not problem. We should just use the nodesource packages for this, we've been doing the same for "intermediate LTSes" before (e.g. node 16 or node 14) not covered by an intree Debian nodejs version. I'll work on this next week.

Looks good to me.

This is complete

In T362628#9722794, @akosiaris wrote:

• If a new image found to be okay, have some script/option/tool to promote the current staging image as the new main production images

Do we want this to be done forced, or should we rather rely on the Monday update automatically promoting ?

In T362852#9725716, @hashar wrote:

I am fine skipping this update.

Do you still want me to import the latest LTS release (like for non-security fixes) or shall we skip this update entirely?

Updated packages have been released and installed on all Wikimedia production systems. Cloud VPS instances are automatically updated via a nightly cron job, so should generally also be updated by now.

In T350179#9722934, @ssingh wrote:

@Papaul deserves a lot of love for fixing this persistent issue. The 21.x firmware (specifically, Network_Firmware_YK81Y_WN64_21.60.22.11_03) worked in the first attempt when reimaging cp1114. I think we can consider this closed given we have observed the fix on two hosts now.

Thanks, 1-800-Call-Papaul!

This isn't just limited to updating PHP, but also extends to the full OS stack underneath (libs used by PHP etc). When those currently get updated in Debian (via a security update or a point release), foe baremetal I check the context of how it's used, keep an eye on regressions and necessary restarts and usually roll out some canaries first. Typically there's always a few updates in flight at any point in time.

It's worth nothing that the stat hosts are on Bullseye/Debian 11, which being provides the following versions:

• Erlang 23.2.6
• Elixir 1.10.3
• cmake 3.18.4

In T362518#9719270, @Jdforrester-WMF wrote:

This has also broken building CI images. Will have to migrate them to bullseye immediately, I suppose.

Do we expect a new access request to add him to "wmf" LDAP group, WMF-NDA in Phabricator etc?

Can you try accessing https://idp.wikimedia.org/logout and then retrying a login to https://idp.wikimedia.org/? You might have still had an SSO session with your previous memberships. I just had a look at the LDAP replicas and your cn=wmf membership is present there.

You don't need 4G of RAM, 2 should be perfectly fine.

In T362518#9712748, @hashar wrote:

integration/config/operations-puppet (shellcheck)

The image is based on Buster since the Puppet master in production use / used Debian Buster. That is to ensure we use / used the same version of ruby as in production. I guess that image can be recreated targeting Bullseye / Bookworm.

In T360795#9707776, @Andrew wrote:

I'd prefer that it go on its own ganeti VM, as I'm trying to pare down on the total number if weird things that run on cloudservices.

@AndyRussG: I've enabled your access. You should already be able to log into stat1010.eqiad.wmnet. The other servers will follow within the next 30 minutes after Puppet has deployed the change everywhere.

@SToyofuku-WMF : I've enabled your access. You should already be able to log into stat1010.eqiad.wmnet. The other servers will follow within the next 30 minutes after Puppet (our configuration managent tool) has deployed the change everywhere. Resolving. If you run into any issues, please reopen the task!

The labtest LDAP currently runs on cloudservices2004/2005-dev. I think we have two options: Either a separate Ganeti VM for the labtest/Bitu instance or we simply install it on cloudservices2004/2005, given that Bitu can simply be installed via a deb and only pulls in some moderate dependencies.

Ah, right. Forgot about the local LDAP data base on it.

What is the intended use case here? We already have a staging instance for Bitu

In T360414#9702570, @andrea.denisse wrote:

I've documented the migration process on Wikitech: https://wikitech.wikimedia.org/wiki/Cergen#Migrating_to_CFSSL

Please take a look at it if you can.

In T362113#9702093, @SToyofuku-WMF wrote:

@MoritzMuehlenhoff apologies, I thought analytics-privatedata-users was the specific level, but I can see I was looking at the wrong table. I'm specifically looking for access to superset/hue, so I shouldn't need Kerberos access - I believe only the first two rows of the table you linked should be sufficient

In T360779#9700772, @Jgreen wrote:

Sounds fine to me. I looked at the puppet code and if I understand correctly, cfssl::cert will automatically generate a new certificate 10 (default) days before expiration. Hopefully we can figure out how to make puppet send a notification when the new cert is available, so we can fetch and deploy it in frack.

In T360784#9661505, @hashar wrote:

@joanna_borun I am mostly likely going to need assistance from people knowing Cumin. Notably I don't know which Debian distribution should be favored (Bullseye, Bookworm?) , I am merely a end user of it and maybe some amendment will be needed in Puppet.

LGTM

grafana-labs.wikimedia.org is just a redirect to https://wikitech.wikimedia.org/wiki/News/2023_Cloud_VPS_metrics_changes it's been nine months so we're probably also good to just remove it entirely now?

In T360636#9698325, @akosiaris wrote:

I 'll finish parsoid and testreduce in T359387

We've disabled U2F, so this no longer applies

We're happy with the current logging for CAS.

@SToyofuku-WMF @NBaca-WMF Can you please clarify which specific type of access you want/need:

@AndyRussG : I've enabled your LDAP access with the wmf group, you should now be able to access the services listed under https://wikitech.wikimedia.org/wiki/SRE/LDAP/Groups#wmf_group

@odimitrijevic @Ahoelzl @WDoranWMF This needs your approval.