In T429129#12030491, @jhathaway wrote:

@MoritzMuehlenhoff what do you think of the patch? Or do you want to find a way to retain the colors?

The patch sounds great, let's take that route. There's no need for colour output for the command to disable Puppet merges.

sounds good, merged

In T429129#12029810, @jhathaway wrote:

@MoritzMuehlenhoff what do you think of the patch? Or do you want to find a way to retain the colors?

In T427897#12028951, @jcrespo wrote:

@MoritzMuehlenhoff The test worked succesfully, thus, I've migrated the backups from cumin2002 to cumin2003. The backups are setup, but disabled on the old host. It would be nice to keep the host available for at least a week to observe a full cycle of backups happening with no errors, should a last minute error happened, monitoring would alert us. After that, I won't need cumin2002 for anything.

In T429446#12028163, @Jgiannelos wrote:

Its not used by CTT team so we don't need it. Can somebody check with the owners though ?

Can be removed

@jcrespo Cumin is now working on cumin2003, you can test backups.

Let's wait until Liberica is available and then go with 3.

Likewise osbpo and the Ubuntu mirror are moved by now, the last remaining part is the Debian mirror.

I opened a ticket at https://anonticket.torproject.org/ and mirrors.wikimedia.org has been removed from the download mirror rotation for Tails.

In T428495#12009735, @Scott_French wrote:

Thank you very much, @MoritzMuehlenhoff. Other than compatibility tests I'll run in the interim, we're unlikely to use the new packages for at least two weeks, so take your time.

In T427897#12013829, @jcrespo wrote:

@elukey Cumin is still not in a working state, see below:

@ayounsi For puppetserver2002 will need to be merged before the maintenance starts: https://gerrit.wikimedia.org/r/c/operations/dns/+/1300766 I'll take care of that.

In T428495#12007498, @Scott_French wrote:

@MoritzMuehlenhoff - How challenging would be to rebuild the forward-port of zookeeper (3.4.13) from bullseye that you prepared in T418915#11851872 for bookworm as well? That would be the last moving part of this intermediate in-place step to bookworm before stepping to trixie.

@brennen Sorry, I missed your reply until now! Yes, that is still needed, repos/sre sounds good to me.

@FCeratto-WMF Can you please import them to the "main" component of apt.wikimedia.org?

Routinator has been upgraded to 0.15.2 in eqiad and codfw.

Access to cn=wmf was granted via Wikimedia IDM.

In T421484#11834275, @jijiki wrote:

@Muehlenhoff following up on our discussion, and unless there are no objections, we could repurpose this machine for the ganeti cluster on eqiad. It was purchased on 2024-06-26.

In T421484#11834275, @jijiki wrote:

@Muehlenhoff following up on our discussion, and unless there are no objections, we could repurpose this machine for the ganeti cluster on eqiad. It was purchased on 2024-06-26.

All Ganeti nodes are back in service

In T421705#11974790, @Ladsgroup wrote:

In T421705#11955130, @MoritzMuehlenhoff wrote:

After migrating pc2022 to nftables we noticed a bump of connections tracked. The root cause is that on ferm connection tracking was disabled for 3306 being the source and dest port:

• /etc/ferm/conf.d/10_mariadb_internal.conf configures NO_TRACK for the entire 3306 port (via /etc/ferm/functions.conf)
• on nftables in /etc/nftables/notrack/10_mariadb_internal.nft we only set notrack for the destination port

This is no issue for pc2022, we have plenty of headroom on it. But we should resolve this before we move further mariadb nodes.

I've been debugging this and haven't made much progress but I have a couple of different ideas:

• Maybe we can add a separate rule on source port of 3306 in mariadb::firewall but I can't find a way to actually set the source port (and not the destination port).
• It could be that we are only allowing dport only in nftables and should allow sport too. Since the puppet code to set the notrack just takes the accept and replaces it with notrack (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1259874/1/modules/nftables/manifests/client.pp#158)
• A bit weirder: I could suggest to leave it as is. This is the source port being 3306 and mariadb itself shouldn't be connecting to stuff and I'd argue it was not the intention of the notrack to ignore those. The only case I can think of is when a replica connects to a master to get replication data and the overhead of those should be small on all dbs.

Just random late night thoughts.

I opened a ticket at https://anonticket.torproject.org/ and mirrors.wikimedia.org has been removed from the download mirror rotation for Tails.

In T427312#11961677, @Scott_French wrote:

Why didn't I run into this issue for php, php-defaults, or dh-php?

All done

I had a look at the debmonitor PKI setup and I agree that the CSR file on pki-root seems unused. since we have all of /etc/cfssl in Bacula we can also always restore it, so it seems safe to me to go ahead and just decom pki-root1001

After migrating pc2022 to nftables we noticed a bump of connections tracked. The root cause is that on ferm connection tracking was disabled for 3306 being the source and dest port:

All done