In T40946#12213070, @Ladsgroup wrote:

I know I'm pasting this in a ticket that has been closed for 13 years now but worth a shot.

I'm trying to upgrade thumbor to trixie and having both fonts causes this:

2.773 E: Unable to correct problems, you have held broken packages.
2.773 E: The following information from --solver 3.0 may provide additional context:
2.773    Unable to satisfy dependencies. Reached two conflicting decisions:
2.773    1. culmus-fancy:amd64=0.0.20240129.1 is selected for install
2.773    2. culmus-fancy:amd64=0.0.20240129.1 is not selected for install because:
2.773       1. culmus:amd64=0.140-3 is selected for install
2.773       2. culmus:amd64 Depends fonts-culmus
2.773       3. fonts-culmus:amd64 Breaks culmus-fancy

So I removed one https://gerrit.wikimedia.org/r/c/operations/software/thumbor-plugins/+/1325499/2/.pipeline/blubber.yaml and it now passes. I'm not sure if this is the right way to do it. Any ideas?

And since the prio was still high, which I missed. this isn't urgent at all anymore, any time next week would be good

In T434681#12212493, @Jhancock.wm wrote:

direct quote from support

JFTR, it's not unlikely the server will come back, since the disks are fine. We'll see what Supermicro says.

In T434681#12210194, @Jhancock.wm wrote:

What was the reboot for? If there was a firmware upgrade, it might have a conflict somewhere. a quick internet search shows a lot of different things. Feel free to try anything while I get some help from SM

Looks good, for eqiad please use A, B, C and for codfw B, C, D

cumin2003 will need an announcement one day ahead so that people don't run any cookbooks on it (some may take a bit, so a day before it best). Also, there are some scheduled DB and backup things running from cumin hosts.

We stopped installing dhcpcd-base on new trixie installations back in May and I've just removed it from the hosts which had been installed prior to that

All done

In T402006#12100175, @taavi wrote:

Hi @MoritzMuehlenhoff @SLyngshede-WMF, I've installed CAS 7 (more specifically, 7.3.7.3+wmf13u1) on a Trixie host (cloudinfra-idp-2.cloudinfra.eqiad1.wikimedia.cloud). It seems like the pac4j plugin needed for delegated authentication that we need for idp.wmcloud.org is either missing or not working in CAS 7, any idea why?

Just to confirm that this requirement is important:

we'd need CI checks validating both services still work after each Chromium bump, since version updates can introduce breaking changes.

mirrors.wikimedia.org has been deregistered from all mirror sources, it's internal use in production servers and the container pipeline has been disabled and the hardware was decommissioned. Marking this resolved. There a a few open sub tasks around other former use of mirrors.wikimedia.org, which can be addressed independently of this task.

In T371375#12090984, @LSobanski wrote:

Is there a plan to also remove prometheus-ethtool-exporter from Trixie hosts provisioned before the change was merged? Collab gets low volume operational noise from the exporter failing every now and then.

Thanks for confirming, resolving the task.

I reviewed the Puppet classes and everything should be ready for Trixie. How about we create a separate parallel, test node in magru first and when it's working we simply reimage the others in place? Or is ~ one hour downtime in the edges for a reimage too long?

I created a Debian package based on the last 1.3.8-rc release and uploaded it to trixie-wikimedia to the main component.

In T429338#12023739, @Gehel wrote:

We had a similar discussion on T427457. I'm not sure I understand well enough the difference between urldownloader and webproxy. webproxy:8080 should default to the local datacenter, webproxy.eqiad.wmnet:8080 should always go through eqiad. So webproxy:8080 is probably a better solution, but does not really matter in this context, since all those workloads will always run in eqiad (until we work on having multi-dc for the data infrastructure, which will no happen until quite a while).

In T427282#12075814, @MoritzMuehlenhoff wrote:

Squid got oom-killed on urldownloader1005. Traffic was stable across the day, but then traffic increase around the time it failed brought it over the edge. Also, Squid 6 seems to generally need more memory than the old Squid 4 we're running on the old URL downloaders (which is why we haven't seen this with the old nodes). The URL downloaders are still specced out from a time when they were mostly just proxying a bit of Citoid traffic and only have 2 GB RAM and 2 vCPUs. To provide more headroom also for increased use of hcaptcha over time, I'll raise this to 8G RAM and 4 vCPUs for the new nodes and then we can use the new Trixie nodes again.

@catherine.kelsey.wmde I've added you to the cn=airflow-wmde-ops LDAP group, please log out of the Wikimedia SSO by accessing https://idp.wikimedia.org/logout and then you should be able to log into https://airflow-wmde.wikimedia.org/

In T430961#12085280, @MoritzMuehlenhoff wrote:

In T430961#12084658, @AndrewTavis_WMDE wrote:

Looking into this a bit, is it that @catherine.kelsey.wmde needs to be added to analytics-wmde-users? This at the very least should be done given the responsibilities for her role, but if memory serves me this group is also the one that provides Airflow privileges.

The Airflow instances were migrated to Kubernetes https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Instances#wmde

With the new setup only the cn=airflow-wmde-ops LDAP group should be relevant.

In T430961#12084658, @AndrewTavis_WMDE wrote:

Looking into this a bit, is it that @catherine.kelsey.wmde needs to be added to analytics-wmde-users? This at the very least should be done given the responsibilities for her role, but if memory serves me this group is also the one that provides Airflow privileges.

In T421484#12005000, @Jgiannelos wrote:

I think this train is the last week we run both envs for our weekly testing (both internal parsoidtest and mw-parsoid). I have one more thing to check for our regression testing and then we are good to decommission.

In T430961#12083751, @Dzahn wrote:

Hi, I'm afraid airflow-wmde-ops is not a valid group.

It's worth noting that essentially the same happened also happened today during the late EMEA shift of @Raine and myself: https://portal.victorops.com/ui/wikimedia/incident/8123/details paged again 24 hours after the originally and then about an hour later the same happened for https://portal.victorops.com/ui/wikimedia/incident/8135/details and https://portal.victorops.com/ui/wikimedia/incident/8134/details (but a least it was all during normal work hours)

Please make sure to configure Java via profile::java, it'll do all the necessary setup magically in the background for you, no need to munually fiddle with repository settings.