I double checked all approvals and checkboxes and merged you change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1354856.

• access request (or expansion) has sign off of group approver indicated by the approval field in data.yaml

Thank you!

The backup conflicted with the version upgrade in T440854.

In T438819#12363816, @Jclark-ctr wrote:

I have installed a new NIC, and it has link. It is also showing up in iDRAC. However, I do not have root access, and I believe the IP address needs to be moved to the new PCIe NIC using root access from the console. Can you assist with this?

All wikikube services have been migrated. The last services were mw-web* and mw-api-ext*. I didn't notice any problem beside the short 5xx spike from the pybal restarts.

The legacy CNAME url-downloader URL was removed from wmf-navigator with the patch above.

Adding Collaboration-Services tag for wmf-navigator

In T438819#12350122, @Jclark-ctr wrote:

@Jelto It looks like the NICs on the motherboard have failed. We could possibly see if we have any PCI cards we can put into the server. Let me know what you would like to do.

All certificates were renewed successfully after updating the network policies. I'll resolve the task.

Great, thanks for the feedback. Then I'll resolve the task.

This is related to moving PKI to a different port, see https://gerrit.wikimedia.org/r/c/operations/puppet/+/1345042 and T436809.

@ABran-WMF I'm wondering why this alert is firing.

I depooled the runners in codfw without any effect. After that I rebooted the gitlab-runners using sre.gitlab.reboot-runner cookbook and triggered one of the failed publish jobs. The job published the image successfully: https://gitlab.wikimedia.org/repos/wmde/wikidata-query-gui/-/jobs/973417

I can try pausing Trusted Runners in codfw for some time. But I'm in the middle of a maintenance in T439037 which takes another ~30m. After that I'll try pausing the runners in codfw.

I downtimed the host for 24h and set it to inactive

I powercycled the host. It restarted but the issues were not resolved.

Management interface is reachable but getsel has no recent entries, most recent entry is from 2025. But it has quite a lot of entries, which looks suspicious.

add a retry policy (exponential backoff limited to 5 ?) to the script to avoid failing on the first error

This is fixed now, the text in all NOTES.txt was updated in the change above with a more detailed version.

eventgate* was migrated to IPIP. None of the migrations caused any latency peaks or other problems so far. So we should be good to convert bigger batches of services. I'll upload changes.

@ssingh reported in wikimedia-traffic a pending pybal restart and manually restarted pybal.

In T377889#12242438, @Arnoldokoth wrote:

@Aklapper I was hoping to use this to track decommissioning phab1004. I think once we've done that we can consider this complete.

The NOTES.txt originates from the scaffold/chart template which also contains

The first certificates in wikikube eqiad and codfw have been renewed successfully: https://grafana.wikimedia.org/d/vo5tiJTnz/cert-manager?from=now-7d&orgId=1&timezone=utc&to=now&var-12:39%20site=codfw&var-cluster=k8s&var-controller=$__all&var-datasource=000000026&var-namespace=$__all&var-site=eqiad&viewPanel=panel-28

Upgrade of production host and last successful restore of the replicas were more than 48h apart. The change above increases the downtime to 60h.

cert-manager has been updated in wikikube staging and production. In production the certificates expire in roughly 13 days and should be renewed in 3 days (see this table). I'll double check renewal in wikikube in a few days.

cert-manager has been updated on wikikube staging-codfw and staging-codfw. I'll proceed with wikikube production on Monday.