sre.ganeti.makevm now supports fractions of gigabytes.

Is there a task about the udp2log porting work to Python 3, or will that be unnecessary due to T205856?

The old-style syntax is used all over the place and it would be a significant effort to change. Since it wil be continue to supported by Apache in the foreeseable future, marking as declined for now

This happened via T337269

Indeed, there's nothing really to fix here (or something changed between 2018 and now): perf-roots grants a few people root access on app servers and caches to help them debug some issues. deployment access is moslty unrelated to that.

I'd say let's just remove legacy_compat, nothing should rely on it anymore.

I think this is resolved. Since this task was opened we obsoleted some controllers and generally shrunk the list of packages we imported for later distro releases.

In T142821#8873575, @jbond wrote:

im going to close this, altough i think its probably still valuable i think its already captured in the IDM planning, but please re-open if you disagree

Yes, I think we can close this. This didn't cause any other issue AFAICT (in fact I don't remeber the issue that prompted to file the task) and we're moving away from Exim anyway.

In T336845#8870600, @jbond wrote:

The auto restarts are needed in Cloud VPS as well; with unattended-upgrades installing a security update of libfoo, this ensures that all services using a copy of libfoo get restarted, otherwise they'd continue to use the old library.

i agree with this however currently the wmf-auto-restart.py config options depends on the debdeploy config file as such i think we need to refactor and decouple this dependency before we can configure options in the cloud environment. Worth noting that the script will work however /etc/debdeploy-client/config.json will not exist as such the relevant settings in hiera will not apply

I've updated https://wikitech.wikimedia.org/wiki/Ganeti to point to the new cookbook

In T336995#8869914, @MoritzMuehlenhoff wrote:

In T336995#8865388, @Dzahn wrote:

This host is still in Icinga.. so not removed from puppet db or something...

We have a rare race in the decom cookbook, where it sometimes fails to properly remove the puppetdb entry (likely a case where a submission is ongoing while the record is pruned), I've now re-run the cookbook.

The old cookbook has been removed and the docs were updated.

In T330884#8865295, @ayounsi wrote:

I copied over samplicator from bullseye-wikimedia to bookworm-wikimedia (the only dependency is glibc itself), but there wasn't a source package on apt.wikimedia.org, do you by chance still have it on your laptop or the build host so that we can import it?

Unfortunately, no, if it can help I think I got it from https://github.com/sleinen/samplicator
Some people tried to generate deb files as well, see https://github.com/hfeeki/samplicator-debian

In T336995#8865388, @Dzahn wrote:

This host is still in Icinga.. so not removed from puppet db or something...

In T336856#8864804, @sbassett wrote:

Unless there are still general concerns about more widely disclosing this issue.

So far we've been only using Apereo CAS for authentication against our self-hosted infrastructure. Given that SemGrep is more along the linesof other SaaS solutions provided to staff I think makes more sense to integrate it into Okta.

New target version is 113.0.5672.126:
https://lists.debian.org/debian-security-announce/2023/msg00095.html

@ayounsi There's now netflow2003 running Bookworm with FNM 1.2.4. If that works fine, we can reimage the other netflow* VMs in-place once Bookworm is stable.

In T336856#8861279, @akosiaris wrote:

Question: do we open this task to the public?

I think I'll simply backport/release the patch to bullseye-security, then we can simply re-use the resulting binary and import it to buster-wikimedia (it only depends on adduser, libc6 (>= 2.4), lsb-base)

I think you should simply add nfsv4 to profile::debdeploy::client::exclude_filesystems, that will entire make the issues vanish. We have similar config snippets in prouction for systems with an HFDS mount (such as the stat* nodes).

Usual IANAL disclaimer ahead: If this were a software license this would not meet the standard required by OSI. They e.g. cover this in the FAQ at https://opensource.org/faq/#evil and one infamous example is the JSON license (http://www.json.org/license.html) for which https://lwn.net/Articles/707510/ is a nice writeup. That said, restrictions might not be fully enforced (I have no idea if "You agree not to use the Model or Derivatives of the Model" is a binding restriction?)

Thanks for all the input, much appreciated! I'll revise the plan and update the task in the next days.

Old task, no longer really actionable at this point and this hasn't been seen since then.

In T271587#8843926, @jbond wrote:

@SLyngshede-WMF, @MoritzMuehlenhoff this seems like something that fits with the IDM. I'm not sure it needs to be part of the bitu software. however perhaps a timer on that machine, thoughts?

The installer is working fine for baremetal and VM installations, but there will be a few more RC releases before the final release, so keeping the task open for now.

In T336231#8836808, @Jgreen wrote:

This cert is used by the frack banner loggers to connect to production kafka. I'm not sure how we would put cergen in frack unless we're also importing the production puppet CA, since fundraising doesn't use production puppet at all. One hitch with the current setup is that we don't get notified when that cert is approaching expiration. We can fix that internally with our own cert monitoring, but that would still leave SRE getting notified separately for a cert used externally.

In T334455#8836415, @Marostegui wrote:

I am not sure if @MoritzMuehlenhoff was involved and might recall something, or could help us figure this out.

One other actionable: We'd need a new LDAP group (e.g. cn=fr-tech-admins or cn=fr-tech-sres) which grants +2 on operations/git.dns

AFAICS are two remaining use cases for the private repo:

The current VMs are quite overdimensioned in terms of CPU core: I'd go with 4G RAM, 4 CPUs and 20G disk space instead for ldap-rw1001/2001

In T334154#8832825, @jijiki wrote:

Dear Infrastructure-Foundations, please choose a name for this new group

In T335981#8829759, @sbassett wrote:

@MoritzMuehlenhoff - I'd agree with @Aklapper that you should probably just link to https://www.mediawiki.org/wiki/Security/SOP/Access_to_Phabricator_Security_Issues in the internal SRE doc as that describes the ideal process to follow. If folks email security-help@ about this, we would just create a Phab task for the request anyways.

Maybe it's time to add "Mail security-help@wikimedia.org to get Security access in Phabricator" as part of the onboarding checklist for SREs. Anyone in SRE needs to be able to react to Security tasks opened by users, so this seems like a sensible default.

In T305147#8828793, @Marostegui wrote:

@herron we've seen this alert being flapping on db2180 a lot lately:

[08:15:42]  <jinxer-wm> (SystemdUnitFailed) firing: ipmiseld.service Failed on db2180:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:30:42]  <jinxer-wm> (SystemdUnitFailed) resolved: ipmiseld.service Failed on db2180:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed

Even during the time of the alert, ipmi-tool keeps working fine. So I am not sure what's going on. Also, there are no wikitech page associated to it and I haven't been able to find any while searching. Can you give some insights here?
Thanks!

You need to email security-help@wikimedia.org, they make the change.

This is resolved:

• apt sources on remaining stretch servers stopped using the mirrors
• stretch-based images are no longer being reported by docker-reporter and no longer being built in the core images
• support for stretch has been removed from the cowbuilders on build2001

I could track this down to the upstream security fix for https://www.cve.org/CVERecord?id=CVE-2020-27759.

Narrowed down to these three patches: