Page MenuHomePhabricator

MoritzMuehlenhoff (Moritz Mühlenhoff)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Sunday

  • Clear sailing ahead.

User Details

User Since
Apr 1 2015, 4:33 PM (378 w, 2 d)
Availability
Available
LDAP User
Moritz Mühlenhoff
MediaWiki User
MMuhlenhoff (WMF) [ Global Accounts ]

Recent Activity

Today

MoritzMuehlenhoff renamed T311384: CVE-2022-31090/CVE-2022-31091: Update "guzzlehttp/guzzle" to 6.5.8/7.4.5 from CVE-2022-27776: Update "guzzlehttp/guzzle" to 6.5.8/7.4.5 to CVE-2022-31090/CVE-2022-31091: Update "guzzlehttp/guzzle" to 6.5.8/7.4.5.
Fri, Jul 1, 8:04 AM · Vuln-Misconfiguration, MW-1.37-notes, MW-1.35-notes, MW-1.39-notes (1.39.0-wmf.18; 2022-06-27), MW-1.38-notes, MW-1.38-release, MW-1.37-release, MW-1.35-release, MediaWiki-Vendor
MoritzMuehlenhoff added a comment to T311384: CVE-2022-31090/CVE-2022-31091: Update "guzzlehttp/guzzle" to 6.5.8/7.4.5.

JFTR, the mediawiki announcement referred to these under the CVE ID assigned for curl, but in the mean time these got assigned separate CVE IDs for guzzle, I'm renaming the task accordingly.

Fri, Jul 1, 8:04 AM · Vuln-Misconfiguration, MW-1.37-notes, MW-1.35-notes, MW-1.39-notes (1.39.0-wmf.18; 2022-06-27), MW-1.38-notes, MW-1.38-release, MW-1.37-release, MW-1.35-release, MediaWiki-Vendor
MoritzMuehlenhoff updated subscribers of T311804: Update Proton to include Chromium 103.0.5060.53.
Fri, Jul 1, 7:43 AM · Proton, Product-Infrastructure-Team-Backlog
MoritzMuehlenhoff created T311804: Update Proton to include Chromium 103.0.5060.53.
Fri, Jul 1, 7:43 AM · Proton, Product-Infrastructure-Team-Backlog

Yesterday

MoritzMuehlenhoff added a comment to T310980: Allow Cassandra to be deployed on Bullseye nodes.

I would propose that the way to think about this might be to ask ourselves how much runway we want/need from here to 4.x. 3.11.x is scheduled to be EOL mid-2023, about the same time we should be off of Buster. That's kind of tight given everything that needs to be done (including Bullseye and Cassandra 4 upgrades of all nodes). Backporting that changeset costs time/effort but lets us use 3.11 on Bullseye, is that a net win (not a rhetorical question)?

My general feeling about OS/service upgrades is that decoupling them makes our life easier, less variables that may interacts with each other leading to unexpected corner cases. I proposed to backport the patch since IIUC the upgrade from 3.11.x to 4.x is a ton of work, meanwhile adapting a commit to the 3.11.x codebase seems less time consuming and more time-boxed. This would allow us to move all our clusters to Bullseye first, and then work on Cassandra 4.x with less pressure. You are completely right about maintaining our own version + backporting patches etc.., but we are already doing it for a lot of other things so it should be doable in my opinion (even if, I agree, a little more painful than now).

Maybe we could set up a spike/task with a time-boxed number of hours, trying to backport the patch to see if it is doable in a reasonable amount of time, report back and then decide how to proceed?

Thu, Jun 30, 6:00 PM · Cassandra, SRE
MoritzMuehlenhoff added a comment to T311732: doc1001 upgrade to buster/bullseye.

Last status updates have been here: https://phabricator.wikimedia.org/T247653#7974301

Thu, Jun 30, 3:17 PM · serviceops
MoritzMuehlenhoff triaged T310980: Allow Cassandra to be deployed on Bullseye nodes as Medium priority.
Thu, Jun 30, 2:01 PM · Cassandra, SRE
MoritzMuehlenhoff triaged T311236: Enable webauthn in CAS to replace U2F as Medium priority.
Thu, Jun 30, 2:00 PM · CAS-SSO, Infrastructure-Foundations, SRE
MoritzMuehlenhoff updated the task description for T311686: Upgrade ganeti/codfw to Bullseye.
Thu, Jun 30, 12:31 PM · Ganeti, Infrastructure-Foundations, SRE
MoritzMuehlenhoff updated the task description for T311686: Upgrade ganeti/codfw to Bullseye.
Thu, Jun 30, 10:28 AM · Ganeti, Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a comment to T310980: Allow Cassandra to be deployed on Bullseye nodes.

If everybody agrees I'd keep Buster for the moment, and possibly ML could be the first cluster to be upgraded when Cassadra 4 is imported and scoped out (so helping Eric in the config and debug etc..).
Does it sound acceptable Moritz?

Thu, Jun 30, 7:49 AM · Cassandra, SRE
MoritzMuehlenhoff triaged T311687: Upgrade ganeti/eqiad to Bullseye as Medium priority.
Thu, Jun 30, 7:26 AM · Ganeti, Infrastructure-Foundations, SRE
MoritzMuehlenhoff triaged T311686: Upgrade ganeti/codfw to Bullseye as Medium priority.
Thu, Jun 30, 7:26 AM · Ganeti, Infrastructure-Foundations, SRE
MoritzMuehlenhoff created T311687: Upgrade ganeti/eqiad to Bullseye.
Thu, Jun 30, 7:26 AM · Ganeti, Infrastructure-Foundations, SRE
MoritzMuehlenhoff created T311686: Upgrade ganeti/codfw to Bullseye.
Thu, Jun 30, 7:25 AM · Ganeti, Infrastructure-Foundations, SRE

Wed, Jun 29

MoritzMuehlenhoff added a comment to T297913: Confirm support of PERC 750 raid controller.

So post dumpsdata1007 install it fails puppet due to megaraid monitoring items it seems?

Wed, Jun 29, 6:18 PM · Patch-For-Review, DC-Ops, SRE
MoritzMuehlenhoff added a comment to T302937: datadumps1007 test installs.

As mentioned at the SRE meeting @BTullis is also looking into this for DSE hosts (review at https://gerrit.wikimedia.org/r/c/operations/puppet/+/808870 and thus the cc).

I don't know if it is helpful at this point, however if wipefs is available you can:

wipefs -a /dev/<device>

To remove the LVM metadata, alternatively obliterating the first few megabytes should do it too:

dd if=/dev/zero of=/dev/sda bs=1M count=50

Ok, I tried this and I get the same issue, says LVM data exists like:

Screen Shot 2022-06-23 at 12.38.40 PM.png (738×982 px, 90 KB)
.

I've even fully initialized and rebuilt the arrays, no luck. wipefs doesn't exist in our installer shell.

Wed, Jun 29, 5:18 PM · SRE, DC-Ops
MoritzMuehlenhoff added a comment to T67270: Default license for operations/puppet.

In such cases it might make sense to align such files by relicensing to Apache 2

starting of with the obligatory IANAL :). My understanding is that if we have something with a GPL licence then that would mean that any resulting body of work, weather that is a specific module or the entire puppet repo would need to be licenced under GPL. This is why, i personally, have been treating GPL a bit different to MIT/BSD. however of course the licence of a piece of work is completely up to the original author and didn't/don't intend for my queries to explore a licence change to be construed as pressure.

Wed, Jun 29, 2:49 PM · Patch-For-Review, SRE, Software-Licensing, Documentation, WMF-Legal, WMF-General-or-Unknown
MoritzMuehlenhoff added a comment to T311236: Enable webauthn in CAS to replace U2F.

but that bails out with a bean error related to the fasterxml parser,

Wonder if this is related to the TODO in the gradle properties

Good point, I'll try to re-add as a test, maybe we in fact still need it.

Wed, Jun 29, 2:06 PM · CAS-SSO, Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a comment to T308659: Validate lemma length in Special:NewLexeme(Alpha) and label/description/aliases length in Special:NewProperty (CVE-2022-34750).

This appeared in the CVE feed as https://www.cve.org/CVERecord?id=CVE-2022-34750

Wed, Jun 29, 9:38 AM · Special:NewLexeme revival (Special:NewLexeme revival - sprint 11), MW-1.39-notes (1.39.0-wmf.18; 2022-06-27), Patch-For-Review, Vuln-DoS, Wikidata Lexicographical data, Wikidata, Security, Security-Team
MoritzMuehlenhoff renamed T308659: Validate lemma length in Special:NewLexeme(Alpha) and label/description/aliases length in Special:NewProperty (CVE-2022-34750) from Validate lemma length in Special:NewLexeme(Alpha) and label/description/aliases length in Special:NewProperty to Validate lemma length in Special:NewLexeme(Alpha) and label/description/aliases length in Special:NewProperty (CVE-2022-34750).
Wed, Jun 29, 9:37 AM · Special:NewLexeme revival (Special:NewLexeme revival - sprint 11), MW-1.39-notes (1.39.0-wmf.18; 2022-06-27), Patch-For-Review, Vuln-DoS, Wikidata Lexicographical data, Wikidata, Security, Security-Team
MoritzMuehlenhoff added a comment to T311593: Decommissioning two hosts end up with: Failed to wipe swraid.

I have a few more hosts to decommission. I can try to do so, but we'd not know whether it helped or it would have just worked without it too :)

Wed, Jun 29, 9:03 AM · Infrastructure-Foundations, SRE-tools
MoritzMuehlenhoff added a comment to T311593: Decommissioning two hosts end up with: Failed to wipe swraid.

Maybe we need run "swapoff -a" prior to the wipefs call?

Wed, Jun 29, 8:39 AM · Infrastructure-Foundations, SRE-tools
MoritzMuehlenhoff added a comment to T310980: Allow Cassandra to be deployed on Bullseye nodes.

Can't we just import the Cassandra 4 debs and use those? The work needs to happen at some point anyway and it's a fresh cluster. Buster is almost three years old, going into LTS stage in a month and per our designated OS lifecycle has a year left...

Wed, Jun 29, 8:12 AM · Cassandra, SRE
MoritzMuehlenhoff closed T305460: Upgrade webperf hosts to Bullseye as Resolved.

This is complete

Wed, Jun 29, 7:44 AM · Patch-For-Review, Performance-Team, SRE
MoritzMuehlenhoff updated the task description for T305460: Upgrade webperf hosts to Bullseye.
Wed, Jun 29, 7:35 AM · Patch-For-Review, Performance-Team, SRE

Tue, Jun 28

MoritzMuehlenhoff added a comment to T306654: Request sudo access for Jclark-ctr.

We discussed this in yesterday's SRE IF meeting: Let's start by adding sudo permissions for the three cookbooks listed, homer be implicitly started by these cookbooks. +2 on Puppet is root-equivalent and there should be very few cases left where it's needed for the server racking workflow (e.g. for extending the partman globbing if there's a new server naming scheme). If those remaining cases are identified, then this can also trickle into future automation work (e.g. the partman config could become a drop-down menu in Netbox at some point).

Tue, Jun 28, 12:35 PM · Patch-For-Review, Infrastructure-Foundations (FY2021/2022-Q4), SRE, SRE-Access-Requests
MoritzMuehlenhoff added a comment to T310980: Allow Cassandra to be deployed on Bullseye nodes.

I was able to bootstrap the cassandra ML cluster in codfw on Bullseye. The only odd thing is that the cassandra package, for some reason, ended up in rc state after some puppet runs and I had to re-install it manually.

Tue, Jun 28, 12:09 PM · Cassandra, SRE

Mon, Jun 27

MoritzMuehlenhoff added a comment to T311386: Install php 7.4 in production.

I have updated our PHP build in component/php74 to the latest 7.4.x release; 7.4.30. Uploaded as 7.4.30-3+0~20220627.69+debian10~1.gbpf2b381+wmf1+buster1

Mon, Jun 27, 2:53 PM · Patch-For-Review, Performance-Team (Radar), serviceops

Sat, Jun 25

dcaro awarded T210993: Deprecate Diamond collectors in Cloud VPS a Love token.
Sat, Jun 25, 12:46 PM · Patch-For-Review, cloud-services-team (Kanban), observability, SRE

Fri, Jun 24

MoritzMuehlenhoff added a comment to T311300: Repurpose the "cas" database for webauthn tokens.

Thanks!

Fri, Jun 24, 12:58 PM · DBA, CAS-SSO, Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a comment to T311300: Repurpose the "cas" database for webauthn tokens.

@MoritzMuehlenhoff so you want me to drop or truncate this table?:

Fri, Jun 24, 12:29 PM · DBA, CAS-SSO, Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a comment to T311131: Site: eqiad : 3 VMs requested for Etcd cluster in support of the new DSE Kubernetes cluster.

These look fine. Our Ganeti cookbook doesn't allow to create disks in plain mode, so these will be created using DRBD and then reconfigured to plain mode. Just ping me if you run into any issues.

Fri, Jun 24, 11:29 AM · DSE-Kubernetes-Cluster, vm-requests, Infrastructure-Foundations, SRE
MoritzMuehlenhoff created T311300: Repurpose the "cas" database for webauthn tokens.
Fri, Jun 24, 11:09 AM · DBA, CAS-SSO, Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a project to T219454: Make Spicerack cookbook to resize ganeti VM: Ganeti.
Fri, Jun 24, 10:06 AM · Ganeti, Infrastructure-Foundations, SRE-tools
MoritzMuehlenhoff added a project to T283320: Cookbook to failover the Ganeti master: Ganeti.
Fri, Jun 24, 10:06 AM · Ganeti, Infrastructure-Foundations, SRE-tools, SRE
MoritzMuehlenhoff added a project to T230712: sre.ganeti.makevm cook book only allows specifying RAM size in full gigabytes: Ganeti.
Fri, Jun 24, 10:06 AM · Ganeti, Infrastructure-Foundations, SRE-tools
MoritzMuehlenhoff added a project to T265904: Remove SLAAC IPs from Ganeti hosts: Ganeti.
Fri, Jun 24, 10:06 AM · Ganeti, Patch-For-Review, Traffic, SRE
MoritzMuehlenhoff added a project to T299034: Write a cookbook to align the "master-capable" state of Ganeti nodes: Ganeti.
Fri, Jun 24, 10:05 AM · Ganeti, SRE, Infrastructure-Foundations
MoritzMuehlenhoff added a project to T300046: sre.ganeti.makevm: Allow passing a secondary disk: Ganeti.
Fri, Jun 24, 10:05 AM · Ganeti, SRE-tools, Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a project to T300152: Investigate Ganeti in routed mode: Ganeti.
Fri, Jun 24, 10:05 AM · Ganeti, Infrastructure-Foundations
MoritzMuehlenhoff added a project to T299560: Enable drbd collector on ganeti nodes: Ganeti.
Fri, Jun 24, 10:05 AM · Ganeti, observability, Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a project to T283319: Cookbooks for Ganeti maintenance tasks: Ganeti.
Fri, Jun 24, 10:04 AM · Ganeti, SRE
MoritzMuehlenhoff added a project to T203964: Create a spicerack cookbook to empty a ganeti node from VMs: Ganeti.
Fri, Jun 24, 10:03 AM · Ganeti, Spicerack, Infrastructure-Foundations, SRE-tools, User-Joe, SRE
MoritzMuehlenhoff added a project to T309724: SSH host key verification failures in Ganeti intra node SSH calls after Bullseye update: Ganeti.
Fri, Jun 24, 10:03 AM · Ganeti, Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a project to T311288: Implement Prometheus exporter for Ganeti capacity data: Ganeti.
Fri, Jun 24, 10:02 AM · Ganeti, Infrastructure-Foundations, SRE
MoritzMuehlenhoff closed T304546: Integrate Buster 10.12 point update as Resolved.

This is completed

Fri, Jun 24, 9:51 AM · Infrastructure-Foundations, SRE
MoritzMuehlenhoff closed T210993: Deprecate Diamond collectors in Cloud VPS as Resolved.

This is complete, all puppet-managed Diamond collectors are gone by now.

Fri, Jun 24, 9:50 AM · Patch-For-Review, cloud-services-team (Kanban), observability, SRE
MoritzMuehlenhoff updated the task description for T304546: Integrate Buster 10.12 point update.
Fri, Jun 24, 9:50 AM · Infrastructure-Foundations, SRE
MoritzMuehlenhoff closed T210993: Deprecate Diamond collectors in Cloud VPS, a subtask of T183454: Deprovision Diamond collectors no longer in use, as Resolved.
Fri, Jun 24, 9:49 AM · Patch-For-Review, User-fgiunchedi, observability, SRE
MoritzMuehlenhoff created P30138 decom cookbook failure after netbox/ganeti cluster change.
Fri, Jun 24, 8:58 AM

Thu, Jun 23

MoritzMuehlenhoff added a comment to T310683: Please add new tag "Ganeti" .

"Wikimedia's installations of Ganeti, a clustered virtual machine management stack" should do, along with a link to https://wikitech.wikimedia.org/wiki/Ganeti

Thu, Jun 23, 5:46 PM · Project-Admins
MoritzMuehlenhoff added a comment to T311236: Enable webauthn in CAS to replace U2F.

but that bails out with a bean error related to the fasterxml parser,

Wonder if this is related to the TODO in the gradle properties

Thu, Jun 23, 3:35 PM · CAS-SSO, Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a comment to T311236: Enable webauthn in CAS to replace U2F.

Status update:
With a hacked-up config on idp-test.w.o and when configuring a user to pass mfa-webauthn to the Groovy script I'm getting into the webauthn device registration dialogue. Registering the token fails since no datastore is defined yet. I tried to set
cas.authn.mfa.web-authn.core.trusted-device-metadata.location to a file URI, but that bails out with a bean error related to the fasterxml parser, I guess need to provide some stub XML file there, needs some poking.

Thu, Jun 23, 3:04 PM · CAS-SSO, Infrastructure-Foundations, SRE
MoritzMuehlenhoff created T311236: Enable webauthn in CAS to replace U2F.
Thu, Jun 23, 2:29 PM · CAS-SSO, Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a comment to T311235: Update CAS to 6.5.

cas 6.5.5 has been built and uploaded to apt.wikimedia.org. It's currently installed on idp-test.wikimedia.org and functionality is working fine. The WMF-specific theming needs to be adapted still, the login screen is currently visually a little distorted.

Thu, Jun 23, 2:29 PM · Patch-For-Review, CAS-SSO, Infrastructure-Foundations, SRE
MoritzMuehlenhoff created T311235: Update CAS to 6.5.
Thu, Jun 23, 2:28 PM · Patch-For-Review, CAS-SSO, Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a parent task for T308214: Migrate the IDPs to Bullseye: T305518: Upgrade IDPs to CAS 6.5/Bullseye and enable webauthn.
Thu, Jun 23, 2:27 PM · Patch-For-Review, Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a subtask for T305518: Upgrade IDPs to CAS 6.5/Bullseye and enable webauthn: T308214: Migrate the IDPs to Bullseye.
Thu, Jun 23, 2:27 PM · CAS-SSO, Infrastructure-Foundations, SRE

Wed, Jun 22

MoritzMuehlenhoff added a comment to T309447: Icinga paged for a host that should have been downtimed.

@Volans: Can this task be closed with https://gerrit.wikimedia.org/r/803317 merged?

Wed, Jun 22, 11:02 AM · Infrastructure-Foundations, SRE-tools, Icinga, observability, SRE
MoritzMuehlenhoff closed T309765: Retire the old Parsoid deb repository? as Resolved.

Since there were no further objections, the repository has now been removed.

Wed, Jun 22, 8:07 AM · Infrastructure-Foundations, SRE, Parsoid

Tue, Jun 21

MoritzMuehlenhoff closed T308214: Migrate the IDPs to Bullseye as Resolved.

This is complete

Tue, Jun 21, 5:42 PM · Patch-For-Review, Infrastructure-Foundations, SRE
MoritzMuehlenhoff updated the task description for T308214: Migrate the IDPs to Bullseye.
Tue, Jun 21, 5:40 PM · Patch-For-Review, Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a comment to T67270: Default license for operations/puppet.

Can we clarify what the goal here is? More recently I've been good about throwing a GPL-3.0-or-later header on substantial scripts committed to puppet (e.g. https://codesearch.wmcloud.org/puppet/?q=%5C(C%5C)%20.*%20Kunal%20Mehta&i=nope&files=&excludeFiles=&repos=), do we actually want/need to relicense those to Apache 2.0?

Tue, Jun 21, 1:15 PM · Patch-For-Review, SRE, Software-Licensing, Documentation, WMF-Legal, WMF-General-or-Unknown
MoritzMuehlenhoff closed T289715: (Need By: TBD) rack/setup/install ganeti4004 as Resolved.

ganeti4004 has been added to the ganeti/ulsfo cluster now. Cluster is currently rebalancing.

Tue, Jun 21, 12:30 PM · SRE, ops-ulsfo, Traffic, DC-Ops
MoritzMuehlenhoff closed T289715: (Need By: TBD) rack/setup/install ganeti4004, a subtask of T288579: decommission bast4002.wikimedia.org, as Resolved.
Tue, Jun 21, 12:29 PM · DC-Ops, ops-ulsfo, SRE, Traffic, decommission-hardware
MoritzMuehlenhoff updated the task description for T304546: Integrate Buster 10.12 point update.
Tue, Jun 21, 9:48 AM · Infrastructure-Foundations, SRE

Mon, Jun 20

MoritzMuehlenhoff added a comment to T310980: Allow Cassandra to be deployed on Bullseye nodes.

Due to a dependency on python-yaml in https://gerrit.wikimedia.org/r/admin/repos/operations/debs/cassandra-tools-wmf, we'd need to create a new package that depends on python3-yaml and upload it to bullseye-wikimedia (at least this is my undestanding, let me know if there is more).

Mon, Jun 20, 10:45 AM · Cassandra, SRE

Fri, Jun 17

MoritzMuehlenhoff added a comment to T289715: (Need By: TBD) rack/setup/install ganeti4004.

Thanks! I'll do that on Tuesday

Fri, Jun 17, 3:32 PM · SRE, ops-ulsfo, Traffic, DC-Ops
MoritzMuehlenhoff added a comment to T308013: Assign SPDX headers to puppet.git.

I reached out to Marc-André Pelletier (Coren) via email and he replied the following (quoted with his permission), as such I'm listing him under CONTRIBUTORS with his marc@uberbox.org email address:

Fri, Jun 17, 2:13 PM · Patch-For-Review, Infrastructure-Foundations, SRE
MoritzMuehlenhoff closed T306499: Upgrade ganeti-test to Bullseye as Resolved.

This is complete.

Fri, Jun 17, 10:28 AM · Infrastructure-Foundations, SRE
MoritzMuehlenhoff closed T308211: Upgrade ganeti/eqsin to Bullseye as Resolved.

This is complete. The eqsin cluster is affected by T309724, but that will be investigated via that task (and it doesn't have a functional impact apart from the fact that gnt-cluster verify fails)

Fri, Jun 17, 10:28 AM · Infrastructure-Foundations, SRE
MoritzMuehlenhoff closed T307997: Upgrade ganeti/ulsfo to Bullseye as Resolved.

This is complete. The ulsfo cluster is affected by T309724, but that will be investigated via that task (and it doesn't have a functional impact apart from the fact that gnt-cluster verify fails)

Fri, Jun 17, 10:28 AM · Infrastructure-Foundations, SRE
MoritzMuehlenhoff updated the task description for T307997: Upgrade ganeti/ulsfo to Bullseye.
Fri, Jun 17, 10:24 AM · Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a comment to T302232: Set up the ml-cache clusters.

Next step is to bootstrap the codfw cluster, and then we should be done. We should try to figure out if we can use Bullseye and not Buster though.

Fri, Jun 17, 10:05 AM · Patch-For-Review, Epic, Lift-Wing, Machine-Learning-Team (Active Tasks)
MoritzMuehlenhoff added a comment to T289715: (Need By: TBD) rack/setup/install ganeti4004.

The server can be powered down any time, while it already has the ganeti role, it's not yet added to the cluster.

Fri, Jun 17, 8:41 AM · SRE, ops-ulsfo, Traffic, DC-Ops
MoritzMuehlenhoff reopened T289715: (Need By: TBD) rack/setup/install ganeti4004 as "Open".

The server doesn't have virtualisation enabled. I tried to enable it via the BIOS over the serial console, but I'm not getting a console, just "Unified Server Configurator does not support console redirection", does this need to be enabled somewhere?

Fri, Jun 17, 8:37 AM · SRE, ops-ulsfo, Traffic, DC-Ops
MoritzMuehlenhoff reopened T289715: (Need By: TBD) rack/setup/install ganeti4004, a subtask of T288579: decommission bast4002.wikimedia.org, as Open.
Fri, Jun 17, 8:37 AM · DC-Ops, ops-ulsfo, SRE, Traffic, decommission-hardware

Thu, Jun 16

MoritzMuehlenhoff closed T308238: Upgrade ganeti/esams to Bullseye as Resolved.

This is complete.

Thu, Jun 16, 3:15 PM · Infrastructure-Foundations, SRE

Wed, Jun 15

MoritzMuehlenhoff added a comment to T122144: Move most (all?) exim personal aliases to WMF ITS.

I also removed logsteralarms@ earlier the day, it's no longer needed.

Wed, Jun 15, 12:58 PM · Infrastructure-Foundations, Epic, Mail, SRE
MoritzMuehlenhoff added a comment to T310686: Re-enable CAS-SSO for hue.wikimedia.org.

Ben and myself did some debugging: While we had been using CAS for Hue for the last two years, it was never explicitly enabled within Hiera: profile::hue::enable_cas was still set to false as in the original commit which introduced it: https://github.com/wikimedia/puppet/commit/b9c17a9a4a5a6c21c04791830713b42c93eb2c1c

Wed, Jun 15, 11:17 AM · Infrastructure-Foundations, Data-Engineering-Kanban, Data-Engineering

Tue, Jun 14

MoritzMuehlenhoff added a comment to T309383: Requesting access to PII in Superset for TheresNoTime.

Sorry, there was something still missing. It should be fixed now, I have just merged the patch (but it will take up to 30 minutes for Puppet to deploy the change fully). Please let me know if that fixes access for you.

Tue, Jun 14, 1:46 PM · SRE, SRE-Access-Requests
MoritzMuehlenhoff updated the task description for T308238: Upgrade ganeti/esams to Bullseye.
Tue, Jun 14, 10:32 AM · Infrastructure-Foundations, SRE

Mon, Jun 13

MoritzMuehlenhoff triaged T309499: Update Proton to include Chromium 102.0.5005.61-1~deb11u1 as High priority.
Mon, Jun 13, 3:10 PM · Product-Infrastructure-Team-Backlog (Kanban), Proton
MoritzMuehlenhoff added a comment to T309499: Update Proton to include Chromium 102.0.5005.61-1~deb11u1.

A new chromium version has been released, the new target version is 102.0.5005.115-1~deb11u1

Mon, Jun 13, 3:10 PM · Product-Infrastructure-Team-Backlog (Kanban), Proton
MoritzMuehlenhoff updated the task description for T308214: Migrate the IDPs to Bullseye.
Mon, Jun 13, 3:07 PM · Patch-For-Review, Infrastructure-Foundations, SRE
MoritzMuehlenhoff added a comment to T310451: hdfs client packages for debian Bullseye.

I think the bigtop15 .deb packages can/should just be copied to bullsye?

Mon, Jun 13, 12:30 PM · cloud-services-team (Kanban), Infrastructure-Foundations, SRE
MoritzMuehlenhoff reassigned T308238: Upgrade ganeti/esams to Bullseye from MoritzMuehlenhoff to RobH.

ganeti3001 is removed from the cluster, downtimed and needs the same firmware/NIC updates to enable the reimage to Bullseye.

Mon, Jun 13, 12:17 PM · Infrastructure-Foundations, SRE
MoritzMuehlenhoff updated the task description for T309724: SSH host key verification failures in Ganeti intra node SSH calls after Bullseye update.
Mon, Jun 13, 10:51 AM · Ganeti, Infrastructure-Foundations, SRE
MoritzMuehlenhoff updated the task description for T308238: Upgrade ganeti/esams to Bullseye.
Mon, Jun 13, 7:55 AM · Infrastructure-Foundations, SRE
MoritzMuehlenhoff created T310463: Check home/HDFS leftovers of dsharpe.
Mon, Jun 13, 7:11 AM · Data-Engineering

Sun, Jun 12

MoritzMuehlenhoff closed T310431: Unable to view all Wikimedia projects as Resolved.

The immediate issue has been resolved, closing. There are some actionables, but rather sub tasks to existing tasks and those will be created early next week.

Sun, Jun 12, 9:11 AM · Wikimedia-Incident, SRE, Traffic

Fri, Jun 10

MoritzMuehlenhoff updated the task description for T308238: Upgrade ganeti/esams to Bullseye.
Fri, Jun 10, 9:56 AM · Infrastructure-Foundations, SRE

Thu, Jun 9

MoritzMuehlenhoff updated the task description for T305460: Upgrade webperf hosts to Bullseye.
Thu, Jun 9, 3:00 PM · Patch-For-Review, Performance-Team, SRE
MoritzMuehlenhoff updated the task description for T305460: Upgrade webperf hosts to Bullseye.
Thu, Jun 9, 2:55 PM · Patch-For-Review, Performance-Team, SRE
MoritzMuehlenhoff updated the task description for T305460: Upgrade webperf hosts to Bullseye.
Thu, Jun 9, 2:45 PM · Patch-For-Review, Performance-Team, SRE
MoritzMuehlenhoff updated the task description for T305460: Upgrade webperf hosts to Bullseye.
Thu, Jun 9, 2:23 PM · Patch-For-Review, Performance-Team, SRE
MoritzMuehlenhoff reassigned T308238: Upgrade ganeti/esams to Bullseye from MoritzMuehlenhoff to RobH.

ganeti3002 is removed from the cluster, downtimed and needs the same firmware/NIC updates to enable the reimage to Bullseye.

Thu, Jun 9, 12:23 PM · Infrastructure-Foundations, SRE
MoritzMuehlenhoff updated the task description for T308238: Upgrade ganeti/esams to Bullseye.
Thu, Jun 9, 12:22 PM · Infrastructure-Foundations, SRE

Wed, Jun 8

MoritzMuehlenhoff closed T309878: Import Debian package of Cassandra 3.11.13 as 'dev' version as Resolved.

It looks like the cassandra-tools package is missing.

Wed, Jun 8, 5:07 PM · Cassandra