Page MenuHomePhabricator

MoritzMuehlenhoff (Moritz Mühlenhoff)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Apr 1 2015, 4:33 PM (315 w, 3 d)
Availability
Available
LDAP User
Moritz Mühlenhoff
MediaWiki User
MMuhlenhoff (WMF) [ Global Accounts ]

Recent Activity

Fri, Apr 16

MoritzMuehlenhoff added a comment to T276473: Proposed changes to the SRE Access request (Phabricator form 8).

Looks good to me!

Fri, Apr 16, 10:16 AM · Phabricator
MoritzMuehlenhoff added a comment to T247364: Forward port Python2 files to Python3 in Puppet Repository.

@crusnov Could you please take modules/raid/files/check-raid.py with precedence? It's part of a Bullseye base install and thus affected by it's lack of Python 2.

Fri, Apr 16, 10:02 AM · Patch-For-Review, User-MoritzMuehlenhoff, User-crusnov, User-jbond, Python3-Porting, SRE-tools, Puppet
MoritzMuehlenhoff updated the task description for T275873: Prepare our base system layer for Debian 11/bullseye.
Fri, Apr 16, 8:54 AM · Patch-For-Review, SRE
MoritzMuehlenhoff added a comment to T276473: Proposed changes to the SRE Access request (Phabricator form 8).

Looks good! Some comments below:

Fri, Apr 16, 8:20 AM · Phabricator

Thu, Apr 15

MoritzMuehlenhoff created T280253: Allow bast1003 in management routers (and drop bast1002).
Thu, Apr 15, 1:44 PM · SRE, netops
MoritzMuehlenhoff updated the task description for T279054: Integrate Buster 10.9 point update.
Thu, Apr 15, 11:45 AM · SRE
MoritzMuehlenhoff added a project to T280242: Requesting access to graphite hosts for awight: SRE-Access-Requests.
Thu, Apr 15, 10:48 AM · SRE, SRE-Access-Requests, Graphite, observability
MoritzMuehlenhoff added a comment to T280242: Requesting access to graphite hosts for awight.

@lmata: for your approval

Thu, Apr 15, 10:46 AM · SRE, SRE-Access-Requests, Graphite, observability

Wed, Apr 14

MoritzMuehlenhoff updated the task description for T279054: Integrate Buster 10.9 point update.
Wed, Apr 14, 4:10 PM · SRE
MoritzMuehlenhoff added a comment to T276220: Internal APT repository backup.

As far as i can tell all the necessary data is in /srv/wikimedia which is already being backed up via

Wed, Apr 14, 1:23 PM · Data-Persistence-Backup
Marostegui awarded T224589: Migrate dbmonitor hosts to Buster a Love token.
Wed, Apr 14, 11:46 AM · Patch-For-Review, SRE
MoritzMuehlenhoff closed T224589: Migrate dbmonitor hosts to Buster as Resolved.

Tendril and dbtree are now running on a new Buster instance dbmonitor1002.wikimedia.org ith PHP 5.6 packages from sury.org (since Tendril needs the mysql extention dropped in PHP 7) and dbmonitor1001/jessie has been removed.

Wed, Apr 14, 11:44 AM · Patch-For-Review, SRE
MoritzMuehlenhoff closed T224589: Migrate dbmonitor hosts to Buster, a subtask of T224549: Track remaining jessie systems in production, as Resolved.
Wed, Apr 14, 11:43 AM · SRE
MoritzMuehlenhoff closed T224589: Migrate dbmonitor hosts to Buster, a subtask of T247045: Migrate all of production metal and VMs to Buster or later, as Resolved.
Wed, Apr 14, 11:43 AM · SRE, Epic
MoritzMuehlenhoff closed T224589: Migrate dbmonitor hosts to Buster, a subtask of T250666: Upgrade WMF database-and-backup-related hosts to buster, as Resolved.
Wed, Apr 14, 11:43 AM · Patch-For-Review, Epic, DBA
MoritzMuehlenhoff added a project to T280110: decommission bast1002.wikimedia.org: ops-eqiad.
Wed, Apr 14, 10:46 AM · SRE, ops-eqiad, decommission-hardware
MoritzMuehlenhoff updated the task description for T280110: decommission bast1002.wikimedia.org.
Wed, Apr 14, 10:45 AM · SRE, ops-eqiad, decommission-hardware
MoritzMuehlenhoff closed T224579: Migrate irc.wikimedia.org/kraz to Buster as Resolved.

kraz has been replaced by two Buster instances (irc1001.wikimedia.org and irc2001.wikimedia.org) was eventually removed.

Wed, Apr 14, 10:12 AM · Patch-For-Review, User-notice, Wikimedia-IRC-RC-Server, SRE
MoritzMuehlenhoff closed T224579: Migrate irc.wikimedia.org/kraz to Buster, a subtask of T224549: Track remaining jessie systems in production, as Resolved.
Wed, Apr 14, 10:11 AM · SRE
MoritzMuehlenhoff updated the task description for T224549: Track remaining jessie systems in production.
Wed, Apr 14, 9:54 AM · SRE
MoritzMuehlenhoff updated the task description for T276399: migrate services from bast1002 to bast1003.
Wed, Apr 14, 8:48 AM · Patch-For-Review, SRE
MoritzMuehlenhoff closed T276399: migrate services from bast1002 to bast1003, a subtask of T276396: (Need By: TBD) rack/setup/install bast1003.wikimedia.org, as Resolved.
Wed, Apr 14, 8:48 AM · SRE, ops-eqiad, DC-Ops
MoritzMuehlenhoff closed T276399: migrate services from bast1002 to bast1003 as Resolved.

bast1003 has now fully replaced bast1002. The decom task for bast1002 is T280110

Wed, Apr 14, 8:48 AM · Patch-For-Review, SRE
MoritzMuehlenhoff created T280110: decommission bast1002.wikimedia.org.
Wed, Apr 14, 8:45 AM · SRE, ops-eqiad, decommission-hardware
MoritzMuehlenhoff added a comment to T128592: Add redundancy to IRC recent changes service.

Is it even possible for IRC to be active-active? Doesn't the client have to maintain a connection with a single server, and if that server drops, they disconnect, retry and get a connection again (maybe internally to a different server)? In that downtime though you're going to miss a few events. Unless the server remembers what your last position was (which the EventStreams protocol does!), I'm not sure how we avoid that.

Wed, Apr 14, 7:37 AM · Sustainability, SRE, codfw-rollout

Tue, Apr 13

Reedy awarded T279628: Porting scap to Python 3 a Mountain of Wealth token.
Tue, Apr 13, 1:08 PM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Python3-Porting, Scap, SRE
MoritzMuehlenhoff triaged T279509: git-fat needs to be ported to Python 3 as High priority.
Tue, Apr 13, 1:03 PM · Python3-Porting, SRE
MoritzMuehlenhoff triaged T279628: Porting scap to Python 3 as High priority.
Tue, Apr 13, 1:03 PM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Python3-Porting, Scap, SRE
MoritzMuehlenhoff added a comment to T224589: Migrate dbmonitor hosts to Buster.

Probably known (sorry) but the other alert I saw recently was: "CRITICAL: the following (6) node(s) change every puppet run: dbmonitor1001.wikimedia.org,...". Probably related to this?

Tue, Apr 13, 12:24 PM · Patch-For-Review, SRE
MoritzMuehlenhoff updated the task description for T224549: Track remaining jessie systems in production.
Tue, Apr 13, 9:34 AM · SRE
MoritzMuehlenhoff added a comment to T276448: Failover m1 master: db1080 -> db1159 Wed 14th April at 10 AM UTC.

@jbond @MoritzMuehlenhoff ok to restart mysql from cas and pki point of view tomorrow 14th April?

Tue, Apr 13, 9:00 AM · DBA
MoritzMuehlenhoff added a comment to T224589: Migrate dbmonitor hosts to Buster.

I have stopped apache on dbmonitor1001 (and done chmod -x to apache2 binary so puppet doesn't bring it up), let's leave it till next week and if nothing breaks, let's decommission it

Tue, Apr 13, 8:57 AM · Patch-For-Review, SRE

Thu, Apr 8

MoritzMuehlenhoff updated the task description for T279054: Integrate Buster 10.9 point update.
Thu, Apr 8, 1:20 PM · SRE
MoritzMuehlenhoff created T279628: Porting scap to Python 3.
Thu, Apr 8, 8:28 AM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Python3-Porting, Scap, SRE
MoritzMuehlenhoff added a project to T277064: Packaging PostGIS 3.1 for the new Maps stack: SRE.
Thu, Apr 8, 8:21 AM · Product-Infrastructure-Team-Backlog, SRE, Packaging, serviceops, Maps
MoritzMuehlenhoff added a comment to T277064: Packaging PostGIS 3.1 for the new Maps stack.

@MSantos, @hnowlan : I've uploaded the postgis 3.1.1 backport to the newly created component/postgis for buster. You can add it to the maps Puppet manifests using apt::package_from_component (feel free to add me as reviewer).

Thu, Apr 8, 8:20 AM · Product-Infrastructure-Team-Backlog, SRE, Packaging, serviceops, Maps

Wed, Apr 7

MoritzMuehlenhoff updated subscribers of T279244: CAS SSO for reedy.

I think racktables is replaced by netbox for Reedy's needs and he does have access to that. This ticket is down to "redirect racktables to netbox" or "add a banner telling users it's outdated" afaict.

Do you mean a banner on the SSO login prompt?

Wed, Apr 7, 3:17 PM · CAS-SSO, SRE, LDAP-Access-Requests
MoritzMuehlenhoff updated the task description for T279033: Upgrade Jenkins to 2.277.x.
Wed, Apr 7, 1:35 PM · Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Release-Engineering-Team (CI & Testing services), Jenkins, Continuous-Integration-Infrastructure
MoritzMuehlenhoff updated the task description for T279033: Upgrade Jenkins to 2.277.x.
Wed, Apr 7, 1:33 PM · Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)), Release-Engineering-Team (CI & Testing services), Jenkins, Continuous-Integration-Infrastructure
MoritzMuehlenhoff added a comment to T279509: git-fat needs to be ported to Python 3.

git-fat is the only package requiring Python 2 in a base bullseye setup at this point.

Wed, Apr 7, 12:21 PM · Python3-Porting, SRE
MoritzMuehlenhoff added a comment to T224579: Migrate irc.wikimedia.org/kraz to Buster.

I've rebooted kraz to force the remaining bots still connected to kraz to reconnect to irc2001.w.o.

Wed, Apr 7, 10:08 AM · Patch-For-Review, User-notice, Wikimedia-IRC-RC-Server, SRE
MoritzMuehlenhoff created T279509: git-fat needs to be ported to Python 3.
Wed, Apr 7, 9:13 AM · Python3-Porting, SRE

Tue, Apr 6

MoritzMuehlenhoff updated the task description for T276399: migrate services from bast1002 to bast1003.
Tue, Apr 6, 11:17 AM · Patch-For-Review, SRE
MoritzMuehlenhoff added a comment to T276399: migrate services from bast1002 to bast1003.

bast1003 is up and running; I've sent an announcement to the ops list so that people update their configs. Will open a decom task next week.

Tue, Apr 6, 11:16 AM · Patch-For-Review, SRE
MoritzMuehlenhoff updated the task description for T224549: Track remaining jessie systems in production.
Tue, Apr 6, 9:03 AM · SRE
MoritzMuehlenhoff added a comment to T279244: CAS SSO for reedy.

tagging @MoritzMuehlenhoff and @RobH as it seems we may need to revisit this decision

Tue, Apr 6, 7:51 AM · CAS-SSO, SRE, LDAP-Access-Requests
MoritzMuehlenhoff added a comment to T278905: Reconsider which mailman3 version we're running.

Our transition plan is to install mailman3 on the current mailman2 host (lists1001) so we can serve both from lists.wikimedia.org (see T278610: Install mailman3 on lists1001.wikimedia.org and T256539: Figure out a way to sync old and new mailman).

Tue, Apr 6, 6:16 AM · Patch-For-Review, SRE, Wikimedia-Mailing-lists
MoritzMuehlenhoff added a comment to T278905: Reconsider which mailman3 version we're running.

What's the timeline for the actual Mailman 3 migration? Early steps for making bullseye usable are ongoing and we'll be able to run a few machines on bullseye even before it's finally released (we've done that for buster before, I'm keeping an eye on security packages and those pilots installs manually and if necessary we pull sec updates from sid (before they migrated or roll out local fixes). If this can wait two more weeks (by then base should be ready), let's directly install the Mailman 3 systems with bullseye?

Tue, Apr 6, 6:03 AM · Patch-For-Review, SRE, Wikimedia-Mailing-lists

Thu, Apr 1

MoritzMuehlenhoff added a comment to T277064: Packaging PostGIS 3.1 for the new Maps stack.

@MSantos You can find a backport for buster at https://people.wikimedia.org/~jmm/postgis/ Can you run some tests whether that's what you need? If all is fine, I'll import them to a repository component on apt.wikimedia.org

Thu, Apr 1, 3:14 PM · Product-Infrastructure-Team-Backlog, SRE, Packaging, serviceops, Maps
MoritzMuehlenhoff created T279054: Integrate Buster 10.9 point update.
Thu, Apr 1, 11:46 AM · SRE
MoritzMuehlenhoff added a comment to T278371: wmf-auto-restart.py + lsof + /mnt/hdfs may need to be tuned.

Yep most of the times it works fine, but when the fuse process gets into its weird state then everything trying to access /mnt/hdfs stalls waiting for it (no idea how to trigger the behavior).

Thu, Apr 1, 6:11 AM · Analytics, SRE

Fri, Mar 26

MoritzMuehlenhoff added a comment to T278371: wmf-auto-restart.py + lsof + /mnt/hdfs may need to be tuned.

Hmmh, I added some debug output to wmf-auto-restart on an-launcher1002 and it seems to correctly pick up the config, the executed lsof command is

Fri, Mar 26, 8:26 AM · Analytics, SRE
MoritzMuehlenhoff added a comment to T278371: wmf-auto-restart.py + lsof + /mnt/hdfs may need to be tuned.

Is an-launcher in anyway different than the rest of Hadoop, like different mount options or so? We would have seen that error also happening on the rest of the Hadoop cluster, wouldn't we?

Fri, Mar 26, 8:15 AM · Analytics, SRE

Thu, Mar 25

MoritzMuehlenhoff removed a member for acl*sre-team: Phamhi.
Thu, Mar 25, 1:52 PM

Wed, Mar 24

MoritzMuehlenhoff closed T278292: eqiad: 1 VMs requested for ircd as Resolved.

VM has been created/install, further setup via T278255 once kraz is gone.

Wed, Mar 24, 2:13 PM · SRE, vm-requests
MoritzMuehlenhoff triaged T278312: Evaluate/integrate eatmydata in d-i as Medium priority.
Wed, Mar 24, 12:05 PM · User-MoritzMuehlenhoff, SRE
MoritzMuehlenhoff created T278312: Evaluate/integrate eatmydata in d-i.
Wed, Mar 24, 12:05 PM · User-MoritzMuehlenhoff, SRE
MoritzMuehlenhoff closed T277657: eqiad: 1 of VMs requested for tendril/buster as Resolved.

This has been created and is being used.

Wed, Mar 24, 9:34 AM · SRE, vm-requests
MoritzMuehlenhoff claimed T278292: eqiad: 1 VMs requested for ircd.
Wed, Mar 24, 9:33 AM · SRE, vm-requests
MoritzMuehlenhoff created T278292: eqiad: 1 VMs requested for ircd.
Wed, Mar 24, 9:33 AM · SRE, vm-requests
MoritzMuehlenhoff added a comment to T238707: Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch).

I've merged https://gerrit.wikimedia.org/r/674392 and shut down deployment-logstash2, it can be removed for good in a few days. Puppet was broken on this instance since September 2020, so if anything really still used it, it would probably be broken anyway...

Wed, Mar 24, 7:37 AM · Cloud-VPS (Debian Jessie Deprecation), Beta-Cluster-Infrastructure
MoritzMuehlenhoff added a comment to T224589: Migrate dbmonitor hosts to Buster.

tendril.w.o and dbtree.w.o are now served from dbmonitor1002.wikimedia.org running Buster. If there are any issues, we can fallback to dbmonitor1001 by reverting https://gerrit.wikimedia.org/r/674303

Wed, Mar 24, 7:10 AM · Patch-For-Review, SRE
MoritzMuehlenhoff updated the task description for T224589: Migrate dbmonitor hosts to Buster.
Wed, Mar 24, 7:09 AM · Patch-For-Review, SRE

Tue, Mar 23

MoritzMuehlenhoff added a comment to T238707: Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch).

Shall we just wholesale point these to deployment-logstash03? Even if some turn out to be unused or broken, that's still better than sending them to a server which will soon need to be removed :-)

Likely yes, but I'm not a project admin on those projects and have not found time or motivation go thru all of them and contact their maintainers.

Tue, Mar 23, 3:52 PM · Cloud-VPS (Debian Jessie Deprecation), Beta-Cluster-Infrastructure
MoritzMuehlenhoff added a comment to T238707: Migrate from deployment-logstash2 (jessie) to deployment-logstash03 (stretch).

Note: other Cloud VPS projects (wikidata-query, striker, ores, phabricator) appear to also be using deployment-logstash2. Not sure if they are actually using it but those at least have hiera keys pointing to logstash2.

Tue, Mar 23, 11:00 AM · Cloud-VPS (Debian Jessie Deprecation), Beta-Cluster-Infrastructure
MoritzMuehlenhoff closed T269558: Integrate Buster 10.7 point update as Resolved.

This is complete

Tue, Mar 23, 9:36 AM · SRE
MoritzMuehlenhoff updated the task description for T269558: Integrate Buster 10.7 point update.
Tue, Mar 23, 9:35 AM · SRE
MoritzMuehlenhoff updated the task description for T269558: Integrate Buster 10.7 point update.
Tue, Mar 23, 9:16 AM · SRE
MoritzMuehlenhoff updated the task description for T274099: Integrate Buster 10.8 point update.
Tue, Mar 23, 9:16 AM · SRE

Mon, Mar 22

MoritzMuehlenhoff added a comment to T277537: ganeti2015 doesn't boot.

Thanks, Papaul

Mon, Mar 22, 4:08 PM · ops-codfw, SRE
MoritzMuehlenhoff updated the task description for T263974: Integrate Buster 10.6 point update.
Mon, Mar 22, 11:00 AM · SRE
MoritzMuehlenhoff updated the task description for T263974: Integrate Buster 10.6 point update.
Mon, Mar 22, 10:57 AM · SRE
MoritzMuehlenhoff renamed T276843: Bundled pygments in REL1_31 / REL1_35 vulnerable to CVE-2021-20270 and CVE-2021-27291 from Bundled pygments in REL1_31 / REL1_35 vulnerable to CVE-2021-20270 to Bundled pygments in REL1_31 / REL1_35 vulnerable to CVE-2021-20270 and CVE-2021-27291.
Mon, Mar 22, 10:49 AM · Patch-For-Review, MW-1.35-notes, MW-1.36-notes (1.36.0-wmf.32; 2021-02-23), SecTeam-Processed, SyntaxHighlight, Vuln-DoS, Security, Security-Team
MoritzMuehlenhoff added a comment to T276843: Bundled pygments in REL1_31 / REL1_35 vulnerable to CVE-2021-20270 and CVE-2021-27291.

There's a second CVE ID for pygments: CVE-2021-27291
https://gist.github.com/b-c-ds/b1a2cc0c68a35c57188575eb496de5ce and fixed via https://github.com/pygments/pygments/commit/2e7e8c4a7b318f4032493773732754e418279a14

Mon, Mar 22, 10:48 AM · Patch-For-Review, MW-1.35-notes, MW-1.36-notes (1.36.0-wmf.32; 2021-02-23), SecTeam-Processed, SyntaxHighlight, Vuln-DoS, Security, Security-Team

Fri, Mar 19

MoritzMuehlenhoff triaged T277841: WebAuthn FIDO2 support in CAS as Medium priority.
Fri, Mar 19, 1:10 PM · CAS-SSO, SRE
MoritzMuehlenhoff triaged T277840: CAS per-service TGT setting as Low priority.
Fri, Mar 19, 1:10 PM · CAS-SSO, SRE
MoritzMuehlenhoff triaged T277837: Investigate/enable new actuators for U2F token management as Low priority.
Fri, Mar 19, 1:10 PM · CAS-SSO, SRE

Mar 19 2021

MoritzMuehlenhoff added a comment to T274736: Grafana may allow access to API unauthenticated [CVE-2021-27358].

This eventually got assigned CVE-2021-27358.

Mar 19 2021, 11:47 AM · Upstream, Security-Team, User-RhinosF1, Vuln-DoS, observability, Security
MoritzMuehlenhoff added a comment to T218729: Migrate deployment-prep away from Debian Jessie to Debian Stretch/Buster.

I think we can simply remove deployment-sca01/sca02? The respective hosts in production have been removed (hardware is still up, but services gone) and removal doesn't need to wait for an eventual k8s installation in beta.

I think a few services that are still running there, at least apertium and recommendation api according to a quick codesearch and a look at deployment-prep proxy list.

Mar 19 2021, 10:48 AM · Cloud-VPS (Debian Jessie Deprecation), Beta-Cluster-Infrastructure
MoritzMuehlenhoff added a comment to T224579: Migrate irc.wikimedia.org/kraz to Buster.

Sure enough, the exporter is out of FDs again. I'm +1 to just remove the exporter since the service doesn't have an owner, the exporter is python2 and afaict we use the metrics anyways. Thoughts ?

Mar 19 2021, 8:49 AM · Patch-For-Review, User-notice, Wikimedia-IRC-RC-Server, SRE
MoritzMuehlenhoff added a comment to T271684: Update CAS to 6.3.

I filed tasks for new features introduced in 6.3:
https://phabricator.wikimedia.org/T277837
https://phabricator.wikimedia.org/T277840
https://phabricator.wikimedia.org/T277841

Mar 19 2021, 8:22 AM · CAS-SSO, SRE
MoritzMuehlenhoff created T277841: WebAuthn FIDO2 support in CAS.
Mar 19 2021, 8:16 AM · CAS-SSO, SRE
MoritzMuehlenhoff created T277840: CAS per-service TGT setting.
Mar 19 2021, 7:58 AM · CAS-SSO, SRE
MoritzMuehlenhoff added a comment to T277837: Investigate/enable new actuators for U2F token management.

Same for "A number of new administrative actuator endpoints are presented to report back on the registered authentication handlers and policies."

Mar 19 2021, 7:45 AM · CAS-SSO, SRE
MoritzMuehlenhoff created T277837: Investigate/enable new actuators for U2F token management.
Mar 19 2021, 7:44 AM · CAS-SSO, SRE

Mar 17 2021

MoritzMuehlenhoff added a comment to T273727: Story idea for Blog: Rollout of single-sign-on (SSO) at the Wikimedia Foundation.

There's still part 2 and 3 coming (but rather next month), should we reopen the task when ready or make a new one?

Mar 17 2021, 2:46 PM · Technical-blog-posts
MoritzMuehlenhoff created T277657: eqiad: 1 of VMs requested for tendril/buster.
Mar 17 2021, 1:04 PM · SRE, vm-requests

Mar 16 2021

MoritzMuehlenhoff created T277537: ganeti2015 doesn't boot.
Mar 16 2021, 1:53 PM · ops-codfw, SRE
MoritzMuehlenhoff added a comment to T224589: Migrate dbmonitor hosts to Buster.

The problem was indeed mysqli, we can try to see if we can run php5 on stretch as you propose.
@jcrespo took a deep look at this a couple of years ago I think, so maybe he can give more context on what he saw at the time (other than what's already on this task) at T224589#5597729 and T224589#5598014

Mar 16 2021, 1:51 PM · Patch-For-Review, SRE
MoritzMuehlenhoff added a comment to T224589: Migrate dbmonitor hosts to Buster.

Ouch, let's move to dbmonitor to Stretch, then? If PHP 5 is the blocker (I remember some issues with PHP7 vaguely), I can make a stretch-wikimedia build of php5, but this really, really needs to move away from jessie: jessie is EOLed for three quarters now and we spend a lot of time on backporting security fixes for jessie-wikimedia internally and this really needs to end now.

Mar 16 2021, 1:42 PM · Patch-For-Review, SRE
MoritzMuehlenhoff added a comment to T224589: Migrate dbmonitor hosts to Buster.

With orchestrator in place, can these be removed now? Support for jessie will cease in two weeks.

Mar 16 2021, 1:36 PM · Patch-For-Review, SRE
MoritzMuehlenhoff added a comment to T244849: Add SSO support to netbox.

I think that it's required to avoid the security issue of a user removed from an LDAP group keeping the previous access and the usability issue of a user that was added to a more privileged group that will not gain the expected privileges.
@jbond thoughts?

Mar 16 2021, 11:12 AM · Patch-For-Review, netbox, SRE
MoritzMuehlenhoff reopened T216611: Icinga check for ircecho should check for actual activity as "Open".
Mar 16 2021, 10:31 AM · IRCecho, observability, Icinga, SRE
MoritzMuehlenhoff added a comment to T216611: Icinga check for ircecho should check for actual activity.

This has been flapping in Icinga, e.g. for today:

Mar 16 2021, 10:31 AM · IRCecho, observability, Icinga, SRE
MoritzMuehlenhoff updated the task description for T269354: Switch Jenkins servers to Java 11.
Mar 16 2021, 8:43 AM · Patch-For-Review, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Release-Engineering-Team (CI & Testing services), Jenkins, Continuous-Integration-Infrastructure

Mar 12 2021

MoritzMuehlenhoff added a comment to T247966: Migrate role::alerting_host to Buster.

What's up with icinga1001/icinga2001, they are still up and running?

Mar 12 2021, 9:10 AM · Patch-For-Review, observability

Mar 11 2021

MoritzMuehlenhoff added a comment to T275294: ((OTRS)) Community Edition 6 is end-of-life; no FOSS replacement provided.

There's now a group of companies related to OTRS which will be collaborating on Znuny: https://www.otter-alliance.de/en/die-allianz.html

Mar 11 2021, 10:09 PM · User-notice, SRE, Security, OTRS

Mar 10 2021

MoritzMuehlenhoff added a comment to T273727: Story idea for Blog: Rollout of single-sign-on (SSO) at the Wikimedia Foundation.

This has been published: https://techblog.wikimedia.org/2021/03/10/wikimedia-sso-evaluation/

Let me know if it looks good to you, and I'll send an announcement out!

Mar 10 2021, 9:45 PM · Technical-blog-posts
MoritzMuehlenhoff added a comment to T277064: Packaging PostGIS 3.1 for the new Maps stack.

Pasting in my comments from IRC (which are based on a very quick look):
[16:33] <moritzm> bullseye has 3.1, but it's hard to tell how complex a backport will be, given that maps is still on stretch
[16:33] <moritzm> and the list of build deps in 3.1 is not small
[16:33] <moritzm> and includes things like protobuf and libgdal
[16:34] <moritzm> I think this can only really be estimated if someone gives it a shot for half an hour

Mar 10 2021, 5:34 PM · Product-Infrastructure-Team-Backlog, SRE, Packaging, serviceops, Maps
MoritzMuehlenhoff added a comment to T273727: Story idea for Blog: Rollout of single-sign-on (SSO) at the Wikimedia Foundation.

Perfect! I plan on publishing this tomorrow 9 March 2021.

Mar 10 2021, 10:43 AM · Technical-blog-posts