Page MenuHomePhabricator
Feed Advanced Search

Yesterday

MoritzMuehlenhoff added a member for acl*sre-team: dcaro.
Wed, Nov 25, 12:04 PM
MoritzMuehlenhoff triaged T268725: Include mail on standard_packages.pp as Medium priority.
Wed, Nov 25, 9:36 AM · User-MoritzMuehlenhoff, Mail, Puppet, Operations

Tue, Nov 24

MoritzMuehlenhoff updated the task description for T245757: Upgrade MediaWiki appservers to Debian Buster (debian 10).
Tue, Nov 24, 3:26 PM · User-jijiki, Patch-For-Review, Operations, Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, serviceops
MoritzMuehlenhoff added a comment to T268327: Request new database for idp.wikimedia.org.

@jbond we already have a cas user that has access to cas_staging database. Do you want to re-use that user/password or use a different one for the new casdatabase?

I think its fine to use the same username/password but tagging @MoritzMuehlenhoff in case he disagrees (moritz this is for the u2f registrations)

Tue, Nov 24, 10:21 AM · User-jbond, DBA, CAS-SSO, Operations
MoritzMuehlenhoff added a comment to T245757: Upgrade MediaWiki appservers to Debian Buster (debian 10).

With all Puppet patches and debs landed, mwdebug1003 should be reimaged again, there were plenty of manual intermediate steps until we made it work, and that way we can conclusively confirm that it works.

Tue, Nov 24, 8:41 AM · User-jijiki, Patch-For-Review, Operations, Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, serviceops
MoritzMuehlenhoff added a comment to T245757: Upgrade MediaWiki appservers to Debian Buster (debian 10).

With all Puppet patches and debs landed, mwdebug1003 should be reimaged again, there were plenty of manual intermediate steps until we made it work, and that way we can conclusively confirm that it works.

Tue, Nov 24, 8:31 AM · User-jijiki, Patch-For-Review, Operations, Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, serviceops

Mon, Nov 23

MoritzMuehlenhoff updated the task description for T245757: Upgrade MediaWiki appservers to Debian Buster (debian 10).
Mon, Nov 23, 2:15 PM · User-jijiki, Patch-For-Review, Operations, Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, serviceops
MoritzMuehlenhoff added a comment to T245757: Upgrade MediaWiki appservers to Debian Buster (debian 10).

While the packages above are installed which is a nice step, we now still have these issues to solve:

Mon, Nov 23, 2:14 PM · User-jijiki, Patch-For-Review, Operations, Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, serviceops
MoritzMuehlenhoff created T268468: captcha.py needs to be ported to Python 3.
Mon, Nov 23, 1:01 PM · Python3-Porting, ConfirmEdit (CAPTCHA extension), Operations

Fri, Nov 20

MoritzMuehlenhoff added a comment to T265857: Update CAS to 6.2.

An overview of changes that landed in 6.2 can be found here:
https://apereo.github.io/2019/11/29/620rc1-release/
https://apereo.github.io/2019/12/27/620rc2-release/
https://apereo.github.io/2020/03/03/620rc3-release/
https://apereo.github.io/2020/04/17/620rc4-release/
https://apereo.github.io/2020/05/29/620rc5-release/

Fri, Nov 20, 4:11 PM · Patch-For-Review, CAS-SSO, Operations
MoritzMuehlenhoff added a comment to T266314: Decom cookbook should also remove keytabs and principals.

Agreed. Printing the steps is a good first step which will prevent that these files slip through. Mid-term it would be nice to have a dedicated Spicerack module for Kerberos, which wraps common kadmin commands and makes them available for cook books.

Fri, Nov 20, 8:52 AM · Patch-For-Review, User-MoritzMuehlenhoff, Operations

Thu, Nov 19

MoritzMuehlenhoff triaged T268243: Broken package state on cp4032 as Medium priority.
Thu, Nov 19, 3:43 PM · Traffic, Operations
MoritzMuehlenhoff created T268243: Broken package state on cp4032.
Thu, Nov 19, 3:43 PM · Traffic, Operations

Wed, Nov 18

MoritzMuehlenhoff added a comment to T267607: upgrade mwmaint1002 to buster.

We should have two mwmaint servers per DC anyway (with some mechanism to flip the active one), some failover capability is needed outside of OS updates as well (reboots e.g. are a total pain with the current SPOF setup that we have)

Wed, Nov 18, 7:23 PM · Operations, serviceops

Mon, Nov 16

MoritzMuehlenhoff added a comment to T267917: LDAP 'nda' access for Tobias Schumann.

JFTR, this would be for the cn=nda LDAP group, not cn=wmf.

Mon, Nov 16, 12:23 PM · Operations, LDAP-Access-Requests

Thu, Nov 12

MoritzMuehlenhoff added a comment to T256367: WMF-NDA access for DannyS712.

I've also added DannyS712 to WMF-NDA now

Thu, Nov 12, 2:33 PM · User-DannyS712, WMF-NDA-Requests
MoritzMuehlenhoff closed T256367: WMF-NDA access for DannyS712 as Resolved.
Thu, Nov 12, 2:33 PM · User-DannyS712, WMF-NDA-Requests
MoritzMuehlenhoff added a member for WMF-NDA: DannyS712.
Thu, Nov 12, 2:26 PM
Kizule awarded T256367: WMF-NDA access for DannyS712 a Like token.
Thu, Nov 12, 1:46 PM · User-DannyS712, WMF-NDA-Requests
MoritzMuehlenhoff closed T256367: WMF-NDA access for DannyS712 as Resolved.

@DannyS712 You've been added to cn=nda, I'm closing the task, but please reopen if you run into any issues :-)

Thu, Nov 12, 1:25 PM · User-DannyS712, WMF-NDA-Requests
MoritzMuehlenhoff added a comment to T256367: WMF-NDA access for DannyS712.

Everything seems to be in place now, I'm adding DannyS712 to cn=nda.

Thu, Nov 12, 1:17 PM · User-DannyS712, WMF-NDA-Requests

Tue, Nov 10

MoritzMuehlenhoff added a comment to T264991: Upgrade the MediaWiki servers to ICU 63.

Sounds good!

Tue, Nov 10, 3:42 PM · Patch-For-Review, Beta-Cluster-Infrastructure, DBA, Operations, serviceops
MoritzMuehlenhoff added a project to T245757: Upgrade MediaWiki appservers to Debian Buster (debian 10): Operations.
Tue, Nov 10, 2:31 PM · User-jijiki, Patch-For-Review, Operations, Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, serviceops
MoritzMuehlenhoff added a comment to T245757: Upgrade MediaWiki appservers to Debian Buster (debian 10).

Before we start reimaging, let's also merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/445604/

Tue, Nov 10, 2:27 PM · User-jijiki, Patch-For-Review, Operations, Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, serviceops
MoritzMuehlenhoff added a comment to T256367: WMF-NDA access for DannyS712.

Thanks Katie!

Tue, Nov 10, 12:18 PM · User-DannyS712, WMF-NDA-Requests
MoritzMuehlenhoff updated the task description for T264991: Upgrade the MediaWiki servers to ICU 63.
Tue, Nov 10, 8:36 AM · Patch-For-Review, Beta-Cluster-Infrastructure, DBA, Operations, serviceops
MoritzMuehlenhoff added a comment to T264991: Upgrade the MediaWiki servers to ICU 63.

One other sanity check for the rollout (in particular when the whole server batch gets upgraded on the 16th);

Tue, Nov 10, 8:09 AM · Patch-For-Review, Beta-Cluster-Infrastructure, DBA, Operations, serviceops

Mon, Nov 9

MoritzMuehlenhoff added a comment to T264991: Upgrade the MediaWiki servers to ICU 63.

That's not enough, these are just the binary packages from the php72 source package, but you also need to upgrade php-apcu, php-cli, php-common, php-excimer, php-geoip, php-igbinary, php-luasandbox, php-memcached, php-mongodb, php-msgpack, php-redis, php-tideways-xhprof and php-wmerrors

Mon, Nov 9, 6:27 PM · Patch-For-Review, Beta-Cluster-Infrastructure, DBA, Operations, serviceops
MoritzMuehlenhoff created T267515: Check home/HDFS leftovers of niedzielski.
Mon, Nov 9, 8:08 AM · Analytics
MoritzMuehlenhoff closed T262512: Enable CAS authentication for Grafana as Resolved.
Mon, Nov 9, 8:05 AM · User-fgiunchedi, Patch-For-Review, observability, Operations

Fri, Nov 6

MoritzMuehlenhoff added a comment to T262512: Enable CAS authentication for Grafana.

There's now:

  • A separate vhost grafana-rw.wikimedia.org using CAS to be used for editing dashboards and internal settings
  • grafana.wikimedia.org remains as-is, but the "Sign in" button now redirects to grafana-rw.wikimedia.org
  • There's a daily sync of editor/admin permissions based on what's stored in LDAP groups into the internal Grafana sqlite database.
Fri, Nov 6, 8:25 AM · User-fgiunchedi, Patch-For-Review, observability, Operations

Thu, Nov 5

MoritzMuehlenhoff added a comment to T267292: dcaro has same ssh key in wmcs and prod, prod ssh key revoked.

Here is a new one:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID46/gY7mfN96ylAdQb6ZBfrq9L3QwemMtN5ZjrJgEmK dcaro@magnum

Would it be possible to know where/when did I use it? (I'd like to avoid using it the wrong way again)

Thu, Nov 5, 8:12 AM · Operations, SRE-Access-Requests

Wed, Nov 4

MoritzMuehlenhoff added a comment to T256367: WMF-NDA access for DannyS712.

@KFrancis This is for the cn=ldap LDAP group.

Wed, Nov 4, 7:51 PM · User-DannyS712, WMF-NDA-Requests

Mon, Nov 2

MoritzMuehlenhoff closed T264388: Migrate LDAP replicas to Buster as Resolved.

ldap-replica1001/1002/2003/2004 are now running Buster, old Stretch instances have been removed.

Mon, Nov 2, 11:55 AM · Patch-For-Review, Operations

Thu, Oct 29

MoritzMuehlenhoff updated the task description for T263974: Integrate Buster 10.6 point update.
Thu, Oct 29, 3:35 PM · Operations
MoritzMuehlenhoff closed T264176: Switch Zookeeper to profile::java as Resolved.
Thu, Oct 29, 2:40 PM · Analytics-Kanban, Analytics-Clusters, Operations
MoritzMuehlenhoff closed T264176: Switch Zookeeper to profile::java, a subtask of T264174: Migrate remaining services using Java to profile::java , as Resolved.
Thu, Oct 29, 2:40 PM · Operations
MoritzMuehlenhoff created T266782: Updated java security policy in OpenJDK 11.9.
Thu, Oct 29, 12:42 PM · Patch-For-Review, Operations
MoritzMuehlenhoff placed T201779: Have a check to prevent non-existent accounts from being added to LDAP groups up for grabs.
Thu, Oct 29, 10:30 AM · User-MoritzMuehlenhoff, Security, LDAP, Operations
MoritzMuehlenhoff placed T168433: Deprecate DSA (ssh-dss) SSH keys for Cloud VPS and Toolforge users up for grabs.
Thu, Oct 29, 10:29 AM · User-MoritzMuehlenhoff, Cloud-VPS, Toolforge, cloud-services-team (Kanban)
MoritzMuehlenhoff closed T148986: Firewall sets not being loaded post-reboot due to a @resolve race as Resolved.

We haven't seen these for a while to be a general problem. Also, there's monitoring in place, so if it happens again we can revisit specific cases. Closing.

Thu, Oct 29, 10:28 AM · Operations

Wed, Oct 28

MoritzMuehlenhoff closed T266086: Nuria's volunteer account as Resolved.

Excellent, thanks! Closing this task since everything is completed now. I'll merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/636936/ (along with updating LDAP groups) after Nuria's last day.

Wed, Oct 28, 2:41 PM · Analytics-Radar, Operations, SRE-Access-Requests
MoritzMuehlenhoff updated subscribers of T256367: WMF-NDA access for DannyS712.

Indeed - given how long this NDA has taken, if possible I'd like to start the process for the logstash one now

{{ping}} just want to make sure you saw this @MoritzMuehlenhoff

Wed, Oct 28, 12:22 PM · User-DannyS712, WMF-NDA-Requests
MoritzMuehlenhoff added a comment to T266656: Run orchestrator as non-root.

If this is solely about the need to bind to a privileged port,

Wed, Oct 28, 11:55 AM · Orchestrator, Operations, DBA
MoritzMuehlenhoff added a comment to T264176: Switch Zookeeper to profile::java.

One gotcha: conf1* is still on jessie (and consequently Java 7), and I don't think anything accounts for Java 7 yet

Wed, Oct 28, 9:14 AM · Analytics-Kanban, Analytics-Clusters, Operations

Tue, Oct 27

MoritzMuehlenhoff closed Restricted Task, a subtask of T116750: 2FA for SSH access to the production cluster, as Declined.
Tue, Oct 27, 3:45 PM · Operations
MoritzMuehlenhoff added a comment to T266479: Puppet Proposal to remove require_package.

Sounds good to me

Tue, Oct 27, 1:32 PM · Patch-For-Review, Operations, Puppet
MoritzMuehlenhoff added a comment to T266086: Nuria's volunteer account.

@gsingers We have three major types of NDA/MOU under which people get access to PII-sensitive data on our servers:

  • Everyone who's WMF staff has signed an NDA as part of their work contract
  • Researchers get access to some data after signing a MOU (with a person on the WMF side being a point of contact and time-limited until the research is completed)
  • Some people from the community have signed an NDA (called the Volunteer NDA in contrast to the NDA which applies to staff) which allows them to e.g. debug problems in production, deploy code changes, review Logstash/Turnilo etc.
Tue, Oct 27, 10:01 AM · Analytics-Radar, Operations, SRE-Access-Requests

Oct 26 2020

MoritzMuehlenhoff renamed T266479: Puppet Proposal to remove require_package from Puppet Proposal to remove require_packages to Puppet Proposal to remove require_package.
Oct 26 2020, 3:30 PM · Patch-For-Review, Operations, Puppet
MoritzMuehlenhoff created T266467: Check home/HDFS leftovers of rodolfovalentim.
Oct 26 2020, 12:43 PM · Analytics
MoritzMuehlenhoff added a comment to T248168: Upgrade Puppet to 5.5.21.

Debian unstable was updated to 5.5.22:
https://packages.qa.debian.org/p/puppet/news/20201025T173952Z.html

Oct 26 2020, 12:13 PM · Puppet, Operations
MoritzMuehlenhoff closed T164819: reprepro: Support for buildinfo files / dbgsym packages as Resolved.

dbgsym files are supported in reprepro for quite a while now and as of today, we can also install dbgsym packages from the Debian archive. Closing.

Oct 26 2020, 12:03 PM · Patch-For-Review, Operations
MoritzMuehlenhoff added a comment to T266086: Nuria's volunteer account.

Mentioned in SAL (#wikimedia-operations) [2020-10-23T22:56:29Z] <mutante> added Nuria to "nda" LDAP group - leaving her in "wmf" until the actual last day - shell account remains so no puppet change needed in ldap_only_admins (T266086)

cc: @KFrancis You have it on file in the right system as well?

Oct 26 2020, 8:02 AM · Analytics-Radar, Operations, SRE-Access-Requests

Oct 23 2020

MoritzMuehlenhoff claimed T264605: Apereo CAS expose CASCookieSameSite via profile::idp::client::http .

I've built an updated mod_cas package with SameSite cookie support for buster-wikimedia (not imported yet to apt.wikimedia.org), will run some tests next week.

Oct 23 2020, 1:10 PM · Operations, User-jbond, CAS-SSO
MoritzMuehlenhoff added a project to T266314: Decom cookbook should also remove keytabs and principals: User-MoritzMuehlenhoff.
Oct 23 2020, 8:19 AM · Patch-For-Review, User-MoritzMuehlenhoff, Operations
MoritzMuehlenhoff created T266314: Decom cookbook should also remove keytabs and principals.
Oct 23 2020, 8:19 AM · Patch-For-Review, User-MoritzMuehlenhoff, Operations
MoritzMuehlenhoff closed T158562: Manage apt sources via puppet as Resolved.

/etc/apt/sources.list is managed by Puppet since a few weeks in production, closing the task (for Cloud VPS it's being considered to also enabled it in a separate task).

Oct 23 2020, 7:27 AM · Patch-For-Review, User-jbond, Operations

Oct 22 2020

MoritzMuehlenhoff added a comment to T264388: Migrate LDAP replicas to Buster.

All new buster replicas are now pooled and the stretch ones have been depooled. I'll keep them around for another week just in case, then they are going to be removed.

Oct 22 2020, 2:58 PM · Patch-For-Review, Operations
MoritzMuehlenhoff added a comment to T266023: orchestrator: Get packages into WMF apt.

@MoritzMuehlenhoff: Is it acceptable to download the pre-build .debs, and upload them into our apt repo?

Oct 22 2020, 8:54 AM · Operations, User-Kormat, DBA

Oct 21 2020

MoritzMuehlenhoff closed T259519: Integrate Buster 10.5 point release as Resolved.

This is complete

Oct 21 2020, 3:00 PM · Operations
MoritzMuehlenhoff updated the task description for T259519: Integrate Buster 10.5 point release.
Oct 21 2020, 3:00 PM · Operations
MoritzMuehlenhoff created T266147: Port prometheus-openldap-exporter to Python 3.
Oct 21 2020, 2:51 PM · Python3-Porting, LDAP, Operations
MoritzMuehlenhoff created T266118: Revisit use of swap and related kernel settings.
Oct 21 2020, 11:59 AM · User-MoritzMuehlenhoff, Operations
MoritzMuehlenhoff added a project to T266106: orchestrator: Support SSO: CAS-SSO.
Oct 21 2020, 10:21 AM · Orchestrator, CAS-SSO, Operations, User-Kormat, DBA

Oct 19 2020

MoritzMuehlenhoff updated the task description for T265147: Offboard Chase Pettet from Security Team.
Oct 19 2020, 4:24 PM · Operations, Security-Team
MoritzMuehlenhoff added a comment to T265147: Offboard Chase Pettet from Security Team.

I've removed Chase from SRE-Access-Requests

Oct 19 2020, 4:23 PM · Operations, Security-Team
MoritzMuehlenhoff removed a member for SRE-Access-Requests: chasemp.
Oct 19 2020, 4:23 PM
MoritzMuehlenhoff updated the task description for T263974: Integrate Buster 10.6 point update.
Oct 19 2020, 3:11 PM · Operations
MoritzMuehlenhoff updated the task description for T263974: Integrate Buster 10.6 point update.
Oct 19 2020, 11:26 AM · Operations
MoritzMuehlenhoff added a comment to T264255: Review request for data export.

@MoritzMuehlenhoff, would it be ok to temporarily re-add @Groceryheist's access while he copies out some data?

Oct 19 2020, 10:27 AM · Analytics-Kanban, Security, Analytics
MoritzMuehlenhoff closed T159750: E-mail for people in different OIT LDAP object unit as Declined.

We can close this task given that the OpenLDAP mirror in going away in favour of JumpCloud

Oct 19 2020, 10:23 AM · WMF-Office-IT, Operations, Mail
MoritzMuehlenhoff closed T262647: Provide failover capacity for package installations from main mirror as Resolved.

This is complete: The package mirror can now be set via profile::base::mirror_server (and still defaults to mirrors.wikimedia.org)

Oct 19 2020, 9:55 AM · Operations
MoritzMuehlenhoff created T265857: Update CAS to 6.2.
Oct 19 2020, 6:48 AM · Patch-For-Review, CAS-SSO, Operations
MoritzMuehlenhoff closed T264472: Requesting access to researchers and analytics-privatedata-users for Leila Zia as Resolved.

For the followup work with the old home there's T264994, so we can close this.

Oct 19 2020, 6:45 AM · SRE-Access-Requests, Operations

Oct 16 2020

MoritzMuehlenhoff added a comment to T253377: WMF deployed EasyTimeline extension depends on Ploticus package which is not available in Debian Buster (but available again in Debian Bullseye).

I've uploaded an NMU (2.42-4.2) for ploticus which correctly enables the toolchain hardening (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=967239), we can import that version to buster-wikimedia for the mw migration of the app servers.

Oct 16 2020, 3:00 PM · Operations, EasyTimeline, Packaging
MoritzMuehlenhoff added a project to T253377: WMF deployed EasyTimeline extension depends on Ploticus package which is not available in Debian Buster (but available again in Debian Bullseye): Operations.
Oct 16 2020, 2:58 PM · Operations, EasyTimeline, Packaging
MoritzMuehlenhoff added a comment to T256367: WMF-NDA access for DannyS712.

Let me clarify this and then I'll report back to this task in the next days.

Oct 16 2020, 10:47 AM · User-DannyS712, WMF-NDA-Requests
MoritzMuehlenhoff added a comment to T265590: ulog: filter out diffscan from ulog.

The servers with a public IP already have lots of noise from random bots/portscans (e.g. on bast3004 40kish log per day), so this doesn't make too much of a difference.

Oct 16 2020, 7:42 AM · observability, Security, Operations, netops, User-jbond
MoritzMuehlenhoff closed T264182: Migrate Gerrit to profile::java, a subtask of T264174: Migrate remaining services using Java to profile::java , as Resolved.
Oct 16 2020, 7:24 AM · Operations
MoritzMuehlenhoff closed T264182: Migrate Gerrit to profile::java as Resolved.

The Gerrit servers have been switched to profile::java (which allowed for quite a few cleanups)

Oct 16 2020, 7:24 AM · Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Release-Engineering-Team (Development services), Gerrit, Operations
MoritzMuehlenhoff added a comment to T265689: Requesting adding rust-ripgrep and rust-fd-find on the gridengine nodes.

FWIW, the dependencies on a Buster system look fine to me and are fulfilled in stretch as well (Stretch has libgcc1 6.3.0 and libc6 2.24):

Oct 16 2020, 7:20 AM · cloud-services-team (Kanban), Patch-For-Review, Toolforge (Software install/update)

Oct 14 2020

MoritzMuehlenhoff created T265447: Check home/HDFS leftovers of joewalsh.
Oct 14 2020, 7:17 AM · Analytics

Oct 13 2020

MoritzMuehlenhoff added a comment to T256367: WMF-NDA access for DannyS712.

From what I can tell the procedure described in https://wikitech.wikimedia.org/wiki/Volunteer_NDA is outdated and no longer accurate. All current NDA access requires an NDA signed with the Legal department (it's still a digital signture, but different from clicking https://phabricator.wikimedia.org/L2.

Oct 13 2020, 8:56 AM · WMF-NDA-Requests, User-DannyS712

Oct 12 2020

MoritzMuehlenhoff added a comment to T253377: WMF deployed EasyTimeline extension depends on Ploticus package which is not available in Debian Buster (but available again in Debian Bullseye).

FWIW ploticus was fixed and will be in the next Debian stable release (bullseye), but it missed buster. In theory we could maintain our own backport of it, but obviously sunsetting EasyTimeline is a better long term solution.

Oct 12 2020, 8:02 AM · Operations, EasyTimeline, Packaging
MoritzMuehlenhoff claimed T250515: Please provide our special component/php72 in buster-wikimedia.

realistically we'll only approach this once we move production to Buster.

Oct 12 2020, 7:21 AM · Patch-For-Review, Packaging, serviceops
MoritzMuehlenhoff added a comment to T264991: Upgrade the MediaWiki servers to ICU 63.

ii prometheus-nutcracker-exporter 0.2+nmu1 all Prometheus exporter for Nutcracker

Oct 12 2020, 7:18 AM · Patch-For-Review, Beta-Cluster-Infrastructure, DBA, Operations, serviceops
MoritzMuehlenhoff added a comment to T264991: Upgrade the MediaWiki servers to ICU 63.
Oct 12 2020, 7:17 AM · Patch-For-Review, Beta-Cluster-Infrastructure, DBA, Operations, serviceops

Oct 9 2020

MoritzMuehlenhoff added a comment to T265147: Offboard Chase Pettet from Security Team.

I have taken care of removing SSH access, LDAP, Networking, Icinga pwstore and some of the mail aliases configured in SRE (root@ etc.). earlier today.

Oct 9 2020, 2:41 PM · Operations, Security-Team
MoritzMuehlenhoff updated the task description for T265147: Offboard Chase Pettet from Security Team.
Oct 9 2020, 2:38 PM · Operations, Security-Team
MoritzMuehlenhoff added a comment to T264991: Upgrade the MediaWiki servers to ICU 63.

After a lot of fist shaking and head scratching I think I've found a workable solution, to the problem that PHP build depends on ICU 63 (for intl) and indirectly to ICU57 via libxml: I tried a few hacks with double-building (exploiting the fact the icu 63 uses pkgconfig, while ICU 57 still uses the old-school "icu-config", but it didn't work out, too complex and too many corner cases. Rebuilding all the libxml2 reverse deps with ICU 63 is also a fragile/error-prone undertaking since there's multiple levels of dependencies in the reverse deps.

Oct 9 2020, 1:34 PM · Patch-For-Review, Beta-Cluster-Infrastructure, DBA, Operations, serviceops
MoritzMuehlenhoff removed a member for acl*sre-team: chasemp.
Oct 9 2020, 9:38 AM
MoritzMuehlenhoff created T265121: Check home/HDFS leftovers of rush.
Oct 9 2020, 9:16 AM · Analytics
MoritzMuehlenhoff added a comment to T264888: Review default ferm INPUT policy.

however i, like @BBlack, prefer reject to drop if possible. As such it would be nice to be good internet citizens and move ahead with the initial proposal even if we can get diffscan to work with a default drop.

Oct 9 2020, 7:31 AM · Patch-For-Review, Security, Operations, netops, User-jbond
MoritzMuehlenhoff added a comment to T264472: Requesting access to researchers and analytics-privatedata-users for Leila Zia.

The "leila" account also needs to be removed from the wmf LDAP group.

Oct 9 2020, 7:26 AM · Operations, SRE-Access-Requests

Oct 8 2020

MoritzMuehlenhoff created T265038: Jenkins plugins security advisory - 2020-10-08.
Oct 8 2020, 1:02 PM · Continuous-Integration-Infrastructure, Jenkins, Security
MoritzMuehlenhoff updated subscribers of T264888: Review default ferm INPUT policy.
Oct 8 2020, 12:13 PM · Patch-For-Review, Security, Operations, netops, User-jbond
MoritzMuehlenhoff added a comment to T264991: Upgrade the MediaWiki servers to ICU 63.

I've created a standalone backport of icu63 in the component/icu63. Rebuilding PHP 7.2 with it is a little tricky, since PHP build-depends on libxml2 (for php7.2-xml), which in itself uses ICU. Also rebuilding libxml2 with ICU would require to test/adapt/rebuild a long list of reverse dependencies (and possible second order dependencies). Ideally this can be avoided, I'm testing a few options today and tomorrow.

Oct 8 2020, 8:17 AM · Patch-For-Review, Beta-Cluster-Infrastructure, DBA, Operations, serviceops

Oct 7 2020

MoritzMuehlenhoff added a comment to T264888: Review default ferm INPUT policy.

Agreed, the service enumation/information disclosure angle is moot for us, so let's give this a shot. If we make it configurable via Hiera we can also test it beforehand on a few hosts.

Oct 7 2020, 2:52 PM · Patch-For-Review, Security, Operations, netops, User-jbond
MoritzMuehlenhoff updated subscribers of T264889: Degraded RAID on wdqs1009.
Oct 7 2020, 2:24 PM · ops-eqiad, Operations

Oct 6 2020

Dzahn awarded T210993: Deprecate Diamond collectors in Cloud VPS a Orange Medal token.
Oct 6 2020, 9:04 PM · Patch-For-Review, cloud-services-team (Kanban), observability, Operations