MoritzMuehlenhoff (Moritz Mühlenhoff)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Monday

  • Clear sailing ahead.

User Details

User Since
Apr 1 2015, 4:33 PM (193 w, 2 d)
Availability
Available
LDAP User
Moritz Mühlenhoff
MediaWiki User
MMuhlenhoff (WMF) [ Global Accounts ]

Recent Activity

Wed, Dec 12

MoritzMuehlenhoff added a comment to T211605: Upgrade Superset to 0.28.1.

The first breaking change that I can see (use of f-strings) happened in commit https://github.com/apache/incubator-superset/commit/cc3a625a4bb6b0e581b30f3112315ff5a8ab6807 that should be in the upcoming release, not in 0.28.1, so in theory reverting https://github.com/lyft/incubator-superset/commit/174ee13b512f8aaa311fe0980276ac970930f4e6 and building with Python 3.5 should be enough for this upgrade.

Wed, Dec 12, 1:39 PM · Patch-For-Review, Analytics-Kanban, Analytics
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Wed, Dec 12, 11:54 AM
MoritzMuehlenhoff added a comment to T211605: Upgrade Superset to 0.28.1.

We'll need to migrate to Buster anyway though to get Python 3.6, my only doubt though is if Superset will require say 3.7/3.8 in the future. At that point, we'll either need to build python3.X by ourself or use something like pyenv? Anyway, too long thinking :)

Wed, Dec 12, 10:09 AM · Patch-For-Review, Analytics-Kanban, Analytics
MoritzMuehlenhoff added a comment to T211605: Upgrade Superset to 0.28.1.

Sure, this sounds like a sensible candidate for next Q's initial evaluation/migration of buster (we'll investigate/fix up the base layer and also migrate a few systems)

Wed, Dec 12, 9:12 AM · Patch-For-Review, Analytics-Kanban, Analytics

Fri, Dec 7

MoritzMuehlenhoff added a comment to T211094: Port DirectorySize diamond collector to a Prometheus exporter.

@colewhite Could you followup with a patch to absent the collector from puppet? Other that, this seems fully resolved, I couldn't find any dashboard where the former metrics were in use.

Fri, Dec 7, 5:37 PM · Patch-For-Review, cloud-services-team, User-fgiunchedi, monitoring, Operations
MoritzMuehlenhoff added a comment to T208722: Add Michael Grosse to 'wmde' LDAP group.

Michael's NDA is now signed and on file. Feel free to proceed w/ access. Thanks!

Fri, Dec 7, 4:27 PM · Operations, LDAP-Access-Requests
MoritzMuehlenhoff closed T211170: Time on new servers different from time on puppetmaster1001 as Resolved.

During the initial installation d-i run the clock-setup component which syncronises the system clock using rdate. The time server is obtained via a DNS alias (for codfw that's ntp.codfw.wikimedia.org) and that alias was stale, it still pointed to acamar, which has since been replaced by dns1001.wikimedia.org. I've pushed a DNS change to fix this, all further installations should have the correct time.

Fri, Dec 7, 3:03 PM · Patch-For-Review, ops-codfw, Operations
MoritzMuehlenhoff updated subscribers of T211419: Test maps stack with new nodejs security update.
Fri, Dec 7, 12:46 PM · Reading-Infrastructure-Team-Backlog (Kanban), Maps
MoritzMuehlenhoff created T211419: Test maps stack with new nodejs security update.
Fri, Dec 7, 12:46 PM · Reading-Infrastructure-Team-Backlog (Kanban), Maps
MoritzMuehlenhoff added a comment to T211170: Time on new servers different from time on puppetmaster1001.

elastic2050 now has a role assigned and thus the current time, is there a server among the 27 you installed which is still in the pristine condition right after the base OS install?

Fri, Dec 7, 12:22 PM · Patch-For-Review, ops-codfw, Operations
MoritzMuehlenhoff added a comment to T209901: WMF Audiences engineers and analysts missing from `wmf` LDAP group.

@Jdforrester-WMF since this task is visible on the ops clinic board, is it ok if we mark this as "Resolved" and let whoever needs access to create a new task as @jcrespo suggested?

Not really. Access to these services is part of the day job of the individuals I mentioned. Making people jump through hoops is decidedly not helpful.

Fri, Dec 7, 11:15 AM · Patch-For-Review, LDAP-Access-Requests
MoritzMuehlenhoff added a comment to T211405: pc2007.codfw.wmnet network blip?.

See IRC/internal channel, during the setup of logstash2001.codfw.wmnet it accidentally reused the 10.192.0.104 A record.

Fri, Dec 7, 6:19 AM · Operations, netops

Thu, Dec 6

MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Thu, Dec 6, 5:43 PM
MoritzMuehlenhoff created T211336: Remove Diamond usage in ORES dashboard.
Thu, Dec 6, 1:09 PM · Scoring-platform-team (Current), ORES, User-Ladsgroup
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Thu, Dec 6, 12:04 PM
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Thu, Dec 6, 12:01 PM

Wed, Dec 5

MoritzMuehlenhoff updated the task description for T210704: Migrate node-based services in production to node10.
Wed, Dec 5, 10:26 AM · Patch-For-Review, Core Platform Team Backlog (Next), Services (next), Operations
MoritzMuehlenhoff edited P7887 Remaining trusty instances in production.
Wed, Dec 5, 10:13 AM
MoritzMuehlenhoff created P7887 Remaining trusty instances in production.
Wed, Dec 5, 10:06 AM

Tue, Dec 4

MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Tue, Dec 4, 3:38 PM
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Tue, Dec 4, 3:34 PM
MoritzMuehlenhoff triaged T211094: Port DirectorySize diamond collector to a Prometheus exporter as Normal priority.
Tue, Dec 4, 11:28 AM · Patch-For-Review, cloud-services-team, User-fgiunchedi, monitoring, Operations

Mon, Dec 3

MoritzMuehlenhoff updated the task description for T183454: Deprovision Diamond collectors no longer in use.
Mon, Dec 3, 10:08 AM · Patch-For-Review, User-fgiunchedi, monitoring, Operations
MoritzMuehlenhoff updated the task description for T183454: Deprovision Diamond collectors no longer in use.
Mon, Dec 3, 10:07 AM · Patch-For-Review, User-fgiunchedi, monitoring, Operations
MoritzMuehlenhoff updated the task description for T183454: Deprovision Diamond collectors no longer in use.
Mon, Dec 3, 10:05 AM · Patch-For-Review, User-fgiunchedi, monitoring, Operations
MoritzMuehlenhoff triaged T210993: Deprecate Diamond collectors in Cloud VPS as Normal priority.
Mon, Dec 3, 10:04 AM · cloud-services-team, User-fgiunchedi, monitoring, Operations
MoritzMuehlenhoff triaged T210991: Deprecate Diamond collectors in Tool Labs / Tool Forge as Normal priority.
Mon, Dec 3, 10:02 AM · Toolforge, cloud-services-team, User-fgiunchedi, monitoring, Operations
MoritzMuehlenhoff added a comment to T210850: WMCS-related dashboards using Diamond metrics.

As for spotting remaining Diamond metrics, https://phabricator.wikimedia.org/P7680 contains a Paste with remaining Diamond metric references (based on a script by Timo), the patch which lists "Matched" refers to the dashboard in question. This is what I used to create the task.

Mon, Dec 3, 9:12 AM · cloud-services-team (Kanban), Operations

Fri, Nov 30

MoritzMuehlenhoff created T210850: WMCS-related dashboards using Diamond metrics.
Fri, Nov 30, 3:54 PM · cloud-services-team (Kanban), Operations
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Fri, Nov 30, 3:49 PM
MoritzMuehlenhoff created T210833: Filter potentially harmful PostScript commands in Commons upload/thumbor.
Fri, Nov 30, 11:30 AM · Operations
MoritzMuehlenhoff added a comment to T210036: upgrade people.wikimedia.org to stretch (replace rutherfordium with people1001).

JFTR, It's better to use the wmf-decommission-host script, it also removes the debmonitor host entry (I fixed that manually).

Fri, Nov 30, 8:55 AM · Technical-Debt, vm-requests, Operations

Thu, Nov 29

MoritzMuehlenhoff added a comment to T210704: Migrate node-based services in production to node10.

Does this mean he have a hard deadline of 2019-04-01 for completing the migrations? Or per the "I can backport security fixes for a while" we have a couple of more months?

Thu, Nov 29, 10:51 AM · Patch-For-Review, Core Platform Team Backlog (Next), Services (next), Operations
MoritzMuehlenhoff updated the task description for T210704: Migrate node-based services in production to node10.
Thu, Nov 29, 9:12 AM · Patch-For-Review, Core Platform Team Backlog (Next), Services (next), Operations
MoritzMuehlenhoff updated subscribers of T210704: Migrate node-based services in production to node10.
Thu, Nov 29, 8:48 AM · Patch-For-Review, Core Platform Team Backlog (Next), Services (next), Operations
MoritzMuehlenhoff created T210704: Migrate node-based services in production to node10.
Thu, Nov 29, 8:46 AM · Patch-For-Review, Core Platform Team Backlog (Next), Services (next), Operations

Wed, Nov 28

MoritzMuehlenhoff added a comment to T210667: Can exfat be used in WMF production?.

exfat-fuse itself is free software (GPL) and part of Debian main. Debian's approach on patents is written up at https://www.debian.org/reports/patent-faq (TLDR; unless patents are actively enforced, they're ignored. Debian has been shipping ffmpeg which implements patent-encumbered algorithms for a long time as well).

Wed, Nov 28, 9:26 PM · Security-Team, Analytics, Software-Licensing, WMF-Legal, Operations
MoritzMuehlenhoff created T210612: Remove neodymium/sarin from router ACLs.
Wed, Nov 28, 2:49 PM · Operations, netops
MoritzMuehlenhoff triaged T210597: Broken elasticsearch-prometheus-exporter service on logstash nodes after reboot as Normal priority.
Wed, Nov 28, 11:52 AM · Discovery-Search (Current work), Elasticsearch, Operations
MoritzMuehlenhoff created T210597: Broken elasticsearch-prometheus-exporter service on logstash nodes after reboot.
Wed, Nov 28, 11:52 AM · Discovery-Search (Current work), Elasticsearch, Operations

Tue, Nov 27

MoritzMuehlenhoff updated the task description for T183454: Deprovision Diamond collectors no longer in use.
Tue, Nov 27, 12:45 PM · Patch-For-Review, User-fgiunchedi, monitoring, Operations
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Tue, Nov 27, 9:00 AM
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Tue, Nov 27, 8:46 AM

Fri, Nov 23

MoritzMuehlenhoff added a comment to T153468: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs.

Status update: Faidon created a patch for Ferm to address this at https://github.com/MaxKellermann/ferm/pull/41. It's not yet reviewed/merged upstream and too intrusive to backport. We'll revisit when a new upstream release with the patches is available (and ideally it lands in buster as well).

Fri, Nov 23, 4:28 PM · monitoring, Beta-Cluster-Infrastructure, Patch-For-Review, Upstream, Operations, Beta-Cluster-reproducible, Traffic, DNS
MoritzMuehlenhoff created T210289: Upgrade Ganeti clusters to 2.15.2-7+deb9u3.
Fri, Nov 23, 1:59 PM · Operations
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Fri, Nov 23, 8:33 AM
MoritzMuehlenhoff updated subscribers of T204993: Update certspotter.

The Icinga servers in production are now running 0.9-1~bpo9+1, but the Cron job still needs to be re-instated.

Fri, Nov 23, 8:05 AM · Patch-For-Review, Traffic, Operations

Tue, Nov 20

MoritzMuehlenhoff added a comment to T207377: Reboot WMCS servers for L1TF.

See the task description, "For labvirt/cloudvirt I'll create a separate ticket as more steps are necessary." This needs a backport of SSBD support for the qemu version cloudvirt is using and some tests for the level of L1TF mitigation required.

Tue, Nov 20, 6:35 PM · Patch-For-Review, cloud-services-team (Kanban), Operations
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Tue, Nov 20, 3:41 PM
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Tue, Nov 20, 3:06 PM
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Tue, Nov 20, 3:05 PM
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Tue, Nov 20, 9:49 AM
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Tue, Nov 20, 9:37 AM
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Tue, Nov 20, 8:53 AM
MoritzMuehlenhoff updated the task description for T183454: Deprovision Diamond collectors no longer in use.
Tue, Nov 20, 8:48 AM · Patch-For-Review, User-fgiunchedi, monitoring, Operations

Mon, Nov 19

MoritzMuehlenhoff lowered the priority of T209802: Cannot vote on votewiki from Unbreak Now! to Normal.

Installing the package gnupg1 and using

$wgSecurePollGPGCommand = 'gpg1';

causes this to be fixed, I have tested it on mwmaint1002. Apparently lightprocess hangs waiting for gpg-agent to exit, which gpg2 autostarts whenever a private key operation is done.

Mon, Nov 19, 1:16 PM · Patch-For-Review, Operations, Wikimedia-production-error, MediaWiki-extensions-SecurePoll
MoritzMuehlenhoff updated subscribers of T209779: Degraded RAID on elastic2021.
Mon, Nov 19, 8:39 AM · Operations, ops-codfw
MoritzMuehlenhoff added projects to T209812: Review Elastic/maps Grafana dashboards: Operations, Elasticsearch, Maps.
Mon, Nov 19, 8:36 AM · Discovery-Search (Current work), Maps, Elasticsearch, Operations
MoritzMuehlenhoff created T209812: Review Elastic/maps Grafana dashboards.
Mon, Nov 19, 8:36 AM · Discovery-Search (Current work), Maps, Elasticsearch, Operations
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Mon, Nov 19, 8:35 AM

Fri, Nov 16

MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Fri, Nov 16, 3:53 PM
MoritzMuehlenhoff added a project to T209642: Remove labnodepool1001.eqiad.wmnet: decommission.
Fri, Nov 16, 9:06 AM · DC-Ops, ops-eqiad, decommission, Operations

Thu, Nov 15

MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Thu, Nov 15, 12:47 PM
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Thu, Nov 15, 10:42 AM
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Thu, Nov 15, 10:37 AM
MoritzMuehlenhoff added a comment to T209527: Set up scratch and maps NFS services on cloudstore1008/9.

IOW: Enabling profile::base::firewall for the role/hosts

Thu, Nov 15, 10:21 AM · Patch-For-Review, cloud-services-team (Kanban)
MoritzMuehlenhoff added a comment to T209527: Set up scratch and maps NFS services on cloudstore1008/9.

Enabling ferm was blocked on some of the labstore hosts as it was difficult to enable it on an already running system, for the new cloudstore* hosts we should enable it from the start. All the ferm rules/services should be available at this point.

Thu, Nov 15, 10:20 AM · Patch-For-Review, cloud-services-team (Kanban)
MoritzMuehlenhoff claimed T205618: rsync puppet module doesn't delete removed config.
Thu, Nov 15, 8:39 AM · Operations

Nov 14 2018

MoritzMuehlenhoff added a comment to T183454: Deprovision Diamond collectors no longer in use.

@Krinkle You've edited https://grafana.wikimedia.org/dashboard/db/cluster-board-graphite back in October the last time, that dashboard is increasingly obsolete as more and more hosts in production drop Diamond (and https://grafana.wikimedia.org/dashboard/db/prometheus-cluster-breakdown should serve as a replacement, can that be removed?)

Nov 14 2018, 2:02 PM · Patch-For-Review, User-fgiunchedi, monitoring, Operations
MoritzMuehlenhoff added a comment to T208584: Decommission old eqiad caches.

Icinga is flagging broken memory on 1053, simply leaving a note here as that host is up for decom anyway.

Nov 14 2018, 11:41 AM · ops-eqiad, decommission, Operations, Traffic
MoritzMuehlenhoff added a comment to T196477: rack/setup/install backup2001.

That point release has happened and I upgraded our netinst images earlier the day, so this should be fine to re-install now.

Nov 14 2018, 10:21 AM · Patch-For-Review, ops-codfw, Operations
MoritzMuehlenhoff reopened T208717: Requesting access to deployment for WMDE-leszek as "Open".

Leszek; you're now using the same SSH key in Cloud VPS as in the production cluster. This is a security risk as WMCS/VPS allows SSH agent forwarding and a malicious privileged user in WMCS can connect to our forwarded agent socket and connect to production on your behalf.

Nov 14 2018, 7:59 AM · Patch-For-Review, Operations, SRE-Access-Requests

Nov 13 2018

MoritzMuehlenhoff added a comment to T207377: Reboot WMCS servers for L1TF.

It's fine to upgrade the kernel, I've installed running what was recent when I created the task and those versions are sufficient to fix L1TF, but it's good to move to a newer kernel for additonal bugfixes in any case:

Nov 13 2018, 6:39 PM · Patch-For-Review, cloud-services-team (Kanban), Operations
MoritzMuehlenhoff added a comment to T207775: newer version of nagios-nrpe-plugin nrpe (check_nrpe) with fixed logging issue on stretch icinga.

While nagios-nrpe-plugin has been upgraded on the Icinga hosts, the nagios-nrpe source package also builds the nagios-nrpe-server binary, which should also be upgraded for consistency: https://debmonitor.wikimedia.org/packages/nagios-nrpe-server

Nov 13 2018, 11:37 AM · Patch-For-Review, monitoring, Operations

Nov 12 2018

MoritzMuehlenhoff added a comment to T209300: Review and make librdkafka-0.11.6 installable from stretch-wikimedia.

I think there are also some inconsistencies in the application of the apt pin. e.g. on an-coord1001, stat100[46] there's _no_ upgrade candidate, while there's one for stat1007.

Nov 12 2018, 5:49 PM · User-herron, Analytics, Services (watching), Core Platform Team Backlog (Watching / External), Wikimedia-Logstash, Operations
MoritzMuehlenhoff added a comment to T209300: Review and make librdkafka-0.11.6 installable from stretch-wikimedia.

I believe we had this problem (and discussion) before...and we decided that apt pinning to backports was better than importing our own version. This allows us to more easily upgrade and keep track of what versions are used where.

@MoritzMuehlenhoff might have an opinion here?

Nov 12 2018, 5:44 PM · User-herron, Analytics, Services (watching), Core Platform Team Backlog (Watching / External), Wikimedia-Logstash, Operations
MoritzMuehlenhoff added a comment to T209300: Review and make librdkafka-0.11.6 installable from stretch-wikimedia.

I'm fairly certain there shouldn't be any stretch hosts using 0.9.3-1, or at least not any running prod services. It should be fine.

Nov 12 2018, 5:43 PM · User-herron, Analytics, Services (watching), Core Platform Team Backlog (Watching / External), Wikimedia-Logstash, Operations
MoritzMuehlenhoff added a comment to T149287: Heating alerts for mw servers in eqiad.

@Cmjohnson : Per the procurement task, thermal paste is now available?

Nov 12 2018, 5:41 PM · Operations, ops-eqiad
MoritzMuehlenhoff added a comment to T153468: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs.

Indeed, thanks! I've tested this on mw2151 with a the ferm upstream change applied on top of our current package and a backport of libnet-dns-perl 1.17 plus upstream revision r1717. Going to to a few more tests with libnet-dns-perl reverse dependencies and then I'll rollout this fleet-wide.

Nov 12 2018, 3:17 PM · monitoring, Beta-Cluster-Infrastructure, Patch-For-Review, Upstream, Operations, Beta-Cluster-reproducible, Traffic, DNS
MoritzMuehlenhoff added a comment to T209279: debmonitor search yields nothing.

@hashar: Search is implemented in general: If you click either of "Hosts", "Kernels", "Packages" or "Source Packages" you can search in there.

Nov 12 2018, 2:04 PM · Operations
MoritzMuehlenhoff added a comment to T209279: debmonitor search yields nothing.

Where did you search specifically? Does e.g. https://debmonitor.wikimedia.org/packages/jenkins list the installed packages for you?

Nov 12 2018, 1:55 PM · Operations
MoritzMuehlenhoff reopened T209264: Upgrade Jenkins instances to 2.138.3 as "Open".

This was closed prematurely.

Nov 12 2018, 12:06 PM · Jenkins, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure
MoritzMuehlenhoff closed T209264: Upgrade Jenkins instances to 2.138.3 as Resolved.

I've uploaded 2.138.3 for jessie and stretch. The component names are correct, we've only introduced the concept of dedicated components for thirdparty with stretch. So for jessie it went into "thirdparty" and for stretch into "thirdparty/ci".

Nov 12 2018, 12:02 PM · Jenkins, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure
MoritzMuehlenhoff triaged T209260: Integrate Stretch 9.6 point update as Normal priority.
Nov 12 2018, 7:20 AM · Operations
MoritzMuehlenhoff created T209260: Integrate Stretch 9.6 point update.
Nov 12 2018, 7:20 AM · Operations

Nov 9 2018

MoritzMuehlenhoff added a comment to T153468: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs.

Ferm merged my pull request for handling of NOERROR empty responses. So now if we can get the next release of ferm, and add the trailing full stops (it doesn't look like there'll be any movement on that problem inside Net::DNS any time soon, if ever), we should be good.

Nov 9 2018, 4:13 PM · monitoring, Beta-Cluster-Infrastructure, Patch-For-Review, Upstream, Operations, Beta-Cluster-reproducible, Traffic, DNS
MoritzMuehlenhoff triaged T209140: Broken mgmt on kubestage1001 as Normal priority.
Nov 9 2018, 10:12 AM · ops-eqiad, Operations
MoritzMuehlenhoff created T209140: Broken mgmt on kubestage1001.
Nov 9 2018, 10:12 AM · ops-eqiad, Operations
MoritzMuehlenhoff added a comment to T209139: Broken memory on mw1239.

Server is depooled for now

Nov 9 2018, 10:07 AM · ops-eqiad, Operations
MoritzMuehlenhoff created T209139: Broken memory on mw1239.
Nov 9 2018, 10:06 AM · ops-eqiad, Operations
MoritzMuehlenhoff updated the task description for T208729: Onboarding Chris Danis (CDanis).
Nov 9 2018, 10:03 AM · User-CDanis, Patch-For-Review, SRE-Access-Requests, Operations, User-herron, LDAP-Access-Requests
MoritzMuehlenhoff closed T209134: pwstore access for cdanis as Resolved.

I've added you to pwstore, please see https://office.wikimedia.org/wiki/Pwstore for some docs. If you run into any issues, ping me on IRC :-)

Nov 9 2018, 10:03 AM · User-CDanis, SRE-Access-Requests, Operations, User-herron
MoritzMuehlenhoff closed T209134: pwstore access for cdanis, a subtask of T208729: Onboarding Chris Danis (CDanis), as Resolved.
Nov 9 2018, 10:03 AM · User-CDanis, Patch-For-Review, SRE-Access-Requests, Operations, User-herron, LDAP-Access-Requests
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Nov 9 2018, 9:48 AM
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Nov 9 2018, 9:13 AM
MoritzMuehlenhoff edited P7680 Masterwork From Distant Lands.
Nov 9 2018, 9:12 AM
MoritzMuehlenhoff updated the task description for T183454: Deprovision Diamond collectors no longer in use.
Nov 9 2018, 9:09 AM · Patch-For-Review, User-fgiunchedi, monitoring, Operations
MoritzMuehlenhoff updated the task description for T164456: Migrate to nginx-light.
Nov 9 2018, 9:07 AM · User-ArielGlenn, Traffic, Operations
MoritzMuehlenhoff updated subscribers of T164456: Migrate to nginx-light.
Nov 9 2018, 9:04 AM · User-ArielGlenn, Traffic, Operations

Nov 8 2018

MoritzMuehlenhoff added a comment to T208715: Onboard Fabián Sellés Rosa to SRE.

Added to pwstore.

Nov 8 2018, 6:19 PM · Patch-For-Review, Operations