MoritzMuehlenhoff (Moritz Mühlenhoff)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Apr 1 2015, 4:33 PM (138 w, 18 h)
Availability
Available
LDAP User
Moritz Mühlenhoff
MediaWiki User
MMuhlenhoff (WMF)

Recent Activity

Yesterday

MoritzMuehlenhoff added a comment to T180978: Switch on http/2 in apache for gerrit.
  1. Since the experimental tag has been removed only recently I strongly suggest to use a recent version of httpd (possibly only debian >= stretch in which we have 2.4.25). On cobalt we are running 2.4.10 that should not be used in my opinion.
Wed, Nov 22, 3:25 PM · Traffic, Patch-For-Review, Operations, Gerrit
MoritzMuehlenhoff added a comment to T181019: Consider using a single MediaWiki releases key instead of individual keys .

My proposal would be the following:

Wed, Nov 22, 1:12 PM · Security, MediaWiki-Releasing
MoritzMuehlenhoff created T181121: Possible memory errors on ganeti1005.
Wed, Nov 22, 8:28 AM · ops-eqiad, Operations

Mon, Nov 20

MoritzMuehlenhoff added a comment to T180935: Various puppet issues in deployment-prep.

deployment-tin seems failing because scap is put on hold, since 3.7.3 is also on apt.wikimedia.org "apt-mark unhold scap" should fix it.

Mon, Nov 20, 12:40 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure
MoritzMuehlenhoff added a comment to T180935: Various puppet issues in deployment-prep.

If deployment-mx is still in use/needed, it should be reimaged to jessie or stretch.

Mon, Nov 20, 12:37 PM · Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure

Fri, Nov 17

MoritzMuehlenhoff added a comment to T180792: Remove 3DES patch from OpenSSL builds.

I've commited this to git, it doesn't warrant to roll new packages for this change alone, this can be piggybacked with the next openssl security update.

Fri, Nov 17, 5:52 PM · Operations, Traffic
MoritzMuehlenhoff added a comment to T180524: Upgrade latest docker-registry.wikimedia.org/nodejs-devel to stretch.

Yeah, I guess that would be an alternative to consider.

Fri, Nov 17, 12:46 PM · Release-Engineering-Team (Kanban), Operations, Release Pipeline

Thu, Nov 16

MoritzMuehlenhoff added a comment to T163778: Decommission db1022 (Was: db1022 broke while changing topology on s6- evaluate if to fix or directly decommission).

JFTR: The host was still showing up in puppetdb (e.g. via https://servermon.wikimedia.org/hosts/). I ran "puppet node deactivate db1022.eqiad.wmnet" on puppetmaster1001, that should properly remove it.

Thu, Nov 16, 4:51 PM · Operations, ops-eqiad, DBA

Wed, Nov 15

MoritzMuehlenhoff added a comment to T180524: Upgrade latest docker-registry.wikimedia.org/nodejs-devel to stretch.

Current npm releases are not packaged in Debian since the list of dependencies exploded. Efforts are tracked in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=794890 and https://wiki.debian.org/Javascript/Nodejs/Tasks/npm

Wed, Nov 15, 9:15 AM · Release-Engineering-Team (Kanban), Operations, Release Pipeline

Tue, Nov 14

MoritzMuehlenhoff added a comment to T179609: Obtain CVE's for 1.27.4/1.29.2 security releases.

I'd say let's skip a CVE ID for "BotPasswords doesn't throttle login attempts" since it has no practical (or marginal at best) security implications.

Tue, Nov 14, 3:25 PM · Security
MoritzMuehlenhoff renamed T119158: Language converter: unsafe attribute injection via glossary rules (CVE-2017-8815) from Language converter: unsafe attribute injection via glossary rules to Language converter: unsafe attribute injection via glossary rules (CVE-2017-8815).
Tue, Nov 14, 9:30 AM · MW-1.31-release-notes (WMF-deploy-2017-11-28 (1.31.0-wmf.10)), MW-1.29-release-notes, MW-1.30-release-notes, Patch-For-Review, User-notice, Community-Liaisons, Security-Team, Parsing-Team, Security
MoritzMuehlenhoff renamed T124404: language converter can be tricked into replacing text inside tags by adding a lot of junk after the rule definition (CVE-2017-8814) from language converter can be tricked into replacing text inside tags by adding a lot of junk after the rule definition to language converter can be tricked into replacing text inside tags by adding a lot of junk after the rule definition (CVE-2017-8814).
Tue, Nov 14, 9:30 AM · MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), MW-1.29-release-notes, MW-1.30-release-notes, Security-Team, Patch-For-Review, Security, MediaWiki-Language-converter, Security-Core
MoritzMuehlenhoff renamed T125163: id attribute on headlines allow raw > [Possible issue in combination with language converter] (CVE-2017-8812) from id attribute on headlines allow raw > [Possible issue in combination with language converter] to id attribute on headlines allow raw > [Possible issue in combination with language converter] (CVE-2017-8812).
Tue, Nov 14, 9:29 AM · MW-1.29-release-notes, Patch-For-Review, MediaWiki-Language-converter, Security, Security-Team
MoritzMuehlenhoff added a comment to T179609: Obtain CVE's for 1.27.4/1.29.2 security releases.

There we go for the three new arrivals:

Tue, Nov 14, 9:26 AM · Security
MoritzMuehlenhoff updated the task description for T179609: Obtain CVE's for 1.27.4/1.29.2 security releases.
Tue, Nov 14, 9:20 AM · Security
MoritzMuehlenhoff added a comment to T180200: Broken memory on mw2108.

Thanks, I ran "scap pull" and repooled the host.

Tue, Nov 14, 9:06 AM · Operations, ops-codfw
MoritzMuehlenhoff closed T180373: Degraded RAID on wtp2017 as Invalid.

Duplicate of T180373

Tue, Nov 14, 9:00 AM · Operations, ops-codfw

Mon, Nov 13

MoritzMuehlenhoff added a comment to T176247: It's possible to mangle HTML via raw message parameter expansion.

This is CVE-2017-8811

Mon, Nov 13, 8:29 AM · MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), MW-1.29-release-notes, MW-1.30-release-notes, Patch-For-Review, Community-Tech, MW-1.31-release, MW-1.30-release, Vuln-XSS, MediaWiki-Parser, Security
MoritzMuehlenhoff added a comment to T134100: On private wikis, login form shouldn't distinguish between login failure due to bad username and bad password.

This is CVE-2017-8810

Mon, Nov 13, 8:28 AM · MW-1.31-release-notes (WMF-deploy-2017-11-28 (1.31.0-wmf.10)), MW-1.29-release-notes, MW-1.30-release-notes, Patch-For-Review, Vuln-Infoleak, Security
MoritzMuehlenhoff added a comment to T128209: Reflected File Download from api.php.

This is CVE-2017-8809

Mon, Nov 13, 8:28 AM · MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), MW-1.29-release-notes, MW-1.30-release-notes, Patch-For-Review, Security-Team, Security
MoritzMuehlenhoff added a comment to T178451: XSS when $wgShowExceptionDetails=false and browser sends non-standard url escaping.

This is CVE-2017-8808

Mon, Nov 13, 8:28 AM · MW-1.31-release-notes (WMF-deploy-2017-11-14 (1.31.0-wmf.8)), MW-1.29-release-notes, MW-1.30-release-notes, Patch-For-Review, Vuln-XSS, Security
MoritzMuehlenhoff added a comment to T179609: Obtain CVE's for 1.27.4/1.29.2 security releases.

I'm assigning CVE IDs from the Debian CNA pool:

Mon, Nov 13, 8:27 AM · Security
MoritzMuehlenhoff updated subscribers of T178271: Allow Kirk and Martijn (JClarity) access to our WDQS production servers.

I'm adding @RStallman-legalteam for preparing the NDAs for Martijn and Kirk.

Mon, Nov 13, 7:38 AM · Operations, Ops-Access-Requests, Discovery-Search (Current work), Discovery, Discovery-Wikidata-Query-Service-Sprint, Wikidata-Query-Service, Wikidata
MoritzMuehlenhoff triaged T154619: Export ipsec counters as Prometheus metrics as Normal priority.
Mon, Nov 13, 7:36 AM · monitoring, Operations
MoritzMuehlenhoff triaged T179984: Update Debian package for Blubber as Normal priority.
Mon, Nov 13, 7:35 AM · Release-Engineering-Team (Watching / External), Release Pipeline, Operations
MoritzMuehlenhoff triaged T179565: Port exim statistics to Prometheus as High priority.
Mon, Nov 13, 7:34 AM · Patch-For-Review, User-fgiunchedi, Goal, Operations
MoritzMuehlenhoff triaged T178575: Add require_package() variant with repository component to wmflib as Normal priority.
Mon, Nov 13, 7:34 AM · Puppet, Operations
MoritzMuehlenhoff triaged T177197: Export Prometheus-compatible JVM metrics from JVMs in production as High priority.
Mon, Nov 13, 7:34 AM · User-fgiunchedi, Goal, Operations
MoritzMuehlenhoff triaged T179230: Puppet wmf-style-guide: array of classes not detected properly as Normal priority.
Mon, Nov 13, 7:33 AM · Puppet, Operations
MoritzMuehlenhoff triaged T179317: Varnish and Apache root for hoo as Normal priority.
Mon, Nov 13, 7:33 AM · Performance-Team (Radar), Operations, Ops-Access-Requests

Fri, Nov 10

MoritzMuehlenhoff assigned T180211: Degraded RAID on wtp2017 to Papaul.
Fri, Nov 10, 10:59 AM · Operations, ops-codfw
MoritzMuehlenhoff created T180200: Broken memory on mw2108.
Fri, Nov 10, 9:57 AM · Operations, ops-codfw

Thu, Nov 9

MoritzMuehlenhoff created T180127: Reboot of dumps hosts.
Thu, Nov 9, 12:35 PM · Datasets-General-or-Unknown, User-ArielGlenn, Operations
MoritzMuehlenhoff added a comment to T150532: Upgrade qemu on ganeti clusters to 2.7.

Your patch is also missing in the Ganeti version in stretch, let's report it to the Debian BTS so that it can possibly be backported to a stretch point release?

Thu, Nov 9, 11:26 AM · Operations

Wed, Nov 8

MoritzMuehlenhoff triaged T167689: Add RIPE atlas data to Prometheus as Normal priority.
Wed, Nov 8, 3:42 PM · monitoring, Operations
MoritzMuehlenhoff triaged T177385: Upgrade Cumin masters to stretch as Normal priority.
Wed, Nov 8, 3:42 PM · Operations-Software-Development, Operations
MoritzMuehlenhoff triaged T178392: Replacement hardware for cumin masters as Normal priority.
Wed, Nov 8, 3:42 PM · hardware-requests, Operations
MoritzMuehlenhoff triaged T179696: Homepage for https://docker-registry.wikimedia.org as Normal priority.
Wed, Nov 8, 3:42 PM · Operations, MediaWiki-Containers
MoritzMuehlenhoff triaged T174431: Migration of mw* servers to stretch as Normal priority.
Wed, Nov 8, 3:41 PM · Patch-For-Review, User-Elukey, HHVM, Operations
MoritzMuehlenhoff added a comment to T179964: Created dedicated elastic component in our APT repository.

New components thirdparty/elastic55 and component/elastic55 have been created and kibana, logstash, elasticsearch wmf-elasticsearch-search-plugins have been imported.

Wed, Nov 8, 10:56 AM · Patch-For-Review, Discovery-Search (Current work), Operations, CirrusSearch, Elasticsearch, Discovery
MoritzMuehlenhoff added a comment to T179729: Adding phedenskog to perf-team.

I've updated the title to reflect the recent creation of perf-team. I'll create a Gerrit patch, but this needs to be approved in next Monday's Ops meeting first.

Wed, Nov 8, 8:44 AM · Performance-Team, Operations, Ops-Access-Requests
MoritzMuehlenhoff claimed T179729: Adding phedenskog to perf-team.
Wed, Nov 8, 8:43 AM · Performance-Team, Operations, Ops-Access-Requests
MoritzMuehlenhoff renamed T179729: Adding phedenskog to perf-team from Requesting access to perf-teams for phedenskog (add phedenskog to perf-roots) to Adding phedenskog to perf-team.
Wed, Nov 8, 8:43 AM · Performance-Team, Operations, Ops-Access-Requests
MoritzMuehlenhoff closed T179728: Create perf-team shell group as Resolved.

The task descriptions mentions "Add aaron to perf-team group", but he has cluster-wide root access, so that's not needed. Since Daniel created the group and there's a separate task for adding Peter, I'm closing this task.

Wed, Nov 8, 8:42 AM · Patch-For-Review, Ops-Access-Requests, Performance-Team (Radar), Operations
MoritzMuehlenhoff closed T179728: Create perf-team shell group, a subtask of T179729: Adding phedenskog to perf-team, as Resolved.
Wed, Nov 8, 8:42 AM · Performance-Team, Operations, Ops-Access-Requests

Tue, Nov 7

MoritzMuehlenhoff added a comment to T179943: Restart Analytics JVM daemons for open-jdk security updates.

Note that the hadoop clusters and kafka* are running Java 7 and there hasn't been an openjdk-7 release yet (so also no update in Debian), so at this point only kafka-jumbo (which runs stretch/java8) and aqs/cassandra need an update.

Tue, Nov 7, 3:59 PM · Analytics-Kanban, User-Elukey
MoritzMuehlenhoff triaged T177914: Switch labstore servers to default SSH configuration as Normal priority.
Tue, Nov 7, 11:13 AM · cloud-services-team (Kanban), Data-Services, Operations
MoritzMuehlenhoff triaged T177195: Reduce technical debt in metrics monitoring as High priority.
Tue, Nov 7, 11:11 AM · User-fgiunchedi, Technical-Debt, Goal, Operations
MoritzMuehlenhoff added a comment to T179317: Varnish and Apache root for hoo.

If what @hoo needs is only a subset of what we have access to, why not create a new group for that?

Tue, Nov 7, 11:11 AM · Performance-Team (Radar), Operations, Ops-Access-Requests
MoritzMuehlenhoff triaged T177821: Allow syslog-tls and syslog in analytics towards wezen/lithium as Normal priority.
Tue, Nov 7, 8:41 AM · netops, Operations
MoritzMuehlenhoff added a project to T177821: Allow syslog-tls and syslog in analytics towards wezen/lithium: netops.
Tue, Nov 7, 8:41 AM · netops, Operations
MoritzMuehlenhoff added a comment to T179380: hhvm-staging.hhvm.eqiad.wmflabs has high user/system CPU.

I think that @MoritzMuehlenhoff probably meant that we don't have (many) Trusty HHVM hosts (?).

Tue, Nov 7, 8:20 AM · User-bd808, cloud-services-team (Kanban), VPS-Projects, HHVM
MoritzMuehlenhoff added a comment to T178189: [spike] Temporarily allow pushing large objects.

Our scb* cluster currently runs jessie. I don't know the time frame for the new setup, but running the electron replacement on stretch should be doable. Depending on the ETA and available resources in the Services team we can either migrate scb* in general to stretch or alternatively we could create a stretch-based scc* cluster based on Ganeti instances.

Tue, Nov 7, 7:48 AM · Spike, Operations, Unplanned-Sprint-Work, Readers-Web-Kanban-Board, Patch-For-Review, Readers-Web-Backlog, Gerrit

Mon, Nov 6

MoritzMuehlenhoff added a comment to T174477: Reimage deployment-tmh01 with Debian Jessie.

I doubt that system is used at all, there's deployment-videoscaler01, which I have been using to test stretch compat. I think we can simply remove deployment-tmh01

Mon, Nov 6, 5:27 PM · Multimedia, Beta-Cluster-Infrastructure
MoritzMuehlenhoff claimed T177371: Phase out DSA keys for SSH access (ssh-dss).
Mon, Nov 6, 8:21 AM · Operations
MoritzMuehlenhoff claimed T177498: Provide a forward port of ICU 52 for stretch / Investigate best ICU update strategy.
Mon, Nov 6, 8:20 AM · User-Elukey, HHVM, Operations
MoritzMuehlenhoff updated the task description for T178799: Revisit Pybal depool thresholds for app servers.
Mon, Nov 6, 8:20 AM · Patch-For-Review, Operations
MoritzMuehlenhoff triaged T178799: Revisit Pybal depool thresholds for app servers as Normal priority.
Mon, Nov 6, 8:19 AM · Patch-For-Review, Operations
MoritzMuehlenhoff triaged T177622: Multiple systems in ulsfo 1.22 showing PSU failures as Normal priority.
Mon, Nov 6, 8:19 AM · ops-ulsfo, Operations
MoritzMuehlenhoff triaged T177623: check lvs4002 power supply redundancy as Normal priority.
Mon, Nov 6, 8:19 AM · ops-ulsfo, Operations
MoritzMuehlenhoff closed T177625: check cp4008 power supply redundancy as Declined.

This server is decommissioned via T176366, so closing the task.

Mon, Nov 6, 8:17 AM · ops-ulsfo, Operations
MoritzMuehlenhoff closed T177625: check cp4008 power supply redundancy, a subtask of T177622: Multiple systems in ulsfo 1.22 showing PSU failures, as Declined.
Mon, Nov 6, 8:17 AM · ops-ulsfo, Operations
MoritzMuehlenhoff closed T177624: check cp4007 power supply redundancy as Declined.

This server is decommissioned via T176366, so closing the task.

Mon, Nov 6, 8:17 AM · ops-ulsfo, Operations
MoritzMuehlenhoff closed T177624: check cp4007 power supply redundancy, a subtask of T177622: Multiple systems in ulsfo 1.22 showing PSU failures, as Declined.
Mon, Nov 6, 8:17 AM · ops-ulsfo, Operations
MoritzMuehlenhoff assigned T177638: check mw2160 power supply redundancy to Papaul.
Mon, Nov 6, 8:15 AM · ops-codfw, Operations
MoritzMuehlenhoff assigned T177639: check mw2176 power supply redundancy to Papaul.
Mon, Nov 6, 8:15 AM · ops-codfw, Operations
MoritzMuehlenhoff triaged T179640: mw1191 ipmi-sel cpu errors as Normal priority.
Mon, Nov 6, 8:14 AM · Operations, ops-eqiad
MoritzMuehlenhoff triaged T179022: Backport firejail 0.9.52 for use on Wikimedia appservers as Normal priority.
Mon, Nov 6, 8:03 AM · Operations
MoritzMuehlenhoff triaged T179353: Scap: Standardize git version as Normal priority.
Mon, Nov 6, 8:03 AM · Operations, Release-Engineering-Team (Watching / External), Scap
MoritzMuehlenhoff triaged T176666: Qualtrics email-LDAP issue as Normal priority.
Mon, Nov 6, 8:03 AM · Operations, Mail, Surveys
MoritzMuehlenhoff triaged T176816: cr2-esams temperature warning as Normal priority.
Mon, Nov 6, 8:02 AM · DC-Ops, ops-esams, netops, Operations
MoritzMuehlenhoff triaged T177196: Port non-deprecated Diamond collectors to Prometheus as High priority.
Mon, Nov 6, 8:01 AM · cloud-services-team (Kanban), Patch-For-Review, User-fgiunchedi, Goal, Operations

Fri, Nov 3

MoritzMuehlenhoff added a comment to T179353: Scap: Standardize git version.

Building a git 2.11 for trusty is probably just a matter of 1-2 hours work, but it's something we would need to repeat for every git security update. I'd totally prefer Mukunda's approach since it feels wrong to invest time into something we're actively working to get rid of.

Fri, Nov 3, 12:09 PM · Operations, Release-Engineering-Team (Watching / External), Scap
MoritzMuehlenhoff added a comment to T179317: Varnish and Apache root for hoo.

Can you elaborate what you need in specific to debug wikidata performance problems? We can arrange access to all the logs you need, but perf-roots grants full root access to nearly half the servers in production.

Fri, Nov 3, 8:05 AM · Performance-Team (Radar), Operations, Ops-Access-Requests

Wed, Nov 1

MoritzMuehlenhoff created P6242 Remaining trusty hosts.
Wed, Nov 1, 10:09 PM
MoritzMuehlenhoff added a comment to T179050: setup bast4002/WMF7218.

+1 We may as well move to stretch here. For the bastion/installserver role it should be pretty simple?

Wed, Nov 1, 1:55 PM · Traffic, Operations, ops-ulsfo
MoritzMuehlenhoff added a comment to T179353: Scap: Standardize git version.

silver will be replaced by the new labweb* hosts using stretch soon, so that should be resolved soon. Is that the only one deployment relevant?

Wed, Nov 1, 9:42 AM · Operations, Release-Engineering-Team (Watching / External), Scap
MoritzMuehlenhoff added a comment to T179380: hhvm-staging.hhvm.eqiad.wmflabs has high user/system CPU.

I never used that VM and we don't have any HHVM hosts in production apart from silver, so from my PoV this can be dropped.

Wed, Nov 1, 9:19 AM · User-bd808, cloud-services-team (Kanban), VPS-Projects, HHVM
MoritzMuehlenhoff added a comment to T168584: Labsdb* servers need to be rebooted.

Let's just keep 1003 running w/o reboot then.

Wed, Nov 1, 8:34 AM · Patch-For-Review, Scoring-platform-team (Current), DBA, cloud-services-team, Operations
MoritzMuehlenhoff added a comment to T168584: Labsdb* servers need to be rebooted.

We should consider labsdb1001 broken for good and decommission it - we need to decide whether we want to continue with the plan and reboot labsdb1003. I wouldn't do it, to be honest.

Wed, Nov 1, 8:29 AM · Patch-For-Review, Scoring-platform-team (Current), DBA, cloud-services-team, Operations
MoritzMuehlenhoff added a comment to T179050: setup bast4002/WMF7218.

This is currently installed with jessie, but if we setup a new box, let's use stretch from the start?

Wed, Nov 1, 8:24 AM · Traffic, Operations, ops-ulsfo

Thu, Oct 26

MoritzMuehlenhoff updated the task description for T178807: Onboard aborrero to WMF.
Thu, Oct 26, 10:14 AM · Patch-For-Review, cloud-services-team
MoritzMuehlenhoff added a comment to T178807: Onboard aborrero to WMF.

I've added Arturo to cn=wmf and cn=ops.

Thu, Oct 26, 10:14 AM · Patch-For-Review, cloud-services-team
MoritzMuehlenhoff added a comment to T177891: Update and use php-wikidiff2 1.5.1 & MovedParagraphDetectionCutoff in production.

@Tobi_WMDE_SW , @Addshore : wikidiff2 1.5.1 is now rolled out in production across all our mediawiki, you can proceed with the wgWikiDiff2MovedParagraphDetectionCutoff from my PoV.

Thu, Oct 26, 9:04 AM · WMDE-QWERTY-Sprint-2017-11-01, Patch-For-Review, Operations, User-Addshore, WMDE-QWERTY-Team-Board, wikidiff2
MoritzMuehlenhoff updated the task description for T177891: Update and use php-wikidiff2 1.5.1 & MovedParagraphDetectionCutoff in production.
Thu, Oct 26, 9:03 AM · WMDE-QWERTY-Sprint-2017-11-01, Patch-For-Review, Operations, User-Addshore, WMDE-QWERTY-Team-Board, wikidiff2

Wed, Oct 25

MoritzMuehlenhoff added a comment to T178807: Onboard aborrero to WMF.

Done the GPG key for pwstore. Talked with @MoritzMuehlenhoff about this.

Wed, Oct 25, 12:25 PM · Patch-For-Review, cloud-services-team
MoritzMuehlenhoff added a comment to T168584: Labsdb* servers need to be rebooted.

I installed the latest trusty kernels on labsdb1001/1003.

Wed, Oct 25, 8:52 AM · Patch-For-Review, Scoring-platform-team (Current), DBA, cloud-services-team, Operations

Oct 23 2017

MoritzMuehlenhoff renamed T164456: Migrate to nginx-light from Build nginx without image filter support to Migrate to nginx-light.
Oct 23 2017, 3:48 PM · Traffic, Operations
MoritzMuehlenhoff created T178799: Revisit Pybal depool thresholds for app servers.
Oct 23 2017, 1:00 PM · Patch-For-Review, Operations

Oct 20 2017

MoritzMuehlenhoff added a comment to T178570: How should we get Chromium for use in puppeteer?.

OTOH there's nothing to stop us from launching a Chromium process ourselves and using command line switches to make it save the page as a PDF: https://peter.sh/experiments/chromium-command-line-switches/#print-to-pdf (this list is linked to from https://www.chromium.org/developers/how-tos/run-chromium-with-flags).

Before we go any further investigating how we can best support using the puppeteer library, we should first revalidate whether we should use it in light of all of this recent (both productive and enlightening!) discussion.

Oct 20 2017, 1:17 PM · Spike, Release-Engineering-Team (Watching / External), Operations, Unplanned-Sprint-Work, Readers-Web-Kanban-Board, Readers-Web-Backlog, Proton, Electron-PDFs
MoritzMuehlenhoff created P6160 API servers pybal status.
Oct 20 2017, 10:21 AM
MoritzMuehlenhoff added a comment to T178570: How should we get Chromium for use in puppeteer?.

I think there are a few things at play here:

  • How do we distribute chromium to the servers in the cluster efficiently?
Oct 20 2017, 6:43 AM · Spike, Release-Engineering-Team (Watching / External), Operations, Unplanned-Sprint-Work, Readers-Web-Kanban-Board, Readers-Web-Backlog, Proton, Electron-PDFs
MoritzMuehlenhoff added a comment to T178570: How should we get Chromium for use in puppeteer?.

Slight race condition here :-) I had just followed up on https://phabricator.wikimedia.org/T178189#3698691 for this

Oct 20 2017, 6:07 AM · Spike, Release-Engineering-Team (Watching / External), Operations, Unplanned-Sprint-Work, Readers-Web-Kanban-Board, Readers-Web-Backlog, Proton, Electron-PDFs
MoritzMuehlenhoff added a comment to T178189: [spike] Temporarily allow pushing large objects.

Also, the latest Debian Jessie has the Chromium version 57.0.2987.98-1~deb8u1, and the headless Chromium first appeared in versoin 59. Does that mean we should compile our own version of Chromium? Wouldn't it defeat the purpose of getting free security fixes from the Debian package maintainers?

Oct 20 2017, 6:06 AM · Spike, Operations, Unplanned-Sprint-Work, Readers-Web-Kanban-Board, Patch-For-Review, Readers-Web-Backlog, Gerrit

Oct 19 2017

MoritzMuehlenhoff created T178575: Add require_package() variant with repository component to wmflib.
Oct 19 2017, 2:18 PM · Puppet, Operations

Oct 18 2017

MoritzMuehlenhoff added a comment to T178457: nutcracker fails to start due to lack of /var/run/nutcracker (ex: deployment-videoscaler01 has memcached failures).

deployment-videoscaler01 is one of the two servers experimentally using stretch, it's not comparable to what we use on the video scalers in production.

Oct 18 2017, 12:20 PM · Patch-For-Review, Operations, Release-Engineering-Team (Kanban), Beta-Cluster-Infrastructure
MoritzMuehlenhoff added a comment to T177385: Upgrade Cumin masters to stretch.

Procurement ticket is T178392

Oct 18 2017, 8:57 AM · Operations-Software-Development, Operations

Oct 17 2017

Volans awarded T178392: Replacement hardware for cumin masters a Like token.
Oct 17 2017, 2:02 PM · hardware-requests, Operations
MoritzMuehlenhoff created T178392: Replacement hardware for cumin masters.
Oct 17 2017, 2:01 PM · hardware-requests, Operations
MoritzMuehlenhoff added a comment to T177385: Upgrade Cumin masters to stretch.

Actually when looking at Racktables both neodymium and sarin had their warranty expired in January 2016, so they're pretty close to our usual five years lifespan. So I think it makes sense to not reimage the existing servers, but setup replacement hardware with stretch.

Oct 17 2017, 1:51 PM · Operations-Software-Development, Operations