MoritzMuehlenhoff (Moritz Mühlenhoff)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Apr 1 2015, 4:33 PM (167 w, 4 d)
Availability
Available
LDAP User
Moritz Mühlenhoff
MediaWiki User
MMuhlenhoff (WMF)

Recent Activity

Wed, Jun 13

MoritzMuehlenhoff closed T197088: Upgrading python-requests on trusty as Declined.

This turned out to be a rabbit hole which would also require upgrading urllib3, so in the end it was worked around in the debmonitor client.

Wed, Jun 13, 12:53 PM · Operations
MoritzMuehlenhoff created T197088: Upgrading python-requests on trusty.
Wed, Jun 13, 10:26 AM · Operations

Tue, Jun 12

MoritzMuehlenhoff added a comment to T196989: mailman listing unresponsive (fermium high latency).

Load was in the 120 ballpark and there were total of 141 "/usr/bin/python -S /var/lib/mailman/scripts/driver listinfo" processes running.

Tue, Jun 12, 12:37 PM · Patch-For-Review, Mail, Operations, Wikimedia-Mailing-lists
MoritzMuehlenhoff closed T196745: Add Cassandra 3.11.2 package to internal APT repository, a subtask of T178905: Upgrade RESTBase cluster to Cassandra release: 3.11.2, as Resolved.
Tue, Jun 12, 11:50 AM · Services (done), User-Eevans, Cassandra
MoritzMuehlenhoff closed T196745: Add Cassandra 3.11.2 package to internal APT repository as Resolved.

Imported via Secure Apt (release key is signed by Eric with whom I've signed keys) and added to component/cassandra311 on apt.wikimedia.org

Tue, Jun 12, 11:50 AM · Operations, User-Eevans, Cassandra

Mon, Jun 11

MoritzMuehlenhoff added a comment to T178905: Upgrade RESTBase cluster to Cassandra release: 3.11.2.

restbase-dev1004 and -1006 still have puppet disabled with a note about upgrading Cassandra, safe to re-enable? Ping me when you're around and I'll upload the new 3.11.2 debs to apt.wikimedia.org

Mon, Jun 11, 7:01 AM · Services (done), User-Eevans, Cassandra
MoritzMuehlenhoff closed T195734: mw1280: CPU error as Resolved.

No new errors have been logged in SEL and the server appears stable, closing the task.

Mon, Jun 11, 6:58 AM · Operations, ops-eqiad

Fri, Jun 8

MoritzMuehlenhoff claimed T135991: Automated service restarts for common low-level system services.
Fri, Jun 8, 10:03 AM · Patch-For-Review, Operations
MoritzMuehlenhoff added a comment to T135991: Automated service restarts for common low-level system services.

dbus is installed by default and while I've successfully tested restarts on a number of servers (sodium, db2093, dns4001, mw1318, ores1001), we're erring on the side of caution and are not adding it to automated restarts:

Fri, Jun 8, 10:02 AM · Patch-For-Review, Operations

Thu, Jun 7

MoritzMuehlenhoff created T196659: cloud vps: disable system-wide apt pinning for OpenStack jessie hosts.
Thu, Jun 7, 5:14 PM · Patch-For-Review, cloud-services-team, Operations
MoritzMuehlenhoff added a comment to T195734: mw1280: CPU error.

Repooled, seems fine so far.

Thu, Jun 7, 12:16 PM · Operations, ops-eqiad
MoritzMuehlenhoff added a comment to T195734: mw1280: CPU error.

@Cmjohnson Not seeing a new CPU error logged in "racadm getsel", but it's also still depooled and thus not receiving traffic (and may show up only under load). Unless you wanna do additional tests, I would go ahead and repool it?

Thu, Jun 7, 8:49 AM · Operations, ops-eqiad

Tue, Jun 5

MoritzMuehlenhoff reopened T177371: Phase out DSA keys for SSH access (ssh-dss) as "Open".

There's one remaining bit we need to clean up; I'll take care of it this week: Our puppet templates for sshd still configure a DSA host key, that should also be cleaned up.

Tue, Jun 5, 12:08 PM · Patch-For-Review, Operations
MoritzMuehlenhoff added a comment to T195690: Degraded RAID on labsdb1009.

I just saw it is in the repo but it wasn't installed
We should add it by default on the puppet recipe for HP hosts probably

Tue, Jun 5, 8:43 AM · DBA, ops-eqiad, Operations

Mon, Jun 4

MoritzMuehlenhoff added a comment to T195690: Degraded RAID on labsdb1009.

@Marostegui : hpssaducli is present in the thirdparty/hwraid component for stretch already.

Mon, Jun 4, 5:50 PM · DBA, ops-eqiad, Operations
MoritzMuehlenhoff added a comment to T195690: Degraded RAID on labsdb1009.

@Marostegui : hpssaducli is present in the thirdparty/hwraid component for stretch already.

Mon, Jun 4, 5:45 PM · DBA, ops-eqiad, Operations
MoritzMuehlenhoff claimed T168433: Deprecate DSA (ssh-dss) SSH keys for Cloud VPS and Toolforge users.
Mon, Jun 4, 4:46 PM · Cloud-VPS, Toolforge, cloud-services-team (Kanban)
MoritzMuehlenhoff created P7210 CPU types of database hosts.
Mon, Jun 4, 12:20 PM
MoritzMuehlenhoff assigned T196339: pc2005 down to Papaul.
Mon, Jun 4, 8:36 AM · Patch-For-Review, DBA, Operations, ops-codfw
MoritzMuehlenhoff triaged T196336: Icinga passive checks go awal and downtime stops working as High priority.
Mon, Jun 4, 8:22 AM · Icinga, monitoring
MoritzMuehlenhoff closed T196003: Request mailman list for upcoming affiliate campaign as Resolved.

@MelodyKramer The list has been created, let me know if you need any additional tweaks.

Mon, Jun 4, 7:19 AM · WMF-Blog-Social-Team, Wikimedia-Mailing-lists, Operations
MoritzMuehlenhoff triaged T196192: Add Reedy to contint-docker group as Normal priority.

Patch looks fine, but this request needs to pass the three day waiting/review period.

Mon, Jun 4, 7:17 AM · Patch-For-Review, Operations, SRE-Access-Requests, Continuous-Integration-Infrastructure
MoritzMuehlenhoff assigned T196251: frdb2001 RAID disk failure to Papaul.
Mon, Jun 4, 7:15 AM · Operations, fundraising-tech-ops, ops-codfw
MoritzMuehlenhoff closed T195837: Requesting access to analytics-privatedata-users for gilles as Resolved.

@Gilles You can now log into stat1004.eqiad.wmnet/stat1005.eqiad.wmnet.

Mon, Jun 4, 7:13 AM · Performance-Team (Radar), Patch-For-Review, Analytics, SRE-Access-Requests, Operations
MoritzMuehlenhoff assigned T196260: Degraded RAID on wtp1043 to Cmjohnson.
Mon, Jun 4, 7:12 AM · ops-eqiad, Operations

Fri, Jun 1

MoritzMuehlenhoff triaged T195252: Update prometheus-varnish-exporter on debian to 1.4 as Normal priority.
Fri, Jun 1, 11:44 AM · Operations
MoritzMuehlenhoff closed T189285: Requesting access for bmansurov to run mwscript in terbium as Resolved.

@bmansurov I changed your group membership, please retry.

Fri, Jun 1, 11:30 AM · Ops-Access-Reviews, Patch-For-Review, Operations, SRE-Access-Requests
MoritzMuehlenhoff closed T189285: Requesting access for bmansurov to run mwscript in terbium, a subtask of T184212: Gather labels as ground truth for section translation, as Resolved.
Fri, Jun 1, 11:30 AM · Research-2017-18-Q4, Research-2017-18-Q3, Research
MoritzMuehlenhoff added a comment to T168490: upgrade planet instances to stretch.

There's quite a bit of crom spam by planet2001:

Fri, Jun 1, 9:13 AM · Wikimedia-Planet, Patch-For-Review, Operations
MoritzMuehlenhoff added a comment to T176370: Migrate to PHP 7 in WMF production.

For unattended scripts I don't think the cost is reason to deploy PHP 7 before we know its ready, given the risk of widespread corruption or hard-to-detect issues.

Fri, Jun 1, 8:54 AM · TechCom-RFC (TechCom-Approved), User-ArielGlenn, HHVM, MediaWiki-Platform-Team, Operations

Thu, May 31

MoritzMuehlenhoff triaged T196044: cassandra 2.2.6-wmf4 is not compatible with python 2.7.13 (debian stretch) as Normal priority.
Thu, May 31, 4:29 PM · Patch-For-Review, Discovery, Maps, Cassandra, Operations
MoritzMuehlenhoff assigned T196045: elastic2018 not rebooting to Papaul.
Thu, May 31, 8:59 AM · ops-codfw, Discovery-Search (Current work), DC-Ops, Operations
MoritzMuehlenhoff edited P7187 Parsoid CPU types.
Thu, May 31, 4:39 AM
MoritzMuehlenhoff triaged T196034: Define scap::sources in a way that is shared between prod and beta as Normal priority.
Thu, May 31, 4:37 AM · Performance-Team (Radar), Operations, Puppet, Beta-Cluster-Infrastructure
MoritzMuehlenhoff edited P7187 Parsoid CPU types.
Thu, May 31, 4:30 AM

Wed, May 30

MoritzMuehlenhoff added a comment to T127825: Re-add intel-microcode.

Test servers for Parsoid: https://phabricator.wikimedia.org/P7187

Wed, May 30, 3:26 PM · Patch-For-Review, Operations
MoritzMuehlenhoff added a comment to T195981: require_package should mark packages as manually installed.

Ack, the analysis and the proposed fix seem entirely correct.

Wed, May 30, 3:21 PM · Operations, Puppet
MoritzMuehlenhoff created P7187 Parsoid CPU types.
Wed, May 30, 2:50 PM
MoritzMuehlenhoff edited P7186 Hadoop CPU types.
Wed, May 30, 2:32 PM
MoritzMuehlenhoff edited P7186 Hadoop CPU types.
Wed, May 30, 2:32 PM
MoritzMuehlenhoff added a comment to T190717: Update wikidiff2 library on the WMF production cluster.

Given that the change is now live, shall we close this ticket or do you expect another update soon for the detection of character-based languages?

Wed, May 30, 2:22 PM · Patch-For-Review, wikidiff2, Operations, WMDE-QWERTY-Team
MoritzMuehlenhoff added a comment to T127825: Re-add intel-microcode.

Test servers for Hadoop cluster at https://phabricator.wikimedia.org/P7186

Wed, May 30, 12:24 PM · Patch-For-Review, Operations
MoritzMuehlenhoff edited P7186 Hadoop CPU types.
Wed, May 30, 12:23 PM
MoritzMuehlenhoff edited P7186 Hadoop CPU types.
Wed, May 30, 12:18 PM
MoritzMuehlenhoff edited P7186 Hadoop CPU types.
Wed, May 30, 12:16 PM
MoritzMuehlenhoff edited P7186 Hadoop CPU types.
Wed, May 30, 12:09 PM
MoritzMuehlenhoff created P7186 Hadoop CPU types.
Wed, May 30, 12:06 PM
MoritzMuehlenhoff updated the title for P7148 microcode updates for mw* servers from untitled to microcode updates for mw* servers.
Wed, May 30, 11:27 AM
MoritzMuehlenhoff added a comment to T127825: Re-add intel-microcode.

Test servers for the elastic clusters at https://phabricator.wikimedia.org/P7184

Wed, May 30, 11:26 AM · Patch-For-Review, Operations
MoritzMuehlenhoff edited P7184 elasticsearch CPU types.
Wed, May 30, 11:25 AM
MoritzMuehlenhoff edited P7184 elasticsearch CPU types.
Wed, May 30, 11:16 AM
MoritzMuehlenhoff created P7184 elasticsearch CPU types.
Wed, May 30, 11:11 AM

Tue, May 29

MoritzMuehlenhoff triaged T195797: Stas needs root access on WDQS test cluster as Normal priority.
Tue, May 29, 5:36 PM · Patch-For-Review, Discovery, SRE-Access-Requests, Wikidata, Wikidata-Query-Service, Operations
MoritzMuehlenhoff added a comment to T195837: Requesting access to analytics-privatedata-users for gilles.

@Gilles: Since this is a non-sudo change, it needs to only pass the three day waiting period.

Tue, May 29, 5:36 PM · Performance-Team (Radar), Patch-For-Review, Analytics, SRE-Access-Requests, Operations
MoritzMuehlenhoff triaged T195837: Requesting access to analytics-privatedata-users for gilles as Normal priority.
Tue, May 29, 5:35 PM · Performance-Team (Radar), Patch-For-Review, Analytics, SRE-Access-Requests, Operations
MoritzMuehlenhoff added a comment to T194835: mw2182 crash.

Thanks, I've repooled the sever. I'm keeping an eye on it throughout the week whether it now holds fine.

Tue, May 29, 5:32 PM · ops-codfw, Operations
MoritzMuehlenhoff triaged T195847: Clean up artifacts from LaTeX based math rendering as Normal priority.

Should we split this into three tickets since the actionables (and people acting on those) are fairly disjunct? (So one task to remove it from the Math extension, one task to remove it from the production installation/repositories and one to clean up the database)

Tue, May 29, 3:03 PM · Patch-For-Review, Operations, Math
MoritzMuehlenhoff updated subscribers of T195837: Requesting access to analytics-privatedata-users for gilles.
Tue, May 29, 2:30 PM · Performance-Team (Radar), Patch-For-Review, Analytics, SRE-Access-Requests, Operations
MoritzMuehlenhoff added a comment to T190717: Update wikidiff2 library on the WMF production cluster.

@Lea_WMDE, @WMDE-Fisch : wikidiff 1.7.0 has been rolled out to all our application servers in production (active and backup data centre), you can proceed with enabling it in wmf-config as planned.

Tue, May 29, 2:04 PM · Patch-For-Review, wikidiff2, Operations, WMDE-QWERTY-Team
MoritzMuehlenhoff added a comment to T194835: mw2182 crash.

@MoritzMuehlenhoff yes we do have some decommissioned servers. This can be also a bad main board. What we can do first is to swap CPU position. Since the error is showing on CPU1 we can move CPU1 to CPU0 and CPU0 to CPU1 if we do have the error on CPU0 than the CPU1 is bad if we do have the error on CPU1 than we need to replace the main board.

Tue, May 29, 1:26 PM · ops-codfw, Operations
MoritzMuehlenhoff created T195835: Broken pinning on some WMCS servers.
Tue, May 29, 12:52 PM · Operations
MoritzMuehlenhoff updated the task description for T177371: Phase out DSA keys for SSH access (ssh-dss).
Tue, May 29, 12:14 PM · Patch-For-Review, Operations
MoritzMuehlenhoff added a comment to T195741: reimage maps-test2004 to stretch and cassandra 2.2.

gehel@maps-test2004:~$ apt-cache show openjdk-8-jre-headless=8u171-b11-1~deb9u1 | grep Provides
Provides: java2-runtime-headless, java5-runtime-headless, java6-runtime-headless, java7-runtime-headless, java8-runtime-headless

Is there something I don't understand here (well, I'm sure there are tons of things I don't understand...) ?
Tue, May 29, 7:35 AM · Patch-For-Review, Operations, Maps-Sprint

Mon, May 28

MoritzMuehlenhoff added a comment to T195741: reimage maps-test2004 to stretch and cassandra 2.2.

It looks like we have a few missing dependencies in Stretch:

  • jvm-tools (only available on jessie, could probably just be copied to stretch, need to check if the java7-runtime-headless dep is satisfied by openjdk-8)
Mon, May 28, 5:58 PM · Patch-For-Review, Operations, Maps-Sprint
MoritzMuehlenhoff added a comment to T190717: Update wikidiff2 library on the WMF production cluster.

Status update: Half of our active data centre and the majority of servers in our backup DC have been upgraded to wikidiff 1.7. The rest will follow tomorrow.

Mon, May 28, 1:50 PM · Patch-For-Review, wikidiff2, Operations, WMDE-QWERTY-Team
MoritzMuehlenhoff claimed T190717: Update wikidiff2 library on the WMF production cluster.
Mon, May 28, 1:49 PM · Patch-For-Review, wikidiff2, Operations, WMDE-QWERTY-Team
MoritzMuehlenhoff triaged T195553: Puppet class systemd needs to throw a more useful error as Low priority.
Mon, May 28, 1:20 PM · Operations, Puppet, cloud-services-team
MoritzMuehlenhoff assigned T195306: Degraded RAID on elastic2020 to Papaul.
Mon, May 28, 11:27 AM · Operations, ops-codfw
MoritzMuehlenhoff triaged T194835: mw2182 crash as Normal priority.
Mon, May 28, 11:25 AM · ops-codfw, Operations
MoritzMuehlenhoff added a comment to T194835: mw2182 crash.

The server is out of warranty since January. @Papaul: Do we have any decommissioned servers from which we could swap the broken CPU?

Mon, May 28, 11:25 AM · ops-codfw, Operations
MoritzMuehlenhoff triaged T195364: Remove pear/mail packages from WMF MW app servers as Normal priority.
Mon, May 28, 11:13 AM · Patch-For-Review, Wikimedia-General-or-Unknown, Operations
MoritzMuehlenhoff created T195734: mw1280: CPU error.
Mon, May 28, 7:48 AM · Operations, ops-eqiad

Fri, May 25

MoritzMuehlenhoff assigned T195569: Degraded RAID on ms-be1034 to Cmjohnson.
Fri, May 25, 10:24 AM · ops-eqiad, Operations
MoritzMuehlenhoff added a comment to T190568: Reimage both phab1001 and phab2001 to stretch.

We could also avoid downtime by temporarily reusing mw1298 (former image scaler) and reinstalling it as phab1002 with stretch. Then we can switch to phab1002 and reimage phab1001 (with an eventual switchback to phab1001/stretch) without having a Phabricator downtime of > 2 hours. The specs are roughly the same, phab1001 has a slightly more powerful CPU than mw1298, but both have 64 GB RAM and looking at Prometheus CPU usage is usually ~ 25% so that be fine.

Fri, May 25, 5:32 AM · Patch-For-Review, Release-Engineering-Team (Watching / External), Phabricator, Operations

Thu, May 24

MoritzMuehlenhoff added a comment to T127825: Re-add intel-microcode.

Six out of the mw* servers have been switched to using microcode updates.

Thu, May 24, 6:34 PM · Patch-For-Review, Operations
MoritzMuehlenhoff added a comment to T190717: Update wikidiff2 library on the WMF production cluster.

The mwdebug have also been upgraded.

Thu, May 24, 2:54 PM · Patch-For-Review, wikidiff2, Operations, WMDE-QWERTY-Team
MoritzMuehlenhoff added a comment to T190717: Update wikidiff2 library on the WMF production cluster.

@Lea_WMDE, @WMDE-Fisch : The canary application servers have been upgraded and so far everything looks fine in the logs. I'll keep an eye on it, but I think we're good to proceed with the wider rollout next Monday.

Thu, May 24, 2:04 PM · Patch-For-Review, wikidiff2, Operations, WMDE-QWERTY-Team
MoritzMuehlenhoff added a comment to T190717: Update wikidiff2 library on the WMF production cluster.

@Lea_WMDE Ack, I'll start upgrading the mediawiki canaries later the (CEST) afternoon.

Thu, May 24, 12:01 PM · Patch-For-Review, wikidiff2, Operations, WMDE-QWERTY-Team
MoritzMuehlenhoff created P7148 microcode updates for mw* servers.
Thu, May 24, 9:29 AM
MoritzMuehlenhoff added a comment to T190568: Reimage both phab1001 and phab2001 to stretch.

PHP 7.2 packages for stretch are available since early March via thirdparty/php72, let me know if anything is missing

Thu, May 24, 6:34 AM · Patch-For-Review, Release-Engineering-Team (Watching / External), Phabricator, Operations

Wed, May 23

MoritzMuehlenhoff added a comment to T190717: Update wikidiff2 library on the WMF production cluster.

@Lea_WMDE : Sure, no problem!

Wed, May 23, 5:02 PM · Patch-For-Review, wikidiff2, Operations, WMDE-QWERTY-Team
MoritzMuehlenhoff added a comment to T190717: Update wikidiff2 library on the WMF production cluster.

Just to let you know, the config change will be done today so we are good to go for next Monday. I will see what I can do. What's the approximate time your planing this?

Wed, May 23, 8:12 AM · Patch-For-Review, wikidiff2, Operations, WMDE-QWERTY-Team

Tue, May 22

MoritzMuehlenhoff added a comment to T190717: Update wikidiff2 library on the WMF production cluster.

So for deployment on beta nothing special is needed and it can be done right away.

Tue, May 22, 11:56 AM · Patch-For-Review, wikidiff2, Operations, WMDE-QWERTY-Team

May 16 2018

MoritzMuehlenhoff added a comment to T190717: Update wikidiff2 library on the WMF production cluster.

Hi @MoritzMuehlenhoff, we have news :)

  • The changes for mobile are now part of the last master (should be version 1.7) now. So we are ready for beta again
May 16 2018, 3:00 PM · Patch-For-Review, wikidiff2, Operations, WMDE-QWERTY-Team
MoritzMuehlenhoff added a comment to T158562: Manage apt sources via puppet?.

Another case I found: ms-be2013-ms-be2021 were unable to install the systemd update that was released via stretch-updates and it turned that that stretch-updates was missing in /etc/apt/sources.list. Those were among the first stretch hosts, so this was probably a one-off issue in early installations only. I've fixed up the apt config on the affected systems.

May 16 2018, 1:51 PM · Operations
MoritzMuehlenhoff added a comment to T194665: Provide an up-to-date mono environment on toolforge.

I have some concerns on using external repos. It could be catastrophic with regards to the integration with the rest of the system. I'm not sure if it worth the risk, considering the scope of the problem.
I always prefer using official debian/ubuntu package as long as possible.

Exactly. https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Local_package_policy

Well, I ignored we had such a policy.

I therefore summon @bd808 for additional advice. Summary:

  • importing the whole mono-project repo into prod reprepro seems overkill, given the only user would be toolforge
May 16 2018, 10:28 AM · cloud-services-team, Toolforge
MoritzMuehlenhoff added a comment to T194814: Reduce amount of headers sent from web responses.

We could also simply avoid X-Powered-By at the source; our PHP configs already use "expose_php=off" and for HHVM per https://github.com/facebook/hhvm/issues/2343 adding "expose_php = 0" to server.ini would be the HHVM equivalent.

May 16 2018, 10:22 AM · Performance-Team (Radar), Patch-For-Review, media-storage, Traffic, Operations

May 15 2018

MoritzMuehlenhoff added a comment to T194665: Provide an up-to-date mono environment on toolforge.

We will also explore the possibility of importing the whole framework in the main reprepro apt repo. Either all the packages or a 'pointer' to the external repo. @MoritzMuehlenhoff what do you think?

May 15 2018, 4:37 PM · cloud-services-team, Toolforge
MoritzMuehlenhoff added a comment to T127825: Re-add intel-microcode.

We have two clusters which need updated microcode to provide support for the new IBPB instruction needed to secure KVM instances against Spectre. In addition to that keeping the microcode updated it also recommended by upstream since microcode updates can also address functional CPU issues.

May 15 2018, 12:32 PM · Patch-For-Review, Operations

May 14 2018

MoritzMuehlenhoff added a comment to T175288: setup/install/deploy deploy1001 as deployment server.

And, let's do this on friday, that leaves us until monday's SWAT (if any).

May 14 2018, 5:30 PM · Patch-For-Review, Release-Engineering-Team (Watching / External), Operations
MoritzMuehlenhoff added a comment to T175288: setup/install/deploy deploy1001 as deployment server.

How do you guys feel about the maintenance server, terbium -> mwmaint1001 (T192092), should we do that first or after tin/deploy1001?

May 14 2018, 5:28 PM · Patch-For-Review, Release-Engineering-Team (Watching / External), Operations
MoritzMuehlenhoff added a comment to T191921: mwscript rebuildLocalisationCache.php takes 40 minutes on HHVM (rather than ~5 on PHP 5).

(..) I'm not sure all of that is necessary if we're just talking about flipping one script over to PHP 7 right now.

May 14 2018, 11:06 AM · Patch-For-Review, Operations, Release-Engineering-Team (Kanban), Scap
MoritzMuehlenhoff added a comment to T175288: setup/install/deploy deploy1001 as deployment server.

I suggest we do the following:

  • Pick a date/time frame of a few hours where no deployments are happening (or cancel existing ones)
  • We switch the deployment server to deploy1001 and test deployments using the PHP7 setup present there
  • If anything breaks, we revert to tin and fix whatever problem we found with PHP 7 and deploy1001 and re-attempt at a later stage
  • If everything works fine, we keep tin for a few weeks as a fallback and then decom it.
May 14 2018, 11:04 AM · Patch-For-Review, Release-Engineering-Team (Watching / External), Operations
MoritzMuehlenhoff updated the task description for T194268: Restart Analytics hosts for Java 8 Security upgrades.
May 14 2018, 10:19 AM · Analytics-Kanban, Analytics, User-Elukey
MoritzMuehlenhoff edited P6996 mediawiki reimages.
May 14 2018, 8:49 AM
MoritzMuehlenhoff assigned T194426: mw2139 failed to boot - hardware check to Papaul.
May 14 2018, 7:07 AM · Patch-For-Review, ops-codfw, DC-Ops, Operations

May 9 2018

MoritzMuehlenhoff edited P6996 mediawiki reimages.
May 9 2018, 2:52 PM
MoritzMuehlenhoff added a comment to T190717: Update wikidiff2 library on the WMF production cluster.

Do you have a time estimate for those mobile changes? Totally depends on the schedule I'd say. Also, let me know when those are ready for deployment to beta ahead of production.

We are currently in the process of making sure the mobile changes do what we expect them to do. If all goes well, we could be ready for beta in 2 weeks.

May 9 2018, 2:30 PM · Patch-For-Review, wikidiff2, Operations, WMDE-QWERTY-Team
MoritzMuehlenhoff closed T88994: use graphite1002 to test dm-cache as Declined.

Host was decomissioned in T187190

May 9 2018, 2:18 PM · Operations, Patch-For-Review, Graphite
MoritzMuehlenhoff closed T88994: use graphite1002 to test dm-cache, a subtask of T88992: consider hybrid caching options for ssd+disk, as Declined.
May 9 2018, 2:18 PM · Operations, Patch-For-Review