Page MenuHomePhabricator
Create Task
Maniphest T127825

Re-add intel-microcode
Closed, ResolvedPublic
Actions

Description

intel-microcode was initially added to the standard packages with https://phabricator.wikimedia.org/rOPUPe6e960b69d6764f5fbe5b1bf513c8a60be008696 but later on reverted with https://phabricator.wikimedia.org/rOPUP0fccee1f88fb312069f94fa634cb2d3b8205651c due to CPU frequency problems.

I'd like to re-add intel-microcode, since it fixes both stability/correctness bugs and around 2013 Intel also used microcode updates to address security problems on the CPU level. (These vulnerabilitiesare also addressed by kernel changes, but there might be some in the future which not easily maskable by the kernel, so I'd also be good to be prepared)

One way to add this gradually would be to not add intel-microcode to standard-packages.pp, but rather to the new meta package for the 4.4 kernel, that way all systems would get it step by step as we move to 4.4 (it needs a reboot to become effective anyway).

Comments/objections?

(We have only a single server with AMD CPUs (stat1001), that one can be dealt with manually)

Details

Show related patches Customize query in gerrit

Event Timeline

MoritzMuehlenhoff created this task.Feb 23 2016, 12:18 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 23 2016, 12:18 PM
gerritbot subscribed.Sep 25 2016, 2:19 AM

Change 312714 had a related patch set uploaded (by Matanya):
standard packages: re-add intel-microcode

https://gerrit.wikimedia.org/r/312714

gerritbot added a project: Patch-For-Review.Sep 25 2016, 2:19 AM
BBlack added a comment.Sep 27 2016, 8:39 AM

+1 on tying this to 4.4+ and it being a good idea to get it going again. I'm not sure (unless someone's investigated already) that our latest 3.1[69] and such kernels don't still have the initramfs cpu frequency issues when installing intel-microcode, but I'd guess 4.4 is safe (also needs testing)?

MoritzMuehlenhoff triaged this task as Medium priority.Feb 7 2017, 12:34 PM
elukey subscribed.Jan 8 2018, 2:01 PM
akosiaris subscribed.Jan 8 2018, 4:16 PM
• ema subscribed.Jan 9 2018, 9:20 AM
MoritzMuehlenhoff added a comment.May 15 2018, 12:31 PM

We have two clusters which need updated microcode to provide support for the new IBPB instruction needed to secure KVM instances against Spectre. In addition to that keeping the microcode updated it also recommended by upstream since microcode updates can also address functional CPU issues.

Updated microcode is only picked up during reboots and we'll enable this per cluster (or rather per hardware generation per cluster for the bigger clusters compromised of several generations of servers). If at some point we're confident to make it the new default, we can simply enable it in standard_packages

gerritbot added a comment.May 15 2018, 2:48 PM

Change 433160 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Allow enabling microcode updates gradually

https://gerrit.wikimedia.org/r/433160

gerritbot added a comment.May 16 2018, 11:18 AM

Change 433160 merged by Muehlenhoff:
[operations/puppet@production] Allow enabling microcode updates gradually

https://gerrit.wikimedia.org/r/433160

MoritzMuehlenhoff added a comment.EditedMay 24 2018, 6:34 PM

Six out of the mw* servers have been switched to using microcode updates. (https://phabricator.wikimedia.org/P7148)

Stashbot subscribed.May 28 2018, 11:31 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-28T11:31:34Z] <ema> cp1008: reboot with intel-microcode T127825

Stashbot added a comment.May 28 2018, 2:21 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-28T14:21:20Z] <ema> cp1045,cp2001,cp3007,cp5001: reboot with intel-microcode T127825

gerritbot added a comment.May 30 2018, 10:50 AM

Change 436271 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable microcode updates for all mediawiki servers

https://gerrit.wikimedia.org/r/436271

MoritzMuehlenhoff added a comment.May 30 2018, 11:26 AM

Test servers for the elastic clusters at https://phabricator.wikimedia.org/P7184

MoritzMuehlenhoff added a comment.May 30 2018, 12:24 PM

Test servers for Hadoop cluster at https://phabricator.wikimedia.org/P7186

• ema added a comment.May 30 2018, 1:37 PM

It would be useful to check if a new microcode is available (and thus a system restart is needed). Something along these lines should do the trick:

#!/bin/bash

running="$(awk '/microcode/ { printf "%x", $3; exit }' /proc/cpuinfo)"
available="$(sudo iucode-tool -tb -lS /lib/firmware/intel-ucode/* 2> /dev/null | awk 'NR>1 { printf "%x", $9 ; exit }')"

msg="running microcode 0x${running}, 0x${available} available"

if [ "${running}" = "${available}" ]; then
    echo "OK - ${msg}"
    exit 0
else
    echo "WARNING - ${msg}. Reboot needed."
    exit 1
fi

Prerequisites:

  • iucode-tool (which is a dependecy of intel-microcode)
  • the cpuid kernel module needs to be loaded
• ema added a comment.May 30 2018, 2:27 PM

The following cache hosts have been running with updated microcodes for the past two days:

HostCPUMicrocode version on dieUpdated microcode version
cp2001CPU E 5-2680 v3 @ 2.50GHz0x2d0x3c
cp5001CPU E 5-2650 v4 @ 2.20GHz0xb0000210xb00002c
cp3007CPU E 5-2640 0 @ 2.50GHz0x70a0x713
cp1045CPU E 5-2680 0 @ 2.70GHz0x70d0x713

I've checked the output of lscpu and none of the above exhibit the low-frequency issues mentioned here: https://phabricator.wikimedia.org/rOPUP0fccee1f88fb312069f94fa634cb2d3b8205651c

MoritzMuehlenhoff added a comment.May 30 2018, 3:26 PM

Test servers for Parsoid: https://phabricator.wikimedia.org/P7187

gerritbot added a comment.May 31 2018, 9:46 AM

Change 436490 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache hosts: enable microcode updates

https://gerrit.wikimedia.org/r/436490

gerritbot added a comment.May 31 2018, 10:16 AM

Change 436271 merged by Muehlenhoff:
[operations/puppet@production] Enable microcode updates for all mediawiki servers

https://gerrit.wikimedia.org/r/436271

gerritbot added a comment.May 31 2018, 12:11 PM

Change 436490 merged by Ema:
[operations/puppet@production] cache hosts: enable microcode updates

https://gerrit.wikimedia.org/r/436490

gerritbot added a comment.May 31 2018, 2:33 PM

Change 436553 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus: export intel-microcode information via node_exporter

https://gerrit.wikimedia.org/r/436553

gerritbot added a comment.Jun 4 2018, 9:53 AM

Change 437206 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable microcode for Hadoop cluster

https://gerrit.wikimedia.org/r/437206

gerritbot added a comment.Jun 4 2018, 10:45 AM

Change 437206 merged by Muehlenhoff:
[operations/puppet@production] Enable microcode for Hadoop cluster

https://gerrit.wikimedia.org/r/437206

gerritbot added a comment.Jun 5 2018, 9:58 AM

Change 437430 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable microcode updates for Parsoid hosts

https://gerrit.wikimedia.org/r/437430

gerritbot added a comment.Jun 5 2018, 11:29 AM

Change 437430 merged by Muehlenhoff:
[operations/puppet@production] Enable microcode updates for Parsoid hosts

https://gerrit.wikimedia.org/r/437430

gerritbot added a comment.Jun 5 2018, 12:49 PM

Change 437464 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] Move microcode hiera call to profile::base

https://gerrit.wikimedia.org/r/437464

gerritbot added a comment.Jun 5 2018, 2:13 PM

Change 437464 merged by Ema:
[operations/puppet@production] Move microcode hiera call to profile::base

https://gerrit.wikimedia.org/r/437464

gerritbot added a comment.Jun 6 2018, 12:25 PM

Change 436553 merged by Ema:
[operations/puppet@production] prometheus: export intel-microcode information via node_exporter

https://gerrit.wikimedia.org/r/436553

gerritbot added a comment.Jun 27 2018, 10:30 AM

Change 442269 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable microcode for all database roles

https://gerrit.wikimedia.org/r/442269

gerritbot added a comment.Jun 28 2018, 1:48 PM

Change 442269 merged by Muehlenhoff:
[operations/puppet@production] Enable microcode for all database roles

https://gerrit.wikimedia.org/r/442269

gerritbot added a comment.Jun 29 2018, 7:50 AM

Change 443038 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable microcode updates for all elasticsearch hosts

https://gerrit.wikimedia.org/r/443038

gerritbot added a comment.Jun 29 2018, 10:15 AM

Change 443038 merged by Muehlenhoff:
[operations/puppet@production] Enable microcode updates for all elasticsearch hosts

https://gerrit.wikimedia.org/r/443038

gerritbot added a comment.Jul 5 2018, 11:15 AM

Change 443944 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable microcode updates for CI masters

https://gerrit.wikimedia.org/r/443944

gerritbot added a comment.Jul 5 2018, 11:17 AM

Change 443944 merged by Muehlenhoff:
[operations/puppet@production] Enable microcode updates for CI masters

https://gerrit.wikimedia.org/r/443944

Stashbot added a comment.Jul 9 2018, 7:40 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-09T07:40:36Z] <ema> reboot lvs canaries for microcode updates: lvs4007 lvs3004 lvs2006 lvs2010 lvs1006 T127825

gerritbot added a comment.Jul 13 2018, 8:02 AM

Change 445573 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable microcode on restbase servers

https://gerrit.wikimedia.org/r/445573

gerritbot added a comment.Jul 13 2018, 8:34 AM

Change 445576 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable microcode for Swift backend servers

https://gerrit.wikimedia.org/r/445576

gerritbot added a comment.Jul 16 2018, 8:28 AM

Change 445573 merged by Muehlenhoff:
[operations/puppet@production] Enable microcode on restbase servers

https://gerrit.wikimedia.org/r/445573

gerritbot added a comment.Jul 16 2018, 8:36 AM

Change 445576 merged by Muehlenhoff:
[operations/puppet@production] Enable microcode for Swift backend servers

https://gerrit.wikimedia.org/r/445576

gerritbot added a comment.Aug 9 2018, 3:14 PM

Change 451646 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable microcode on spare hosts

https://gerrit.wikimedia.org/r/451646

gerritbot added a comment.Aug 9 2018, 5:49 PM

Change 451646 merged by Muehlenhoff:
[operations/puppet@production] Enable microcode on spare hosts

https://gerrit.wikimedia.org/r/451646

gerritbot added a comment.Aug 10 2018, 8:27 AM

Change 451830 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable microcode for WMCS puppet masters

https://gerrit.wikimedia.org/r/451830

gerritbot added a comment.Aug 10 2018, 8:36 AM

Change 451830 merged by Muehlenhoff:
[operations/puppet@production] Enable microcode for WMCS puppet masters

https://gerrit.wikimedia.org/r/451830

gerritbot added a comment.Aug 10 2018, 2:20 PM

Change 451858 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable microcode for LVS load balancers

https://gerrit.wikimedia.org/r/451858

gerritbot added a comment.Aug 13 2018, 8:17 AM

Change 451858 merged by Muehlenhoff:
[operations/puppet@production] Enable microcode for LVS load balancers

https://gerrit.wikimedia.org/r/451858

gerritbot added a comment.Aug 20 2018, 11:14 AM

Change 453997 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable intel-microcode for all bare metal servers with an Intel CPU

https://gerrit.wikimedia.org/r/453997

gerritbot added a comment.Aug 21 2018, 7:47 AM

Change 453997 merged by Muehlenhoff:
[operations/puppet@production] Enable intel-microcode for all bare metal servers with an Intel CPU

https://gerrit.wikimedia.org/r/453997

gerritbot added a comment.Aug 23 2018, 8:20 AM

Change 454203 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove enable_microcode logic

https://gerrit.wikimedia.org/r/454203

gerritbot added a comment.Sep 1 2018, 9:26 PM

Change 312714 abandoned by Matanya:
standard packages: re-add intel-microcode

Reason:
Per comment

https://gerrit.wikimedia.org/r/312714

gerritbot added a comment.Sep 3 2018, 11:41 AM

Change 454203 merged by Muehlenhoff:
[operations/puppet@production] Remove enable_microcode logic

https://gerrit.wikimedia.org/r/454203

MoritzMuehlenhoff closed this task as Resolved.Sep 3 2018, 11:48 AM

Microcode is now enabled on all baremetal servers with an Intel CPU and we haven't seen any issues so far. Closing the task.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits