Page MenuHomePhabricator

Build a non-trunk 3.19 kernel for jessie
Closed, ResolvedPublic

Event Timeline

BBlack assigned this task to MoritzMuehlenhoff.
BBlack raised the priority of this task from to High.
BBlack updated the task description. (Show Details)
BBlack added projects: Traffic, acl*sre-team.

Change 208601 had a related patch set uploaded (by Muehlenhoff):
Update to 3.19.4 (Bug: T97411)

https://gerrit.wikimedia.org/r/208601

Change 208602 had a related patch set uploaded (by Muehlenhoff):
Update to 3.19.5 (Bug: T97411)

https://gerrit.wikimedia.org/r/208602

Change 208662 had a related patch set uploaded (by Muehlenhoff):
Update to 3.19.6 (Bug: T97411)

https://gerrit.wikimedia.org/r/208662

Change 209181 had a related patch set uploaded (by Muehlenhoff):
(Bug: T97411) Refresh the control file and change the version scheme; we forked off the last 3.19 Debian upload (3.19.3) and all further updates will be folded in via the stable patchsets.

https://gerrit.wikimedia.org/r/209181

BBlack lowered the priority of this task from High to Medium.May 6 2015, 3:57 PM
BBlack moved this task from Upcoming to Traffic team actively servicing on the Traffic board.

Change 208601 merged by Muehlenhoff:
Update to 3.19.4 (Bug: T97411)

https://gerrit.wikimedia.org/r/208601

Change 208602 merged by Muehlenhoff:
Update to 3.19.5 (Bug: T97411)

https://gerrit.wikimedia.org/r/208602

Change 208662 merged by Muehlenhoff:
Update to 3.19.6 (Bug: T97411)

https://gerrit.wikimedia.org/r/208662

Change 209181 merged by Muehlenhoff:
(Bug: T97411) Refresh the control file and change the version scheme; we forked off the last 3.19 Debian upload (3.19.3) and all further updates will be folded in via the stable patchsets.

https://gerrit.wikimedia.org/r/209181

The kernel is now in operations/deb/linux git (currently updated to 3.19.6) and available on apt.wikimedia.org in the jessie-wikimedia suite.

I'll add an additional meta package so that we can ensure updates if we need to bump the ABI (resulting in new binary names for the debs) or switch to a future 4.x kernel.

It's running on cp1008 + cp3030 now for testing as well, looks fine so far.

I've added the meta package to operations/debs/linux-meta.git. It has been built on copper and is available on apt.wikimedia.org

"apt-get install linux-meta" should always setup the most recent kernel, even if we need an ABI bump or move to 4.x later.

If we decide to use this kernel for all jessie servers, we should also update the d-i profiles to use the meta package for new installations

The meta package would also be the canonical place if we need to depend on firmware not present in the stock linux package or to set deps for kernel-related packages installed across the fleet (perf, irqbalance, etc.)

I've updated cp1070 via the meta package, everything worked out there with automatic initramfs/grub/etc. I also updated our hacky late_command stuff for the caches in case of new-/re- installs for now. Some of that could be eliminated if we did leverage the meta package and d-i and such, but it will work for now.

Personally, I think we probably should just go ahead and have the jessie installer d-i use our kernel for all jessies going forward, and perhaps bundle in our firmware update as a dep too?

The firmware issue is: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779128 . Upstream has yet to release a 0.44 package containing the required files we're missing from 0.43 for bnx2x adapters with 3.18+, so I stuck a hacky version 0.43-1~wmf1 in our local apt repo to work around it for now...

Personally, I think we probably should just go ahead and have the jessie installer d-i use our kernel for all jessies going forward, and perhaps bundle in our firmware update as a dep too?

I would agree with that. Even if the mm problems you saw with the varnish caches are not exposed with other use cases, it would be good to have consistency in behaviour across all jessie systems.

Maybe let's propose this on the ops@ exploder, so that everyone is in the loop and can raise a concern?

This kernel is now installed on berkelium & curium.

  • IPsec ESNs work (fixed in 3.19.3)
  • Aesni security patch for CVE-2015-3331 is included (fixed in 3.19.3)
  • Aes256gcm does not work. (fixed in 4.0, but we don't care because we plan to use aes128gcm which works in 3.19.)

This kernel is fine from an IPsec perspective and personally I think using this kernel for all Jessie nodes sounds preferable to having a mix of kernels on different hosts.

Change 211688 had a related patch set uploaded (by Muehlenhoff):
Use 3.19 on jessie by default (Bug: T97411)

https://gerrit.wikimedia.org/r/211688

BBlack moved this task from Traffic team actively servicing to Done on the Traffic board.