Page MenuHomePhabricator

toolforge: pin key packages by version
Closed, ResolvedPublic

Description

Let's create apt pinning for some key packages by versions on tooldorge instances.

Benefits:

  • we have well-known kernel version in all nodes
  • nodes don't upgrade kernel packages
  • more 'software-defined' infra

Proposed packages:

  • linux kernel
  • pam libs
  • kubernetes packages
  • nginx

Event Timeline

aborrero created this task.

Change 410177 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] WIP: toollabs: add apt pinnings for key packages

https://gerrit.wikimedia.org/r/410177

aborrero renamed this task from toolforge: pin kernels by version to toolforge: pin key packages by version.Feb 13 2018, 5:25 PM
aborrero updated the task description. (Show Details)

Change 410177 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: add apt pinnings for key packages

https://gerrit.wikimedia.org/r/410177

Change 413202 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: add apt pinnings for key packages

https://gerrit.wikimedia.org/r/413202

Change 413202 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: add apt pinnings for key packages

https://gerrit.wikimedia.org/r/413202

Change 413206 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] toolforge: apply pinning to k8s components

https://gerrit.wikimedia.org/r/413206

Change 413206 merged by Rush:
[operations/puppet@production] toolforge: apply pinning to k8s components

https://gerrit.wikimedia.org/r/413206

Change 413213 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] toolforge: update pin for kubernetes-client

https://gerrit.wikimedia.org/r/413213

Change 413213 merged by Rush:
[operations/puppet@production] toolforge: update pin for kubernetes-client

https://gerrit.wikimedia.org/r/413213

There is something weird with trusty linux pinning.

The pinning we have is (file /etc/apt/preferences.d/toolforge_linux_pinning.pref):

Package: linux-image-generic
Pin: version 3.13.0.141.151
Pin-Priority: 1001

But then, if I see what's pending an update, I see:

aborrero@tools-bastion-02:~$ sudo apt-upgrade -u report
tools-bastion-02: trusty-updates: linux-tools-common 3.13.0-141.190 --> 3.13.0-142.191 
tools-bastion-02: trusty-updates: linux-image-virtual 3.13.0.141.151 --> 3.13.0.142.152 
tools-bastion-02: trusty-updates: linux-tools-generic 3.13.0.141.151 --> 3.13.0.142.152 
tools-bastion-02: trusty-updates: linux-tools-3.13.0-142-generic [absent] --> 3.13.0-142.191 
tools-bastion-02: trusty-updates: linux-tools-3.13.0-142 [absent] --> 3.13.0-142.191 
tools-bastion-02: trusty-updates: gcj-4.8-jre-lib 4.8.4-2ubuntu1~14.04.3 --> 4.8.4-2ubuntu1~14.04.4 
tools-bastion-02: trusty-updates: linux-image-3.13.0-142-generic [absent] --> 3.13.0-142.191 
tools-bastion-02: trusty-updates: linux-libc-dev 3.13.0-141.190 --> 3.13.0-142.191

The important packages here are:

  • linux-image-virtual which is virtual package which depends to linux-image-XXXX-generic
  • linux-image-3.13.0-142-generic which is the actual kernel package

This gets complicated because the /boot/directory doesn't show the complete version string of the kernel being run:

aborrero@tools-bastion-02:~$ ll /boot/vmlinuz-*
-rw------- 1 root root 5852400 Dec  4 20:50 /boot/vmlinuz-3.13.0-137-generic
-rw------- 1 root root 5856336 Jan  9 15:29 /boot/vmlinuz-3.13.0-139-generic
-rw------- 1 root root 5856528 Jan 19 14:32 /boot/vmlinuz-3.13.0-141-generic

And we are not even running a kernel from one of the involved packages:

aborrero@tools-bastion-02:~$ uname -vr
3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018

And this seems rather heterogeneous over the fleet:

tools-bastion-02.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-bastion-03.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-bastion-05.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-checker-01.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-checker-02.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-clushmaster-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-cron-01.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-docker-builder-05.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-docker-registry-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-docker-registry-02.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-elastic-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-elastic-02.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-elastic-03.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-exec-1401.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1402.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1403.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1404.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1405.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1406.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1407.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1408.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1409.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1410.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1411.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1412.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1413.tools.eqiad.wmflabs: 3.13.0-141-generic #190-Ubuntu SMP Fri Jan 19 12:52:38 UTC 2018
tools-exec-1414.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1415.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1416.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1417.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1418.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1419.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1420.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1421.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1422.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1423.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1424.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1425.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1426.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1427.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1428.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1429.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1430.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1431.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1432.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1433.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1434.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1435.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1436.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1437.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1438.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1439.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1440.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1441.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1442.tools.eqiad.wmflabs: 3.13.0-141-generic #190-Ubuntu SMP Fri Jan 19 12:52:38 UTC 2018
tools-exec-gift-trusty-01.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-flannel-etcd-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-flannel-etcd-02.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-flannel-etcd-03.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-grid-master.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-grid-shadow.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-k8s-etcd-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-k8s-etcd-02.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-k8s-etcd-03.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-k8s-master-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-logs-02.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-mail.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-package-builder-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-paws-master-01.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1001.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1002.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1003.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1005.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1006.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1007.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1010.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1013.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1016.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1017.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1019.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-prometheus-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-prometheus-02.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-proxy-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-proxy-02.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-redis-1001.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-redis-1002.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-services-01.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-services-02.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-static-10.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-static-11.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-static-12.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-webgrid-generic-1401.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-generic-1402.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-generic-1403.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-generic-1404.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1401.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1402.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1403.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1404.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1405.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1406.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1407.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1408.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1409.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1410.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1412.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1413.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1414.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1415.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1416.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1417.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1418.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1419.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1420.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1421.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1422.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1424.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1425.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1426.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1427.tools.eqiad.wmflabs: 3.13.0-141-generic #190-Ubuntu SMP Fri Jan 19 12:52:38 UTC 2018
tools-webgrid-lighttpd-1428.tools.eqiad.wmflabs: 3.13.0-141-generic #190-Ubuntu SMP Fri Jan 19 12:52:38 UTC 2018
tools-worker-1001.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1002.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1003.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1004.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1005.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1006.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1007.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1008.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1009.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1010.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1011.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1012.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1013.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1014.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1015.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1016.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1017.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1018.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1019.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1020.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1021.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1022.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1023.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1025.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1026.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1027.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)

Our current pinning only affects linux-image-generic (v3.13.0.141.151) which is virtual and depends on: linux-image-3.13.0-141-generic.
So, installing both kernels 3.13.0.141.151and 3.13.0.142.152 isn't a policy violation and the resolves just tries it.

Proposed solution (for ubuntu trusty) is to either:

  • extend the pinning to linux-image-virtual as well
  • ban linux-image-virtual from being installed at all, since we are installing linux-image-generic for our running kernel

For jessie and stretch, another similar analysis should be conducted.

Change 413725 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: apt_pinning: be more strict in linux kernel pinning

https://gerrit.wikimedia.org/r/413725

Change 413725 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: apt_pinning: be more strict in linux kernel pinning

https://gerrit.wikimedia.org/r/413725

For some reasons Ubuntu makes every kernel package co-installable (in Debian the binary package is only changed if the kernel ABI changes). So you'll probably need to manually update the latest trusty kernel you want to install and pin to that specific binary package name.

More weird stuff.

aborrero@tools-proxy-01:~$ sudo apt-upgrade -u report
tools-proxy-01: jessie-backports: linux-image-4.9.0-0.bpo.4-amd64 4.9.51-1~bpo8+1 --> 4.9.65-3+deb9u1~bpo8+1 
tools-proxy-01: jessie-wikimedia: linux-meta-4.9 1.16 --> 1.17 
tools-proxy-01: jessie-wikimedia: linux-image-4.9.0-0.bpo.6-amd64 [absent] --> 4.9.82-1~wmf1 
tools-proxy-01: now: linux-meta 1.16 --> [remove] 
tools-proxy-01: oldstable: libpam0g 1.1.8-3.1+deb8u1+b1 --> 1.1.8-3.1+deb8u2+b1 
tools-proxy-01: oldstable: libpam-modules 1.1.8-3.1+deb8u1+b1 --> 1.1.8-3.1+deb8u2+b1 
tools-proxy-01: oldstable: libpam-ldapd 0.9.4-3+deb8u1 --> 0.9.4-3+deb8u2 
tools-proxy-01: oldstable: libpam-modules-bin 1.1.8-3.1+deb8u1+b1 --> 1.1.8-3.1+deb8u2+b1 
tools-proxy-01: oldstable-updates: linux-image-3.16.0-4-amd64 3.16.43-2+deb8u5 --> 3.16.51-3
  • there is a now archive which is a fake one, meaning the package candidate is not being pulled from any archive (is being removed). That is probably something to address in the apt-upgrade code, i.e. print ([none]) or something like that
  • the linux-meta-4.9 package is being upgraded. I would expect it to be following the Depends of linux-meta, but that's not true. A patch will follow-up to address it.
  • the linux-meta package is being removed. I couldn't figure out why, since we have a proper pinning for it. This is even more weird if we inspect the policy for it (see below).
aborrero@tools-proxy-01:~$ apt-cache policy linux-meta
linux-meta:
  Installed: 1.16
  Candidate: 1.16
  Package pin: 1.16
  Version table:
     1.17 1001
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages
 *** 1.16 1001
        100 /var/lib/dpkg/status
     1.3 1001
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/backports amd64 Packages

Change 413776 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: apt_pinning: extend linux pinning in jessie

https://gerrit.wikimedia.org/r/413776

Change 413776 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: apt_pinning: extend linux pinning in jessie

https://gerrit.wikimedia.org/r/413776

I don't feel confident with these pam packages being upgraded as well. Let's pin them:

aborrero@tools-proxy-01:~$ sudo apt-upgrade -u report
[...]
tools-proxy-01: oldstable: libpam0g 1.1.8-3.1+deb8u1+b1 --> 1.1.8-3.1+deb8u2+b1 
tools-proxy-01: oldstable: libpam-modules 1.1.8-3.1+deb8u1+b1 --> 1.1.8-3.1+deb8u2+b1 
tools-proxy-01: oldstable: libpam-ldapd 0.9.4-3+deb8u1 --> 0.9.4-3+deb8u2 
tools-proxy-01: oldstable: libpam-modules-bin 1.1.8-3.1+deb8u1+b1 --> 1.1.8-3.1+deb8u2+b1

Change 413780 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: apt_pinning: extend pinnigs for pam libs

https://gerrit.wikimedia.org/r/413780

Change 413780 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: apt_pinning: extend pinnigs for pam libs

https://gerrit.wikimedia.org/r/413780

Servers with no apt pinning right now:

aborrero@tools-clushmaster-01:~$ clush -w @all "[ ! -r /etc/apt/preferences.d/toolforge_linux_pinning.pref ] && echo 'no apt pinning' || true"
tools-clushmaster-01.tools.eqiad.wmflabs: no apt pinning
tools-docker-registry-02.tools.eqiad.wmflabs: no apt pinning
tools-docker-builder-05.tools.eqiad.wmflabs: no apt pinning
tools-elastic-03.tools.eqiad.wmflabs: no apt pinning
tools-elastic-01.tools.eqiad.wmflabs: no apt pinning
tools-docker-registry-01.tools.eqiad.wmflabs: no apt pinning
tools-elastic-02.tools.eqiad.wmflabs: no apt pinning
tools-flannel-etcd-03.tools.eqiad.wmflabs: no apt pinning
tools-logs-02.tools.eqiad.wmflabs: no apt pinning
tools-k8s-etcd-02.tools.eqiad.wmflabs: no apt pinning
tools-k8s-etcd-01.tools.eqiad.wmflabs: no apt pinning
tools-package-builder-01.tools.eqiad.wmflabs: no apt pinning
tools-k8s-etcd-03.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1001.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1003.tools.eqiad.wmflabs: no apt pinning
tools-static-11.tools.eqiad.wmflabs: ssh: Could not resolve hostname tools-static-11.tools.eqiad.wmflabs: Name or service not known
clush: tools-static-11.tools.eqiad.wmflabs: exited with exit code 255
tools-paws-worker-1005.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1002.tools.eqiad.wmflabs: no apt pinning
tools-paws-master-01.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1006.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1007.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1010.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1013.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1016.tools.eqiad.wmflabs: no apt pinning
tools-prometheus-01.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1017.tools.eqiad.wmflabs: no apt pinning
tools-prometheus-02.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1019.tools.eqiad.wmflabs: no apt pinning
tools-redis-1001.tools.eqiad.wmflabs: no apt pinning
tools-redis-1002.tools.eqiad.wmflabs: no apt pinning
tools-static-12.tools.eqiad.wmflabs: no apt pinning
tools-flannel-etcd-01.tools.eqiad.wmflabs: no apt pinning
tools-static-13.tools.eqiad.wmflabs: no apt pinning
tools-flannel-etcd-02.tools.eqiad.wmflabs: no apt pinning

Change 415057 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: introduce base class for all toolforge roles

https://gerrit.wikimedia.org/r/415057

Change 415057 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: introduce base class for all toolforge roles

https://gerrit.wikimedia.org/r/415057

After applying the patch above, this is the status:

aborrero@tools-clushmaster-01:~$ clush -w @all "[ ! -r /etc/apt/preferences.d/toolforge_linux_pinning.pref ] && echo 'no apt pinning' || true"
tools-static-11.tools.eqiad.wmflabs: ssh: Could not resolve hostname tools-static-11.tools.eqiad.wmflabs: Name or service not known
clush: tools-static-11.tools.eqiad.wmflabs: exited with exit code 255
tools-package-builder-01.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1001.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1003.tools.eqiad.wmflabs: no apt pinning
tools-paws-master-01.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1006.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1002.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1005.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1007.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1013.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1016.tools.eqiad.wmflabs: no apt pinning
tools-prometheus-01.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1017.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1010.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1019.tools.eqiad.wmflabs: no apt pinning
tools-prometheus-02.tools.eqiad.wmflabs: no apt pinning

@chasemp has some ideas about addressing them.

Change 415851 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: introduce role::toollabs::base

https://gerrit.wikimedia.org/r/415851

Change 415851 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: introduce role::toollabs::base

https://gerrit.wikimedia.org/r/415851

I applied the new role::toollabs::base to tools-package-builder-01 via horizon and everything seems fine.

aborrero@tools-package-builder-01:~$ sudo puppet agent -t -v
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-package-builder-01.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1519999559'
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]/ensure: defined content as '{md5}f102d5398f8b0a7a487c2373270108a5'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]/ensure: defined content as '{md5}0c54105e63f5f824feb6039d39cef473'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]/ensure: defined content as '{md5}394825857b5a5b4405d4825e5184fcde'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]/ensure: defined content as '{md5}3a070faf67463002c3e503117405666b'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-node-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_node_pinning.pref]/ensure: defined content as '{md5}1b637d9f8c6a46c2756dd9931e3104ee'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-node-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_node_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-master-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_master_pinning.pref]/ensure: defined content as '{md5}3e8f5fd6e543b94231439b5a58c5553b'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-master-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_master_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-client-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_client_pinning.pref]/ensure: defined content as '{md5}13349b03abd355b7f12997223536123e'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-client-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_client_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libnginx-mod-pinning]/File[/etc/apt/preferences.d/toolforge_libnginx_mod_pinning.pref]/ensure: defined content as '{md5}0e95aa2b1c6bf38be142a261da953f3c'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libnginx-mod-pinning]/File[/etc/apt/preferences.d/toolforge_libnginx_mod_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-nginx-pinning]/File[/etc/apt/preferences.d/toolforge_nginx_pinning.pref]/ensure: defined content as '{md5}ef91132a930b665e2e577f4b04ee7e21'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-nginx-pinning]/File[/etc/apt/preferences.d/toolforge_nginx_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Apt/Exec[apt-get update]: Triggered 'refresh' from 9 events
Notice: /Stage[main]/Role::Toollabs::Base/System::Role[toollabs::base]/Motd::Script[role-toollabs::base]/File[/etc/update-motd.d/05-role-toollabs--base]/ensure: defined content as '{md5}243b4d00f7f483d050b02741c9bdbaa1'
Notice: Applied catalog in 15.10 seconds

I guess we could follow the same approach for the remaining nodes.

Change 415885 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: base role: adjust system role string

https://gerrit.wikimedia.org/r/415885

Change 415885 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: base role: adjust system role string

https://gerrit.wikimedia.org/r/415885

Mentioned in SAL (#wikimedia-cloud) [2018-03-05T12:10:47Z] <arturo> apply role::toollabs::base to tools-prometheus prefix in horizon for T187193

No issues in tools-prometheus nodes:

aborrero@tools-prometheus-01:~$ sudo puppet agent -t -v
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-prometheus-01.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1520251886'
Notice: /Stage[main]/Prometheus::Blackbox_exporter/Service[prometheus-blackbox-exporter]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Prometheus::Blackbox_exporter/Service[prometheus-blackbox-exporter]: Unscheduling refresh on Service[prometheus-blackbox-exporter]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]/ensure: defined content as '{md5}f102d5398f8b0a7a487c2373270108a5'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]/ensure: defined content as '{md5}0c54105e63f5f824feb6039d39cef473'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]/ensure: defined content as '{md5}394825857b5a5b4405d4825e5184fcde'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]/ensure: defined content as '{md5}3a070faf67463002c3e503117405666b'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-node-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_node_pinning.pref]/ensure: defined content as '{md5}1b637d9f8c6a46c2756dd9931e3104ee'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-node-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_node_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-master-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_master_pinning.pref]/ensure: defined content as '{md5}3e8f5fd6e543b94231439b5a58c5553b'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-master-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_master_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-client-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_client_pinning.pref]/ensure: defined content as '{md5}13349b03abd355b7f12997223536123e'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-client-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_client_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libnginx-mod-pinning]/File[/etc/apt/preferences.d/toolforge_libnginx_mod_pinning.pref]/ensure: defined content as '{md5}0e95aa2b1c6bf38be142a261da953f3c'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libnginx-mod-pinning]/File[/etc/apt/preferences.d/toolforge_libnginx_mod_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-nginx-pinning]/File[/etc/apt/preferences.d/toolforge_nginx_pinning.pref]/ensure: defined content as '{md5}ef91132a930b665e2e577f4b04ee7e21'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-nginx-pinning]/File[/etc/apt/preferences.d/toolforge_nginx_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Apt/Exec[apt-get update]: Triggered 'refresh' from 9 events
Notice: /Stage[main]/Role::Toollabs::Base/System::Role[toollabs::base]/Motd::Script[role-toollabs::base]/File[/etc/update-motd.d/05-role-toollabs--base]/ensure: defined content as '{md5}c6b451bd56f68eac9957453333dfc4d3'
Notice: Applied catalog in 15.48 seconds
aborrero@tools-prometheus-02:~$ sudo puppet agent -t -v
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-prometheus-02.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1520251901'
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]/ensure: defined content as '{md5}f102d5398f8b0a7a487c2373270108a5'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]/ensure: defined content as '{md5}0c54105e63f5f824feb6039d39cef473'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]/ensure: defined content as '{md5}394825857b5a5b4405d4825e5184fcde'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]/ensure: defined content as '{md5}3a070faf67463002c3e503117405666b'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-node-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_node_pinning.pref]/ensure: defined content as '{md5}1b637d9f8c6a46c2756dd9931e3104ee'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-node-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_node_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-master-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_master_pinning.pref]/ensure: defined content as '{md5}3e8f5fd6e543b94231439b5a58c5553b'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-master-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_master_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-client-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_client_pinning.pref]/ensure: defined content as '{md5}13349b03abd355b7f12997223536123e'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-client-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_client_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libnginx-mod-pinning]/File[/etc/apt/preferences.d/toolforge_libnginx_mod_pinning.pref]/ensure: defined content as '{md5}0e95aa2b1c6bf38be142a261da953f3c'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libnginx-mod-pinning]/File[/etc/apt/preferences.d/toolforge_libnginx_mod_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-nginx-pinning]/File[/etc/apt/preferences.d/toolforge_nginx_pinning.pref]/ensure: defined content as '{md5}ef91132a930b665e2e577f4b04ee7e21'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-nginx-pinning]/File[/etc/apt/preferences.d/toolforge_nginx_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Apt/Exec[apt-get update]: Triggered 'refresh' from 9 events
Notice: /Stage[main]/Role::Toollabs::Base/System::Role[toollabs::base]/Motd::Script[role-toollabs::base]/File[/etc/update-motd.d/05-role-toollabs--base]/ensure: defined content as '{md5}05b3d38101bb575732d57a4198350238'
Notice: Applied catalog in 14.00 seconds

Mentioned in SAL (#wikimedia-cloud) [2018-03-05T12:16:38Z] <arturo> apply role::toollabs::base to tools-paws prefix in horizon for T187193

No issues in tools-paws- nodes:

aborrero@tools-paws-master-01:~$ sudo puppet agent -t -v
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-paws-master-01.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1520252271'
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]/ensure: defined content as '{md5}f18136942d414a775c6e2f25efce946a'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]/ensure: defined content as '{md5}f0fd0f92d6b4aab85121644b21c71302'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]/ensure: defined content as '{md5}2f2a04c0371c230575fc6265976ba105'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]/ensure: defined content as '{md5}3010d078b1ca4b38ee3ce102898c7973'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubeadm-pinning]/File[/etc/apt/preferences.d/toolforge_kubeadm_pinning.pref]/ensure: defined content as '{md5}656ef7f39a466f6948e761a621f115d7'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubeadm-pinning]/File[/etc/apt/preferences.d/toolforge_kubeadm_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubelet-pinning]/File[/etc/apt/preferences.d/toolforge_kubelet_pinning.pref]/ensure: defined content as '{md5}64963ff518ebbd8978c6a5ef93b5ef56'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubelet-pinning]/File[/etc/apt/preferences.d/toolforge_kubelet_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubectl-pinning]/File[/etc/apt/preferences.d/toolforge_kubectl_pinning.pref]/ensure: defined content as '{md5}51dd462fbeeb914641ea92847f67fbe5'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubectl-pinning]/File[/etc/apt/preferences.d/toolforge_kubectl_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-cni-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_cni_pinning.pref]/ensure: defined content as '{md5}50e85e88368a72cd3cb13a004e801ab7'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-cni-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_cni_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Apt/Exec[apt-get update]: Triggered 'refresh' from 8 events
Notice: /Stage[main]/Role::Toollabs::Base/System::Role[toollabs::base]/Motd::Script[role-toollabs::base]/File[/etc/update-motd.d/05-role-toollabs--base]/ensure: defined content as '{md5}ddd5f0b176d3aa732d35fa2be16759cc'
Notice: Applied catalog in 15.08 seconds
aborrero@tools-paws-worker-1019:~$ sudo puppet agent -t -v
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-paws-worker-1019.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1520252368'
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]/ensure: defined content as '{md5}f18136942d414a775c6e2f25efce946a'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]/ensure: defined content as '{md5}f0fd0f92d6b4aab85121644b21c71302'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]/ensure: defined content as '{md5}2f2a04c0371c230575fc6265976ba105'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]/ensure: defined content as '{md5}3010d078b1ca4b38ee3ce102898c7973'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubeadm-pinning]/File[/etc/apt/preferences.d/toolforge_kubeadm_pinning.pref]/ensure: defined content as '{md5}656ef7f39a466f6948e761a621f115d7'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubeadm-pinning]/File[/etc/apt/preferences.d/toolforge_kubeadm_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubelet-pinning]/File[/etc/apt/preferences.d/toolforge_kubelet_pinning.pref]/ensure: defined content as '{md5}64963ff518ebbd8978c6a5ef93b5ef56'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubelet-pinning]/File[/etc/apt/preferences.d/toolforge_kubelet_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubectl-pinning]/File[/etc/apt/preferences.d/toolforge_kubectl_pinning.pref]/ensure: defined content as '{md5}51dd462fbeeb914641ea92847f67fbe5'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubectl-pinning]/File[/etc/apt/preferences.d/toolforge_kubectl_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-cni-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_cni_pinning.pref]/ensure: defined content as '{md5}50e85e88368a72cd3cb13a004e801ab7'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-cni-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_cni_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Apt/Exec[apt-get update]: Triggered 'refresh' from 8 events
Notice: /Stage[main]/Role::Toollabs::Base/System::Role[toollabs::base]/Motd::Script[role-toollabs::base]/File[/etc/update-motd.d/05-role-toollabs--base]/ensure: defined content as '{md5}a891451e1bca8c0ed489766d4053061a'
Notice: Applied catalog in 15.80 seconds

Now all toolforge nodes contains apt-pinning enabled.

aborrero@tools-clushmaster-01:~$ clush -w @all "[ ! -r /etc/apt/preferences.d/toolforge_linux_pinning.pref ] && echo 'no apt pinning' || true"
tools-static-11.tools.eqiad.wmflabs: ssh: Could not resolve hostname tools-static-11.tools.eqiad.wmflabs: Name or service not known
clush: tools-static-11.tools.eqiad.wmflabs: exited with exit code 255

Now it's just a matter of adjusting package versions and adding/delete package pinning as required. Closing task.

We need to pin more nss/ldap/pam packages as seen in T189018

Change 416934 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: apt_pinning: pin more nss/ldap/pam packages

https://gerrit.wikimedia.org/r/416934

Change 416934 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: apt_pinning: pin more nss/ldap/pam packages

https://gerrit.wikimedia.org/r/416934