Page MenuHomePhabricator
Create Task
Maniphest T187193

toolforge: pin key packages by version
Closed, ResolvedPublic
Actions

Description

Let's create apt pinning for some key packages by versions on tooldorge instances.

Benefits:

  • we have well-known kernel version in all nodes
  • nodes don't upgrade kernel packages
  • more 'software-defined' infra

Proposed packages:

  • linux kernel
  • pam libs
  • kubernetes packages
  • nginx

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

aborrero triaged this task as High priority.Feb 13 2018, 2:00 PM
aborrero created this task.
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptFeb 13 2018, 2:00 PM
gerritbot added a comment.Feb 13 2018, 2:16 PM

Change 410177 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] WIP: toollabs: add apt pinnings for key packages

https://gerrit.wikimedia.org/r/410177

gerritbot added a project: Patch-For-Review.Feb 13 2018, 2:16 PM
aborrero renamed this task from toolforge: pin kernels by version to toolforge: pin key packages by version.Feb 13 2018, 5:25 PM
aborrero updated the task description. (Show Details)
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Feb 15 2018, 11:15 AM
• chasemp updated the task description. (Show Details)Feb 20 2018, 7:34 PM
gerritbot added a comment.Feb 21 2018, 4:54 PM

Change 410177 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: add apt pinnings for key packages

https://gerrit.wikimedia.org/r/410177

gerritbot added a comment.Feb 21 2018, 5:12 PM

Change 413202 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: add apt pinnings for key packages

https://gerrit.wikimedia.org/r/413202

gerritbot added a comment.Feb 21 2018, 5:21 PM

Change 413202 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: add apt pinnings for key packages

https://gerrit.wikimedia.org/r/413202

gerritbot added a comment.Feb 21 2018, 5:32 PM

Change 413206 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] toolforge: apply pinning to k8s components

https://gerrit.wikimedia.org/r/413206

gerritbot added a comment.Feb 21 2018, 5:37 PM

Change 413206 merged by Rush:
[operations/puppet@production] toolforge: apply pinning to k8s components

https://gerrit.wikimedia.org/r/413206

gerritbot added a comment.Feb 21 2018, 6:32 PM

Change 413213 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] toolforge: update pin for kubernetes-client

https://gerrit.wikimedia.org/r/413213

gerritbot added a comment.Feb 21 2018, 6:38 PM

Change 413213 merged by Rush:
[operations/puppet@production] toolforge: update pin for kubernetes-client

https://gerrit.wikimedia.org/r/413213

aborrero added a comment.Feb 22 2018, 6:36 PM

There is something weird with trusty linux pinning.

The pinning we have is (file /etc/apt/preferences.d/toolforge_linux_pinning.pref):

Package: linux-image-generic
Pin: version 3.13.0.141.151
Pin-Priority: 1001

But then, if I see what's pending an update, I see:

aborrero@tools-bastion-02:~$ sudo apt-upgrade -u report
tools-bastion-02: trusty-updates: linux-tools-common 3.13.0-141.190 --> 3.13.0-142.191 
tools-bastion-02: trusty-updates: linux-image-virtual 3.13.0.141.151 --> 3.13.0.142.152 
tools-bastion-02: trusty-updates: linux-tools-generic 3.13.0.141.151 --> 3.13.0.142.152 
tools-bastion-02: trusty-updates: linux-tools-3.13.0-142-generic [absent] --> 3.13.0-142.191 
tools-bastion-02: trusty-updates: linux-tools-3.13.0-142 [absent] --> 3.13.0-142.191 
tools-bastion-02: trusty-updates: gcj-4.8-jre-lib 4.8.4-2ubuntu1~14.04.3 --> 4.8.4-2ubuntu1~14.04.4 
tools-bastion-02: trusty-updates: linux-image-3.13.0-142-generic [absent] --> 3.13.0-142.191 
tools-bastion-02: trusty-updates: linux-libc-dev 3.13.0-141.190 --> 3.13.0-142.191

The important packages here are:

  • linux-image-virtual which is virtual package which depends to linux-image-XXXX-generic
  • linux-image-3.13.0-142-generic which is the actual kernel package

This gets complicated because the /boot/directory doesn't show the complete version string of the kernel being run:

aborrero@tools-bastion-02:~$ ll /boot/vmlinuz-*
-rw------- 1 root root 5852400 Dec  4 20:50 /boot/vmlinuz-3.13.0-137-generic
-rw------- 1 root root 5856336 Jan  9 15:29 /boot/vmlinuz-3.13.0-139-generic
-rw------- 1 root root 5856528 Jan 19 14:32 /boot/vmlinuz-3.13.0-141-generic

And we are not even running a kernel from one of the involved packages:

aborrero@tools-bastion-02:~$ uname -vr
3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018

And this seems rather heterogeneous over the fleet:

tools-bastion-02.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-bastion-03.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-bastion-05.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-checker-01.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-checker-02.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-clushmaster-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-cron-01.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-docker-builder-05.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-docker-registry-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-docker-registry-02.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-elastic-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-elastic-02.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-elastic-03.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-exec-1401.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1402.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1403.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1404.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1405.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1406.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1407.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1408.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1409.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1410.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1411.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1412.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1413.tools.eqiad.wmflabs: 3.13.0-141-generic #190-Ubuntu SMP Fri Jan 19 12:52:38 UTC 2018
tools-exec-1414.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1415.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1416.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1417.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1418.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1419.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1420.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1421.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1422.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1423.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1424.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1425.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1426.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1427.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1428.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1429.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1430.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1431.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1432.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1433.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1434.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1435.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1436.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1437.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1438.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1439.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1440.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1441.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-exec-1442.tools.eqiad.wmflabs: 3.13.0-141-generic #190-Ubuntu SMP Fri Jan 19 12:52:38 UTC 2018
tools-exec-gift-trusty-01.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-flannel-etcd-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-flannel-etcd-02.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-flannel-etcd-03.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-grid-master.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-grid-shadow.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-k8s-etcd-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-k8s-etcd-02.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-k8s-etcd-03.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-k8s-master-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-logs-02.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-mail.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-package-builder-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-paws-master-01.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1001.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1002.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1003.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1005.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1006.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1007.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1010.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1013.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1016.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1017.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-paws-worker-1019.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-prometheus-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-prometheus-02.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-proxy-01.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-proxy-02.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-redis-1001.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-redis-1002.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-services-01.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-services-02.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-static-10.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-static-11.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-static-12.tools.eqiad.wmflabs: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)
tools-webgrid-generic-1401.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-generic-1402.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-generic-1403.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-generic-1404.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1401.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1402.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1403.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1404.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1405.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1406.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1407.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1408.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1409.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1410.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1412.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1413.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1414.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1415.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1416.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1417.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1418.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1419.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1420.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1421.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1422.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1424.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1425.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1426.tools.eqiad.wmflabs: 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018
tools-webgrid-lighttpd-1427.tools.eqiad.wmflabs: 3.13.0-141-generic #190-Ubuntu SMP Fri Jan 19 12:52:38 UTC 2018
tools-webgrid-lighttpd-1428.tools.eqiad.wmflabs: 3.13.0-141-generic #190-Ubuntu SMP Fri Jan 19 12:52:38 UTC 2018
tools-worker-1001.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1002.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1003.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1004.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1005.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1006.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1007.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1008.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1009.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1010.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1011.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1012.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1013.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1014.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1015.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1016.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1017.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1018.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1019.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1020.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1021.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1022.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1023.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1025.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1026.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)
tools-worker-1027.tools.eqiad.wmflabs: 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04)

Our current pinning only affects linux-image-generic (v3.13.0.141.151) which is virtual and depends on: linux-image-3.13.0-141-generic.
So, installing both kernels 3.13.0.141.151and 3.13.0.142.152 isn't a policy violation and the resolves just tries it.

Proposed solution (for ubuntu trusty) is to either:

  • extend the pinning to linux-image-virtual as well
  • ban linux-image-virtual from being installed at all, since we are installing linux-image-generic for our running kernel

For jessie and stretch, another similar analysis should be conducted.

gerritbot added a comment.Feb 23 2018, 1:42 PM

Change 413725 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: apt_pinning: be more strict in linux kernel pinning

https://gerrit.wikimedia.org/r/413725

gerritbot added a comment.Feb 23 2018, 1:44 PM

Change 413725 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: apt_pinning: be more strict in linux kernel pinning

https://gerrit.wikimedia.org/r/413725

MoritzMuehlenhoff subscribed.Feb 23 2018, 1:46 PM

For some reasons Ubuntu makes every kernel package co-installable (in Debian the binary package is only changed if the kernel ABI changes). So you'll probably need to manually update the latest trusty kernel you want to install and pin to that specific binary package name.

aborrero added a comment.Feb 23 2018, 7:06 PM

More weird stuff.

aborrero@tools-proxy-01:~$ sudo apt-upgrade -u report
tools-proxy-01: jessie-backports: linux-image-4.9.0-0.bpo.4-amd64 4.9.51-1~bpo8+1 --> 4.9.65-3+deb9u1~bpo8+1 
tools-proxy-01: jessie-wikimedia: linux-meta-4.9 1.16 --> 1.17 
tools-proxy-01: jessie-wikimedia: linux-image-4.9.0-0.bpo.6-amd64 [absent] --> 4.9.82-1~wmf1 
tools-proxy-01: now: linux-meta 1.16 --> [remove] 
tools-proxy-01: oldstable: libpam0g 1.1.8-3.1+deb8u1+b1 --> 1.1.8-3.1+deb8u2+b1 
tools-proxy-01: oldstable: libpam-modules 1.1.8-3.1+deb8u1+b1 --> 1.1.8-3.1+deb8u2+b1 
tools-proxy-01: oldstable: libpam-ldapd 0.9.4-3+deb8u1 --> 0.9.4-3+deb8u2 
tools-proxy-01: oldstable: libpam-modules-bin 1.1.8-3.1+deb8u1+b1 --> 1.1.8-3.1+deb8u2+b1 
tools-proxy-01: oldstable-updates: linux-image-3.16.0-4-amd64 3.16.43-2+deb8u5 --> 3.16.51-3
  • there is a now archive which is a fake one, meaning the package candidate is not being pulled from any archive (is being removed). That is probably something to address in the apt-upgrade code, i.e. print ([none]) or something like that
  • the linux-meta-4.9 package is being upgraded. I would expect it to be following the Depends of linux-meta, but that's not true. A patch will follow-up to address it.
  • the linux-meta package is being removed. I couldn't figure out why, since we have a proper pinning for it. This is even more weird if we inspect the policy for it (see below).
aborrero@tools-proxy-01:~$ apt-cache policy linux-meta
linux-meta:
  Installed: 1.16
  Candidate: 1.16
  Package pin: 1.16
  Version table:
     1.17 1001
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages
 *** 1.16 1001
        100 /var/lib/dpkg/status
     1.3 1001
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/backports amd64 Packages
gerritbot added a comment.Feb 23 2018, 7:08 PM

Change 413776 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: apt_pinning: extend linux pinning in jessie

https://gerrit.wikimedia.org/r/413776

gerritbot added a comment.Feb 23 2018, 7:10 PM

Change 413776 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: apt_pinning: extend linux pinning in jessie

https://gerrit.wikimedia.org/r/413776

aborrero added a comment.Feb 23 2018, 7:14 PM

I don't feel confident with these pam packages being upgraded as well. Let's pin them:

aborrero@tools-proxy-01:~$ sudo apt-upgrade -u report
[...]
tools-proxy-01: oldstable: libpam0g 1.1.8-3.1+deb8u1+b1 --> 1.1.8-3.1+deb8u2+b1 
tools-proxy-01: oldstable: libpam-modules 1.1.8-3.1+deb8u1+b1 --> 1.1.8-3.1+deb8u2+b1 
tools-proxy-01: oldstable: libpam-ldapd 0.9.4-3+deb8u1 --> 0.9.4-3+deb8u2 
tools-proxy-01: oldstable: libpam-modules-bin 1.1.8-3.1+deb8u1+b1 --> 1.1.8-3.1+deb8u2+b1
gerritbot added a comment.Feb 23 2018, 7:22 PM

Change 413780 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: apt_pinning: extend pinnigs for pam libs

https://gerrit.wikimedia.org/r/413780

gerritbot added a comment.Feb 26 2018, 10:36 AM

Change 413780 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: apt_pinning: extend pinnigs for pam libs

https://gerrit.wikimedia.org/r/413780

aborrero added a comment.Feb 27 2018, 2:02 PM

Servers with no apt pinning right now:

aborrero@tools-clushmaster-01:~$ clush -w @all "[ ! -r /etc/apt/preferences.d/toolforge_linux_pinning.pref ] && echo 'no apt pinning' || true"
tools-clushmaster-01.tools.eqiad.wmflabs: no apt pinning
tools-docker-registry-02.tools.eqiad.wmflabs: no apt pinning
tools-docker-builder-05.tools.eqiad.wmflabs: no apt pinning
tools-elastic-03.tools.eqiad.wmflabs: no apt pinning
tools-elastic-01.tools.eqiad.wmflabs: no apt pinning
tools-docker-registry-01.tools.eqiad.wmflabs: no apt pinning
tools-elastic-02.tools.eqiad.wmflabs: no apt pinning
tools-flannel-etcd-03.tools.eqiad.wmflabs: no apt pinning
tools-logs-02.tools.eqiad.wmflabs: no apt pinning
tools-k8s-etcd-02.tools.eqiad.wmflabs: no apt pinning
tools-k8s-etcd-01.tools.eqiad.wmflabs: no apt pinning
tools-package-builder-01.tools.eqiad.wmflabs: no apt pinning
tools-k8s-etcd-03.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1001.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1003.tools.eqiad.wmflabs: no apt pinning
tools-static-11.tools.eqiad.wmflabs: ssh: Could not resolve hostname tools-static-11.tools.eqiad.wmflabs: Name or service not known
clush: tools-static-11.tools.eqiad.wmflabs: exited with exit code 255
tools-paws-worker-1005.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1002.tools.eqiad.wmflabs: no apt pinning
tools-paws-master-01.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1006.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1007.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1010.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1013.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1016.tools.eqiad.wmflabs: no apt pinning
tools-prometheus-01.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1017.tools.eqiad.wmflabs: no apt pinning
tools-prometheus-02.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1019.tools.eqiad.wmflabs: no apt pinning
tools-redis-1001.tools.eqiad.wmflabs: no apt pinning
tools-redis-1002.tools.eqiad.wmflabs: no apt pinning
tools-static-12.tools.eqiad.wmflabs: no apt pinning
tools-flannel-etcd-01.tools.eqiad.wmflabs: no apt pinning
tools-static-13.tools.eqiad.wmflabs: no apt pinning
tools-flannel-etcd-02.tools.eqiad.wmflabs: no apt pinning
gerritbot added a comment.Feb 27 2018, 5:53 PM

Change 415057 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: introduce base class for all toolforge roles

https://gerrit.wikimedia.org/r/415057

gerritbot added a comment.Mar 1 2018, 1:18 PM

Change 415057 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: introduce base class for all toolforge roles

https://gerrit.wikimedia.org/r/415057

aborrero added a comment.Mar 1 2018, 1:55 PM

After applying the patch above, this is the status:

aborrero@tools-clushmaster-01:~$ clush -w @all "[ ! -r /etc/apt/preferences.d/toolforge_linux_pinning.pref ] && echo 'no apt pinning' || true"
tools-static-11.tools.eqiad.wmflabs: ssh: Could not resolve hostname tools-static-11.tools.eqiad.wmflabs: Name or service not known
clush: tools-static-11.tools.eqiad.wmflabs: exited with exit code 255
tools-package-builder-01.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1001.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1003.tools.eqiad.wmflabs: no apt pinning
tools-paws-master-01.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1006.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1002.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1005.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1007.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1013.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1016.tools.eqiad.wmflabs: no apt pinning
tools-prometheus-01.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1017.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1010.tools.eqiad.wmflabs: no apt pinning
tools-paws-worker-1019.tools.eqiad.wmflabs: no apt pinning
tools-prometheus-02.tools.eqiad.wmflabs: no apt pinning

@chasemp has some ideas about addressing them.

gerritbot added a comment.Mar 2 2018, 1:46 PM

Change 415851 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: introduce role::toollabs::base

https://gerrit.wikimedia.org/r/415851

gerritbot added a comment.Mar 2 2018, 1:57 PM

Change 415851 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: introduce role::toollabs::base

https://gerrit.wikimedia.org/r/415851

aborrero added a comment.Mar 2 2018, 2:22 PM

I applied the new role::toollabs::base to tools-package-builder-01 via horizon and everything seems fine.

aborrero@tools-package-builder-01:~$ sudo puppet agent -t -v
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-package-builder-01.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1519999559'
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]/ensure: defined content as '{md5}f102d5398f8b0a7a487c2373270108a5'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]/ensure: defined content as '{md5}0c54105e63f5f824feb6039d39cef473'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]/ensure: defined content as '{md5}394825857b5a5b4405d4825e5184fcde'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]/ensure: defined content as '{md5}3a070faf67463002c3e503117405666b'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-node-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_node_pinning.pref]/ensure: defined content as '{md5}1b637d9f8c6a46c2756dd9931e3104ee'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-node-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_node_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-master-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_master_pinning.pref]/ensure: defined content as '{md5}3e8f5fd6e543b94231439b5a58c5553b'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-master-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_master_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-client-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_client_pinning.pref]/ensure: defined content as '{md5}13349b03abd355b7f12997223536123e'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-client-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_client_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libnginx-mod-pinning]/File[/etc/apt/preferences.d/toolforge_libnginx_mod_pinning.pref]/ensure: defined content as '{md5}0e95aa2b1c6bf38be142a261da953f3c'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libnginx-mod-pinning]/File[/etc/apt/preferences.d/toolforge_libnginx_mod_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-nginx-pinning]/File[/etc/apt/preferences.d/toolforge_nginx_pinning.pref]/ensure: defined content as '{md5}ef91132a930b665e2e577f4b04ee7e21'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-nginx-pinning]/File[/etc/apt/preferences.d/toolforge_nginx_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Apt/Exec[apt-get update]: Triggered 'refresh' from 9 events
Notice: /Stage[main]/Role::Toollabs::Base/System::Role[toollabs::base]/Motd::Script[role-toollabs::base]/File[/etc/update-motd.d/05-role-toollabs--base]/ensure: defined content as '{md5}243b4d00f7f483d050b02741c9bdbaa1'
Notice: Applied catalog in 15.10 seconds

I guess we could follow the same approach for the remaining nodes.

gerritbot added a comment.Mar 2 2018, 4:39 PM

Change 415885 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: base role: adjust system role string

https://gerrit.wikimedia.org/r/415885

gerritbot added a comment.Mar 2 2018, 4:46 PM

Change 415885 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: base role: adjust system role string

https://gerrit.wikimedia.org/r/415885

Stashbot added a comment.Mar 5 2018, 12:10 PM

Mentioned in SAL (#wikimedia-cloud) [2018-03-05T12:10:47Z] <arturo> apply role::toollabs::base to tools-prometheus prefix in horizon for T187193

aborrero added a comment.Mar 5 2018, 12:14 PM

No issues in tools-prometheus nodes:

aborrero@tools-prometheus-01:~$ sudo puppet agent -t -v
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-prometheus-01.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1520251886'
Notice: /Stage[main]/Prometheus::Blackbox_exporter/Service[prometheus-blackbox-exporter]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Prometheus::Blackbox_exporter/Service[prometheus-blackbox-exporter]: Unscheduling refresh on Service[prometheus-blackbox-exporter]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]/ensure: defined content as '{md5}f102d5398f8b0a7a487c2373270108a5'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]/ensure: defined content as '{md5}0c54105e63f5f824feb6039d39cef473'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]/ensure: defined content as '{md5}394825857b5a5b4405d4825e5184fcde'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]/ensure: defined content as '{md5}3a070faf67463002c3e503117405666b'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-node-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_node_pinning.pref]/ensure: defined content as '{md5}1b637d9f8c6a46c2756dd9931e3104ee'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-node-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_node_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-master-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_master_pinning.pref]/ensure: defined content as '{md5}3e8f5fd6e543b94231439b5a58c5553b'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-master-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_master_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-client-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_client_pinning.pref]/ensure: defined content as '{md5}13349b03abd355b7f12997223536123e'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-client-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_client_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libnginx-mod-pinning]/File[/etc/apt/preferences.d/toolforge_libnginx_mod_pinning.pref]/ensure: defined content as '{md5}0e95aa2b1c6bf38be142a261da953f3c'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libnginx-mod-pinning]/File[/etc/apt/preferences.d/toolforge_libnginx_mod_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-nginx-pinning]/File[/etc/apt/preferences.d/toolforge_nginx_pinning.pref]/ensure: defined content as '{md5}ef91132a930b665e2e577f4b04ee7e21'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-nginx-pinning]/File[/etc/apt/preferences.d/toolforge_nginx_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Apt/Exec[apt-get update]: Triggered 'refresh' from 9 events
Notice: /Stage[main]/Role::Toollabs::Base/System::Role[toollabs::base]/Motd::Script[role-toollabs::base]/File[/etc/update-motd.d/05-role-toollabs--base]/ensure: defined content as '{md5}c6b451bd56f68eac9957453333dfc4d3'
Notice: Applied catalog in 15.48 seconds
aborrero@tools-prometheus-02:~$ sudo puppet agent -t -v
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-prometheus-02.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1520251901'
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]/ensure: defined content as '{md5}f102d5398f8b0a7a487c2373270108a5'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]/ensure: defined content as '{md5}0c54105e63f5f824feb6039d39cef473'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]/ensure: defined content as '{md5}394825857b5a5b4405d4825e5184fcde'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]/ensure: defined content as '{md5}3a070faf67463002c3e503117405666b'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-node-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_node_pinning.pref]/ensure: defined content as '{md5}1b637d9f8c6a46c2756dd9931e3104ee'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-node-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_node_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-master-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_master_pinning.pref]/ensure: defined content as '{md5}3e8f5fd6e543b94231439b5a58c5553b'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-master-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_master_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-client-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_client_pinning.pref]/ensure: defined content as '{md5}13349b03abd355b7f12997223536123e'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-client-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_client_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libnginx-mod-pinning]/File[/etc/apt/preferences.d/toolforge_libnginx_mod_pinning.pref]/ensure: defined content as '{md5}0e95aa2b1c6bf38be142a261da953f3c'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libnginx-mod-pinning]/File[/etc/apt/preferences.d/toolforge_libnginx_mod_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-nginx-pinning]/File[/etc/apt/preferences.d/toolforge_nginx_pinning.pref]/ensure: defined content as '{md5}ef91132a930b665e2e577f4b04ee7e21'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-nginx-pinning]/File[/etc/apt/preferences.d/toolforge_nginx_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Apt/Exec[apt-get update]: Triggered 'refresh' from 9 events
Notice: /Stage[main]/Role::Toollabs::Base/System::Role[toollabs::base]/Motd::Script[role-toollabs::base]/File[/etc/update-motd.d/05-role-toollabs--base]/ensure: defined content as '{md5}05b3d38101bb575732d57a4198350238'
Notice: Applied catalog in 14.00 seconds
Stashbot added a comment.Mar 5 2018, 12:16 PM

Mentioned in SAL (#wikimedia-cloud) [2018-03-05T12:16:38Z] <arturo> apply role::toollabs::base to tools-paws prefix in horizon for T187193

aborrero added a comment.Mar 5 2018, 12:21 PM

No issues in tools-paws- nodes:

aborrero@tools-paws-master-01:~$ sudo puppet agent -t -v
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-paws-master-01.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1520252271'
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]/ensure: defined content as '{md5}f18136942d414a775c6e2f25efce946a'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]/ensure: defined content as '{md5}f0fd0f92d6b4aab85121644b21c71302'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]/ensure: defined content as '{md5}2f2a04c0371c230575fc6265976ba105'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]/ensure: defined content as '{md5}3010d078b1ca4b38ee3ce102898c7973'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubeadm-pinning]/File[/etc/apt/preferences.d/toolforge_kubeadm_pinning.pref]/ensure: defined content as '{md5}656ef7f39a466f6948e761a621f115d7'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubeadm-pinning]/File[/etc/apt/preferences.d/toolforge_kubeadm_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubelet-pinning]/File[/etc/apt/preferences.d/toolforge_kubelet_pinning.pref]/ensure: defined content as '{md5}64963ff518ebbd8978c6a5ef93b5ef56'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubelet-pinning]/File[/etc/apt/preferences.d/toolforge_kubelet_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubectl-pinning]/File[/etc/apt/preferences.d/toolforge_kubectl_pinning.pref]/ensure: defined content as '{md5}51dd462fbeeb914641ea92847f67fbe5'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubectl-pinning]/File[/etc/apt/preferences.d/toolforge_kubectl_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-cni-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_cni_pinning.pref]/ensure: defined content as '{md5}50e85e88368a72cd3cb13a004e801ab7'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-cni-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_cni_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Apt/Exec[apt-get update]: Triggered 'refresh' from 8 events
Notice: /Stage[main]/Role::Toollabs::Base/System::Role[toollabs::base]/Motd::Script[role-toollabs::base]/File[/etc/update-motd.d/05-role-toollabs--base]/ensure: defined content as '{md5}ddd5f0b176d3aa732d35fa2be16759cc'
Notice: Applied catalog in 15.08 seconds
aborrero@tools-paws-worker-1019:~$ sudo puppet agent -t -v
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for tools-paws-worker-1019.tools.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1520252368'
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]/ensure: defined content as '{md5}f18136942d414a775c6e2f25efce946a'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-meta-pinning]/File[/etc/apt/preferences.d/toolforge_linux_meta_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]/ensure: defined content as '{md5}f0fd0f92d6b4aab85121644b21c71302'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-linux-pinning]/File[/etc/apt/preferences.d/toolforge_linux_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]/ensure: defined content as '{md5}2f2a04c0371c230575fc6265976ba105'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]/ensure: defined content as '{md5}3010d078b1ca4b38ee3ce102898c7973'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-libpam-ldapd-pinning]/File[/etc/apt/preferences.d/toolforge_libpam_ldapd_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubeadm-pinning]/File[/etc/apt/preferences.d/toolforge_kubeadm_pinning.pref]/ensure: defined content as '{md5}656ef7f39a466f6948e761a621f115d7'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubeadm-pinning]/File[/etc/apt/preferences.d/toolforge_kubeadm_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubelet-pinning]/File[/etc/apt/preferences.d/toolforge_kubelet_pinning.pref]/ensure: defined content as '{md5}64963ff518ebbd8978c6a5ef93b5ef56'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubelet-pinning]/File[/etc/apt/preferences.d/toolforge_kubelet_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubectl-pinning]/File[/etc/apt/preferences.d/toolforge_kubectl_pinning.pref]/ensure: defined content as '{md5}51dd462fbeeb914641ea92847f67fbe5'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubectl-pinning]/File[/etc/apt/preferences.d/toolforge_kubectl_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-cni-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_cni_pinning.pref]/ensure: defined content as '{md5}50e85e88368a72cd3cb13a004e801ab7'
Info: /Stage[main]/Toollabs::Apt_pinning/Apt::Pin[toolforge-kubernetes-cni-pinning]/File[/etc/apt/preferences.d/toolforge_kubernetes_cni_pinning.pref]: Scheduling refresh of Exec[apt-get update]
Notice: /Stage[main]/Apt/Exec[apt-get update]: Triggered 'refresh' from 8 events
Notice: /Stage[main]/Role::Toollabs::Base/System::Role[toollabs::base]/Motd::Script[role-toollabs::base]/File[/etc/update-motd.d/05-role-toollabs--base]/ensure: defined content as '{md5}a891451e1bca8c0ed489766d4053061a'
Notice: Applied catalog in 15.80 seconds
aborrero mentioned this in T188911: toolforge: cleanup kernel packages.Mar 5 2018, 1:15 PM
aborrero closed this task as Resolved.Mar 5 2018, 1:18 PM

Now all toolforge nodes contains apt-pinning enabled.

aborrero@tools-clushmaster-01:~$ clush -w @all "[ ! -r /etc/apt/preferences.d/toolforge_linux_pinning.pref ] && echo 'no apt pinning' || true"
tools-static-11.tools.eqiad.wmflabs: ssh: Could not resolve hostname tools-static-11.tools.eqiad.wmflabs: Name or service not known
clush: tools-static-11.tools.eqiad.wmflabs: exited with exit code 255

Now it's just a matter of adjusting package versions and adding/delete package pinning as required. Closing task.

aborrero reopened this task as Open.Mar 7 2018, 12:33 PM

We need to pin more nss/ldap/pam packages as seen in T189018

gerritbot added a comment.Mar 7 2018, 1:09 PM

Change 416934 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: apt_pinning: pin more nss/ldap/pam packages

https://gerrit.wikimedia.org/r/416934

gerritbot added a comment.Mar 7 2018, 1:38 PM

Change 416934 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: apt_pinning: pin more nss/ldap/pam packages

https://gerrit.wikimedia.org/r/416934

aborrero closed this task as Resolved.Mar 8 2018, 10:55 AM
bd808 moved this task from Doing to Done on the cloud-services-team (Kanban) board.Mar 19 2018, 9:46 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits