Page MenuHomePhabricator

Audit /etc/apt directories
Closed, ResolvedPublic

Description

create a script which can audit the /etc/apt/preference.d and /etc/apt/source.d folder to ensure only managed resources exist. Once complete we should be able to add

resource {'apt::pin':
  purge => true
}
resource {'apt::source':
  purge => true
}

or more likley

class {'apt': purge_sources =>true}

we should be able to do this comparing /var/lib/puppet/state/last_run_report.yaml with the relevent folders

Event Timeline

Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptJan 24 2019, 5:27 PM

Hi @jbond. Could you please associate at least one project with this task so others can find this task when searching in the corresponding project(s)? Thanks.

jbond triaged this task as Low priority.Feb 7 2019, 2:12 PM

Change 559812 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] package_builder: manage source repositories using apt::repository

https://gerrit.wikimedia.org/r/559812

I have generated a list of managed sources files with the following snippet

from pypuppetdb import connect
from pypuppetdb.QueryBuilder import RegexOperator
from os.path import basename

found_files = set()
db = connect()
query_regex = RegexOperator('title', '^\/etc\/apt\/sources.list.d\/')
resources = db.resources('file', query=query_regex)
for resource in resources:
  found_files.add(basename(resource.name))
  print('|'.join(found_files))

The on cumin ran the following to find any unmanaged files

$ sudo cumin 'A:all' 'ls -1 /etc/apt/sources.list.d | grep -Ev "getenvoy-jessie.list|component-puppet5.list|thirdparty-ceph-nautilus-buster.list|wikimedia-spicerack.list|thirdparty-cloudera.list|wikimedia.list|thirdparty-k8s.list|component-facter3.list|component-pyall.list|wikimedia-cassandra311.list|ffmpeg-vp9.list|wikimedia-curator.list|thirdparty-grafana.list|thirdparty-ci.list|openstack-ocata-stretch.list|wikimedia-cassandra22.list|wikimedia-php72.list|wikimedia-thumbor.list|wikimedia-openjdk8.list|wikimedia-elastic.list|openstack-mitaka-jessie.list|component-ci.list|thirdparty-confluent.list|wikimedia-zookeeper349.list|amd-rocm271.list|stretch-node10.list|thirdparty-tor.list|wikimedia-node10.list|openjdk-8.list|debian-backports.list|buster-cergen.list|package-build-deb-src.list|stretch-wikimedia-thirdparty-oath.list|jenkins-thirdparty-ci.list" || true'

This resulted in no output

Change 559812 merged by Jbond:
[operations/puppet@production] package_builder: manage source repositories using apt::repository

https://gerrit.wikimedia.org/r/559812

I have generated a list of managed sources files with the following snippet

from pypuppetdb import connect
from pypuppetdb.QueryBuilder import RegexOperator
from os.path import basename

found_files = set()
db = connect()
query_regex = RegexOperator('title', '^\/etc\/apt\/sources.list.d\/')
resources = db.resources('file', query=query_regex)
for resource in resources:
  found_files.add(basename(resource.name))
  print('|'.join(found_files))

The on cumin ran the following to find any unmanaged files

$ sudo cumin 'A:all' 'ls -1 /etc/apt/sources.list.d | grep -Ev "getenvoy-jessie.list|component-puppet5.list|thirdparty-ceph-nautilus-buster.list|wikimedia-spicerack.list|thirdparty-cloudera.list|wikimedia.list|thirdparty-k8s.list|component-facter3.list|component-pyall.list|wikimedia-cassandra311.list|ffmpeg-vp9.list|wikimedia-curator.list|thirdparty-grafana.list|thirdparty-ci.list|openstack-ocata-stretch.list|wikimedia-cassandra22.list|wikimedia-php72.list|wikimedia-thumbor.list|wikimedia-openjdk8.list|wikimedia-elastic.list|openstack-mitaka-jessie.list|component-ci.list|thirdparty-confluent.list|wikimedia-zookeeper349.list|amd-rocm271.list|stretch-node10.list|thirdparty-tor.list|wikimedia-node10.list|openjdk-8.list|debian-backports.list|buster-cergen.list|package-build-deb-src.list|stretch-wikimedia-thirdparty-oath.list|jenkins-thirdparty-ci.list" || true'

This resulted in no output

which all makes senses because off

../puppet/hieradata/common/profile/base.yaml:profile::base::purge_apt_sources: true

Macro facepalm:

remove the followinf old files

  • /etc/apt/preferences.d/puppet_all.pref
  • /etc/apt/preferences.d/facter.pref
  • /etc/apt/preferences.d/smartmontools.pref

The following unmanaged preferences exist

labstore[1004-1005].eqiad.wmnet                                                                               
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
jessie_mitaka_pinning_nova_compute_kvm.pref                                                                       
jessie_mitaka_pinning_nova_compute.pref                                                                           
nethogs.pref
python3_ldap3.pref
python3-ldap3.pref
python3_pyasn1.pref
python3-pyasn1.pref
===== NODE GROUP =====                                                                                            
(31) cloudcontrol[1003-1004].wikimedia.org,cloudnet[1003-1004].eqiad.wmnet,cloudservices[1003-1004].wikimedia.org,cloudvirt[1001-1009,1012-1014,1016-1023,1025-1029].eqiad.wmnet                                                    
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
mitaka_stretch_nojessiebpo_clientpackages.pref                                                                    
mitaka_stretch_nojessiebpo.pref                                                                                   
===== NODE GROUP =====                                                                                            
(1) cloudvirt1030.eqiad.wmnet                                                                                     
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
mitaka_stretch_nojessiebpo_clientpackages.pref                                                                    
mitaka_stretch_nojessiebpo.pref                                                                                   
t2.pref
t.pref
===== NODE GROUP =====                                                                                            
(1) elnath.codfw.wmnet                                                                                            
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
ruby-json.pref                                                                                                    
===== NODE GROUP =====                                                                                            
(2) install[1002,2002].wikimedia.org                                                                              
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
reprepro.pref                                                                                                     
===== NODE GROUP =====                                                                                            
(10) scb[2001-2006].codfw.wmnet,scb[1001-1004].eqiad.wmnet                                                        
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
python3_tornado.pref                                                                                              
===== NODE GROUP =====                                                                                            
(4) cloudservices2002-dev.wikimedia.org,cloudvirt[2001-2003]-dev.codfw.wmnet                                      
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
mitaka_stretch_nojessiebpo.pref                                                                                   
===== NODE GROUP =====                                                                                            
(6) logstash[2004-2006].codfw.wmnet,logstash[1007-1009].eqiad.wmnet                                               
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
elasticsearch_curator.pref                                                                                        
===== NODE GROUP =====                                                                                            
(1) fermium.wikimedia.org                                                                                         
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
libapache2_mod_security2.pref                                                                                     
===== NODE GROUP =====                                                                                            
(10) kubernetes[2001-2004].codfw.wmnet,kubernetes[1001-1004].eqiad.wmnet,kubestage[1001-1002].eqiad.wmnet         
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
go.pref                                                                                                           
===== NODE GROUP =====                                                                                            
(2) contint[1001,2001].wikimedia.org                                                                              
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
git_from_backports.pref

Change 561876 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] apt::package_from_component: Don't ping if already covered

https://gerrit.wikimedia.org/r/561876

  • remove /etc/apt/preferences.d/reprepro.pref from install servers as package is now in jessie-wikimedia/backports
  • remove /etc/apt/preferences.d/go.pref on kubeernetes servers as it refered to jessie and theses machines are stretch
  • remove /etc/apt/preferences.d/git_from_backports.pref from contint as git provided by component/ci
  • remove /etc/apt/preferences.d/libapache2_mod_security2.pref from fermium as package is now in jessie-wikimedia/backports
  • remove as package is now provided by stretch-wikimedia/thirdparty/elasticsearch-curator5 and ping is maintained via generic pin[1]
  • sudo cumin 'cloudvirt[2001-2003]-dev.codfw.wmnet,cloudservices2002-dev.wikimedia.org' 'rm /etc/apt/preferences.d/mitaka_stretch_nojessiebpo.pref' as systems are no longer jessie
  • sudo cumin 'scb[2001-2006].codfw.wmnet,scb[1001-1004].eqiad.wmnet' 'rm -f /etc/apt/preferences.d/python3_tornado.pref' as now provided by jessie-wikimedia/backports
  • remove /etc/apt/preferences.d/ruby-json.pref from elnath as system is spare

[1] https://github.com/wikimedia/puppet/blob/production/modules/apt/manifests/init.pp#L35-L40

remaining

(2) labstore[1004-1005].eqiad.wmnet                                                                               
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
jessie_mitaka_pinning_nova_compute_kvm.pref                                                                       
jessie_mitaka_pinning_nova_compute.pref                                                                           
nethogs.pref
python3_ldap3.pref
python3-ldap3.pref
python3_pyasn1.pref
python3-pyasn1.pref
===== NODE GROUP =====                                                                                            
(31) cloudcontrol[1003-1004].wikimedia.org,cloudnet[1003-1004].eqiad.wmnet,cloudservices[1003-1004].wikimedia.org,cloudvirt[1001-1009,1012-1014,1016-1023,1025-1029].eqiad.wmnet                                                    
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
mitaka_stretch_nojessiebpo_clientpackages.pref                                                                    
mitaka_stretch_nojessiebpo.pref                                                                                   
===== NODE GROUP =====                                                                                            
(1) cloudvirt1030.eqiad.wmnet                                                                                     
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
mitaka_stretch_nojessiebpo_clientpackages.pref                                                                    
mitaka_stretch_nojessiebpo.pref                                                                                   
t2.pref
t.pref

Change 561876 merged by Jbond:
[operations/puppet@production] apt::package_from_component: Don't ping if already covered

https://gerrit.wikimedia.org/r/561876

  • remove python3{-,_}{ldap3,pyasn}.pref and nethogs.pref as package available in jessie-wikimedia/openstack-mitaka-jessie

remaining unmanaged files under /etc/apt/prefrences.d/

===== NODE GROUP =====                                                                                            
(31) cloudcontrol[1003-1004].wikimedia.org,cloudnet[1003-1004].eqiad.wmnet,cloudservices[1003-1004].wikimedia.org,cloudvirt[1001-1009,1012-1014,1016-1023,1025-1029].eqiad.wmnet                                                    
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
mitaka_stretch_nojessiebpo_clientpackages.pref                                                                    
mitaka_stretch_nojessiebpo.pref                                                                                   
===== NODE GROUP =====                                                                                            
(1) cloudvirt1030.eqiad.wmnet                                                                                     
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
mitaka_stretch_nojessiebpo_clientpackages.pref                                                                    
mitaka_stretch_nojessiebpo.pref                                                                                   
t2.pref
t.pref
===== NODE GROUP =====                                                                                            
(2) labstore[1004-1005].eqiad.wmnet                                                                               
----- OUTPUT of 'ls -1 /etc/apt/p...|wikimedia.pref"' -----                                                       
jessie_mitaka_pinning_nova_compute_kvm.pref                                                                       
jessie_mitaka_pinning_nova_compute.pref
  • removed /etc/apt/preferences.d/t.pref and /etc/apt/preferences.d/t2.pref as pined version is no longer available
  • remove mitaka_stretch_nojessiebpo_clientpackages.pref and mitaka_stretch_nojessiebpo_clientpackages.pref as component openstack-mitaka-jessie no longer avalible and systems are jessie
  • remove jessie_mitaka_pinning_nova_compute_kvm.pref and jessie_mitaka_pinning_nova_compute.pref package is not installed but version specified is allready most prefered via jessie-wikimedia/openstack-mitaka-jessie

only managed files now exist in /etc/apt/prefrences.d

 sudo cumin 'A:all' 'ls -1 /etc/apt/preferences.d/ | egrep -v "jessie_mitaka_pinning_python_keystoneclient.pref|jessie_mitaka_pinning_python3_novaclient.pref|jessie_mitaka_pinning_python_keystoneauth1.pref|strech_wikimedia_docker_registry_27.pref|jessie_mitaka_pinning_python3_oslo_serialization.pref|jessie_mitaka_pinning_python_novaclient.pref|jessie_mitaka_pinning_python_openstacksdk.pref|wikimedia_thumbor.pref|jessie_mitaka_pinning_python_designateclient.pref|jessie_mitaka_pinning_python_cinderclient.pref|jessie_mitaka_pinning_python_oslo_config.pref|librdkafka1.pref|jessie_mitaka_pinning_python3_oslo_i18n.pref|php_wikidiff2.pref|jessie_mitaka_pinning_python_glanceclient.pref|jessie_mitaka_pinning_python3_oslo_config.pref|jessie_mitaka_pinning_python3_keystoneauth1.pref|libhwloc5.pref|jessie_mitaka_pinning_python3_glanceclient.pref|node10_stretch_wikimedia.pref|jessie_mitaka_pinning_python_oslo_serialization.pref|lintian.pref|jessie_mitaka_pinning_nova_common.pref|jessie_mitaka_pinning_python_openstackclient.pref|osm2pgsql.pref|git_lfs.pref|npm_stretch_backports.pref|thirdparty_cloudera.pref|systemd_239_slice_all_users.pref|facter3_stretch.pref|jessie_mitaka_pinning_python_os_client_config.pref|jessie_mitaka_pinning_python3_keystoneclient.pref|ffmpeg_vp9_stretch.pref|puppet5_stretch.pref|jessie_mitaka_pinning_python_oslo_i18n.pref|jessie_mitaka_pinning_python_oslo_utils.pref|mtail.pref|jessie_mitaka_pinning_python3_oslo_utils.pref|wikimedia.pref"'
1391 hosts will be targeted:
acmechief2001.codfw.wmnet,acmechief1001.eqiad.wmnet,acmechief-test2001.codfw.wmnet,acmechief-test1001.eqiad.wmnet,acrab.codfw.wmnet,acrux.codfw.wmnet,actinium.wikimedia.org,alcyone.wikimedia.org,alsafi.wikimedia.org,aluminium.wikimedia.org,an-airflow1001.eqiad.wmnet,an-conf[1001-1003].eqiad.wmnet,an-coord1001.eqiad.wmnet,an-master[1001-1002].eqiad.wmnet,an-presto[1001-1005].eqiad.wmnet,an-tool[1005-1007].eqiad.wmnet,an-worker[1078-1095].eqiad.wmnet,analytics[1028-1031,1033-1077].eqiad.wmnet,analytics-tool[1001,1004].eqiad.wmnet,aqs[1004-1009].eqiad.wmnet,archiva1001.wikimedia.org,argon.eqiad.wmnet,auth2001.codfw.wmnet,auth1002.eqiad.wmnet,authdns[1001,2001].wikimedia.org,backup2001.codfw.wmnet,backup1001.eqiad.wmnet,bast[1002,2002,3004,4002,5001].wikimedia.org,boron.eqiad.wmnet,bromine.eqiad.wmnet,centrallog2001.codfw.wmnet,centrallog1001.eqiad.wmnet,chlorine.eqiad.wmnet,cloudbackup[2001-2002].codfw.wmnet,cloudcephmon[1001-1003].wikimedia.org,cloudcephosd[1001-1003].wikimedia.org,cloudcontrol[2001,2003]-dev.wikimedia.org,cloudcontrol[1003-1004].wikimedia.org,clouddb2001-dev.codfw.wmnet,cloudelastic[1001-1004].wikimedia.org,cloudmetrics[1001-1002].eqiad.wmnet,cloudnet[2002-2003]-dev.codfw.wmnet,cloudnet[1003-1004].eqiad.wmnet,cloudservices2002-dev.wikimedia.org,cloudservices[1003-1004].wikimedia.org,cloudstore[1008-1009].wikimedia.org,cloudvirt[2001-2003]-dev.codfw.wmnet,cloudvirt[1001-1009,1012-1014,1016-1030].eqiad.wmnet,cloudweb2001-dev.wikimedia.org,conf[2001-2003].codfw.wmnet,conf[1004-1006].eqiad.wmnet,contint[1001,2001].wikimedia.org,cp[2001-2026].codfw.wmnet,cp[1071,1073-1090,1099].eqiad.wmnet,cp[5001-5012].eqsin.wmnet,cp[3050-3065].esams.wmnet,cp[4021-4032].ulsfo.wmnet,cp1008.wikimedia.org,cumin2001.codfw.wmnet,cumin1001.eqiad.wmnet,db[2071-2135].codfw.wmnet,db[1074-1140].eqiad.wmnet,dbmonitor[1001,2001].wikimedia.org,dbprov[2001-2002].codfw.wmnet,dbprov[1001-1002].eqiad.wmnet,dbproxy[2001-2004].codfw.wmnet,dbproxy[1001-1003,1007-1008,1010-1021].eqiad.wmnet,dbstore[1003-1005].eqiad.wmnet,debmonitor2001.codfw.wmnet,debmonitor1001.eqiad.wmnet,deploy2001.codfw.wmnet,deploy1001.eqiad.wmnet,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org,doc1001.eqiad.wmnet,druid[1001-1006].eqiad.wmnet,dumpsdata[1001-1003].eqiad.wmnet,elastic[2025-2054].codfw.wmnet,elastic[1032-1067].eqiad.wmnet,elnath.codfw.wmnet,es[2001-2004,2011-2019].codfw.wmnet,es[1011-1019].eqiad.wmnet,etcd[1001-1006].eqiad.wmnet,etherpad1001.eqiad.wmnet,eventlog1002.eqiad.wmnet,failoid2001.codfw.wmnet,failoid1001.eqiad.wmnet,fermium.wikimedia.org,flerovium.eqiad.wmnet,francium.eqiad.wmnet,furud.codfw.wmnet,ganeti[2001-2018].codfw.wmnet,ganeti[1001-1008].eqiad.wmnet,ganeti[3001-3003].esams.wmnet,ganeti[4001-4003].ulsfo.wmnet,gerrit[1001,2001].wikimedia.org,grafana[1001-1002].eqiad.wmnet,graphite2003.codfw.wmnet,graphite1004.eqiad.wmnet,hassaleh.codfw.wmnet,hassium.eqiad.wmnet,helium.eqiad.wmnet,heze.codfw.wmnet,icinga[1001,2001].wikimedia.org,idp[1001,2001].wikimedia.org,install[1002,2002].wikimedia.org,kafka-jumbo[1001-1006].eqiad.wmnet,kafka-main[2001-2005].codfw.wmnet,kafka-main[1001-1005].eqiad.wmnet,kafkamon2001.codfw.wmnet,kafkamon1001.eqiad.wmnet,kraz.wikimedia.org,krb2001.codfw.wmnet,krb1001.eqiad.wmnet,kubernetes[2001-2006].codfw.wmnet,kubernetes[1001-1006].eqiad.wmnet,kubestage[1001-1002].eqiad.wmnet,kubestagetcd[1004-1006].eqiad.wmnet,kubetcd[2001-2006].codfw.wmnet,kubetcd[1004-1006].eqiad.wmnet,labsdb[1009-1012].eqiad.wmnet,labstore[2001-2004].codfw.wmnet,labstore[1004-1005].eqiad.wmnet,labstore[1006-1007].wikimedia.org,labtestpuppetmaster2001.wikimedia.org,labtestservices2003.wikimedia.org,labtestvirt2003.codfw.wmnet,labweb[1001-1002].wikimedia.org,ldap-corp[1001,2001].wikimedia.org,ldap-eqiad-replica[01-02].wikimedia.org,ldap-replica[2001-2002].wikimedia.org,logstash[2001-2006,2020-2025].codfw.wmnet,logstash[1007-1012,1020-1025].eqiad.wmnet,lvs[2001-2006,2009-2010].codfw.wmnet,lvs[1013-1016].eqiad.wmnet,lvs[5001-5003].eqsin.wmnet,lvs[3005-3007].esams.wmnet,lvs[4005-4007].ulsfo.wmnet,maps[2001-2004].codfw.wmnet,maps[1001-1004].eqiad.wmnet,matomo1001.eqiad.wmnet,mc[2019-2036].codfw.wmnet,mc[1019-1036].eqiad.wmnet,mendelevium.eqiad.wmnet,miscweb2001.codfw.wmnet,miscweb1001.eqiad.wmnet,moscovium.eqiad.wmnet,ms-be[2016-2056].codfw.wmnet,ms-be[1016-1026,1028-1059].eqiad.wmnet,ms-fe[2005-2008].codfw.wmnet,ms-fe[1005-1008].eqiad.wmnet,mw[2135-2147,2150-2212,2214-2290].codfw.wmnet,mw[1221-1235,1238-1258,1261-1290,1293-1348].eqiad.wmnet,mwdebug[2001-2002].codfw.wmnet,mwdebug[1001-1002].eqiad.wmnet,mwlog2001.codfw.wmnet,mwlog1001.eqiad.wmnet,mwmaint2001.codfw.wmnet,mwmaint1002.eqiad.wmnet,mx[1001,2001].wikimedia.org,ncredir[2001-2002].codfw.wmnet,ncredir[1001-1002].eqiad.wmnet,neon.eqiad.wmnet,netbox[1001,2001].wikimedia.org,netboxdb2001.codfw.wmnet,netboxdb1001.eqiad.wmnet,netflow2001.codfw.wmnet,netflow1001.eqiad.wmnet,netflow3001.esams.wmnet,netflow4001.ulsfo.wmnet,netmon[1002,2001].wikimedia.org,notebook[1003-1004].eqiad.wmnet,ores[2001-2009].codfw.wmnet,ores[1001-1009].eqiad.wmnet,orespoolcounter[2003-2004].codfw.wmnet,orespoolcounter[1003-1004].eqiad.wmnet,oresrdb[2001-2002].codfw.wmnet,oresrdb[1001-1002].eqiad.wmnet,pc[2007-2010].codfw.wmnet,pc[1007-1010].eqiad.wmnet,people1001.eqiad.wmnet,phab2001.codfw.wmnet,phab[1001,1003].eqiad.wmnet,ping2001.codfw.wmnet,ping1001.eqiad.wmnet,planet2001.codfw.wmnet,planet1001.eqiad.wmnet,poolcounter[2003-2004].codfw.wmnet,poolcounter[1004-1005].eqiad.wmnet,prometheus[2003-2004].codfw.wmnet,prometheus[1003-1004].eqiad.wmnet,proton[2001-2002].codfw.wmnet,proton[1001-1002].eqiad.wmnet,puppetboard2001.codfw.wmnet,puppetboard1001.eqiad.wmnet,puppetdb2002.codfw.wmnet,puppetdb1002.eqiad.wmnet,puppetmaster[2001-2002].codfw.wmnet,puppetmaster[1001-1003].eqiad.wmnet,pybal-test[2001-2003].codfw.wmnet,rdb[2003-2006].codfw.wmnet,rdb[1005-1006,1009-1010].eqiad.wmnet,registry[2001-2002].codfw.wmnet,registry[1001-1002].eqiad.wmnet,releases2001.codfw.wmnet,releases1001.eqiad.wmnet,relforge[1001-1002].eqiad.wmnet,restbase[2009-2020].codfw.wmnet,restbase[1016-1027].eqiad.wmnet,restbase-dev[1004-1006].eqiad.wmnet,rpki2001.codfw.wmnet,rpki1001.eqiad.wmnet,scandium.eqiad.wmnet,scb[2001-2006].codfw.wmnet,scb[1001-1004].eqiad.wmnet,schema[2001-2002].codfw.wmnet,schema[1001-1002].eqiad.wmnet,seaborgium.wikimedia.org,serpens.wikimedia.org,sessionstore[2001-2003].codfw.wmnet,sessionstore[1001-1003].eqiad.wmnet,snapshot[1005-1009].eqiad.wmnet,sodium.wikimedia.org,stat[1004-1007].eqiad.wmnet,theemin.codfw.wmnet,thorium.eqiad.wmnet,thumbor[2001-2004].codfw.wmnet,thumbor[1001-1004].eqiad.wmnet,torrelay1001.wikimedia.org,tungsten.eqiad.wmnet,vega.codfw.wmnet,wdqs[2001-2006].codfw.wmnet,wdqs[1003-1010].eqiad.wmnet,weblog1001.eqiad.wmnet,webperf[2001-2002].codfw.wmnet,webperf[1001-1002].eqiad.wmnet,wtp[2001-2020].codfw.wmnet,wtp[1025-1048].eqiad.wmnet,xhgui2001.codfw.wmnet,xhgui1001.eqiad.wmnet                                     
Confirm to continue [y/n]? y
===== NO OUTPUT =====                                                                                             
PASS:  |                                                                  |   0% (0/1391) [00:14<?, ?hosts/s]     
FAIL:  |โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 100% (1391/1391) [00:14<00:00, 93.54hosts/s]     
100.0% (1391/1391) of nodes failed to execute command 'ls -1 /etc/apt/p...|wikimedia.pref"': acmechief2001.codfw.wmnet,acmechief1001.eqiad.wmnet,acmechief-test2001.codfw.wmnet,acmechief-test1001.eqiad.wmnet,acrab.codfw.wmnet,acrux.codfw.wmnet,actinium.wikimedia.org,alcyone.wikimedia.org,alsafi.wikimedia.org,aluminium.wikimedia.org,an-airflow1001.eqiad.wmnet,an-conf[1001-1003].eqiad.wmnet,an-coord1001.eqiad.wmnet,an-master[1001-1002].eqiad.wmnet,an-presto[1001-1005].eqiad.wmnet,an-tool[1005-1007].eqiad.wmnet,an-worker[1078-1095].eqiad.wmnet,analytics[1028-1031,1033-1077].eqiad.wmnet,analytics-tool[1001,1004].eqiad.wmnet,aqs[1004-1009].eqiad.wmnet,archiva1001.wikimedia.org,argon.eqiad.wmnet,auth2001.codfw.wmnet,auth1002.eqiad.wmnet,authdns[1001,2001].wikimedia.org,backup2001.codfw.wmnet,backup1001.eqiad.wmnet,bast[1002,2002,3004,4002,5001].wikimedia.org,boron.eqiad.wmnet,bromine.eqiad.wmnet,centrallog2001.codfw.wmnet,centrallog1001.eqiad.wmnet,chlorine.eqiad.wmnet,cloudbackup[2001-2002].codfw.wmnet,cloudcephmon[1001-1003].wikimedia.org,cloudcephosd[1001-1003].wikimedia.org,cloudcontrol[2001,2003]-dev.wikimedia.org,cloudcontrol[1003-1004].wikimedia.org,clouddb2001-dev.codfw.wmnet,cloudelastic[1001-1004].wikimedia.org,cloudmetrics[1001-1002].eqiad.wmnet,cloudnet[2002-2003]-dev.codfw.wmnet,cloudnet[1003-1004].eqiad.wmnet,cloudservices2002-dev.wikimedia.org,cloudservices[1003-1004].wikimedia.org,cloudstore[1008-1009].wikimedia.org,cloudvirt[2001-2003]-dev.codfw.wmnet,cloudvirt[1001-1009,1012-1014,1016-1030].eqiad.wmnet,cloudweb2001-dev.wikimedia.org,conf[2001-2003].codfw.wmnet,conf[1004-1006].eqiad.wmnet,contint[1001,2001].wikimedia.org,cp[2001-2026].codfw.wmnet,cp[1071,1073-1090,1099].eqiad.wmnet,cp[5001-5012].eqsin.wmnet,cp[3050-3065].esams.wmnet,cp[4021-4032].ulsfo.wmnet,cp1008.wikimedia.org,cumin2001.codfw.wmnet,cumin1001.eqiad.wmnet,db[2071-2135].codfw.wmnet,db[1074-1140].eqiad.wmnet,dbmonitor[1001,2001].wikimedia.org,dbprov[2001-2002].codfw.wmnet,dbprov[1001-1002].eqiad.wmnet,dbproxy[2001-2004].codfw.wmnet,dbproxy[1001-1003,1007-1008,1010-1021].eqiad.wmnet,dbstore[1003-1005].eqiad.wmnet,debmonitor2001.codfw.wmnet,debmonitor1001.eqiad.wmnet,deploy2001.codfw.wmnet,deploy1001.eqiad.wmnet,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org,doc1001.eqiad.wmnet,druid[1001-1006].eqiad.wmnet,dumpsdata[1001-1003].eqiad.wmnet,elastic[2025-2054].codfw.wmnet,elastic[1032-1067].eqiad.wmnet,elnath.codfw.wmnet,es[2001-2004,2011-2019].codfw.wmnet,es[1011-1019].eqiad.wmnet,etcd[1001-1006].eqiad.wmnet,etherpad1001.eqiad.wmnet,eventlog1002.eqiad.wmnet,failoid2001.codfw.wmnet,failoid1001.eqiad.wmnet,fermium.wikimedia.org,flerovium.eqiad.wmnet,francium.eqiad.wmnet,furud.codfw.wmnet,ganeti[2001-2018].codfw.wmnet,ganeti[1001-1008].eqiad.wmnet,ganeti[3001-3003].esams.wmnet,ganeti[4001-4003].ulsfo.wmnet,gerrit[1001,2001].wikimedia.org,grafana[1001-1002].eqiad.wmnet,graphite2003.codfw.wmnet,graphite1004.eqiad.wmnet,hassaleh.codfw.wmnet,hassium.eqiad.wmnet,helium.eqiad.wmnet,heze.codfw.wmnet,icinga[1001,2001].wikimedia.org,idp[1001,2001].wikimedia.org,install[1002,2002].wikimedia.org,kafka-jumbo[1001-1006].eqiad.wmnet,kafka-main[2001-2005].codfw.wmnet,kafka-main[1001-1005].eqiad.wmnet,kafkamon2001.codfw.wmnet,kafkamon1001.eqiad.wmnet,kraz.wikimedia.org,krb2001.codfw.wmnet,krb1001.eqiad.wmnet,kubernetes[2001-2006].codfw.wmnet,kubernetes[1001-1006].eqiad.wmnet,kubestage[1001-1002].eqiad.wmnet,kubestagetcd[1004-1006].eqiad.wmnet,kubetcd[2001-2006].codfw.wmnet,kubetcd[1004-1006].eqiad.wmnet,labsdb[1009-1012].eqiad.wmnet,labstore[2001-2004].codfw.wmnet,labstore[1004-1005].eqiad.wmnet,labstore[1006-1007].wikimedia.org,labtestpuppetmaster2001.wikimedia.org,labtestservices2003.wikimedia.org,labtestvirt2003.codfw.wmnet,labweb[1001-1002].wikimedia.org,ldap-corp[1001,2001].wikimedia.org,ldap-eqiad-replica[01-02].wikimedia.org,ldap-replica[2001-2002].wikimedia.org,logstash[2001-2006,2020-2025].codfw.wmnet,logstash[1007-1012,1020-1025].eqiad.wmnet,lvs[2001-2006,2009-2010].codfw.wmnet,lvs[1013-1016].eqiad.wmnet,lvs[5001-5003].eqsin.wmnet,lvs[3005-3007].esams.wmnet,lvs[4005-4007].ulsfo.wmnet,maps[2001-2004].codfw.wmnet,maps[1001-1004].eqiad.wmnet,matomo1001.eqiad.wmnet,mc[2019-2036].codfw.wmnet,mc[1019-1036].eqiad.wmnet,mendelevium.eqiad.wmnet,miscweb2001.codfw.wmnet,miscweb1001.eqiad.wmnet,moscovium.eqiad.wmnet,ms-be[2016-2056].codfw.wmnet,ms-be[1016-1026,1028-1059].eqiad.wmnet,ms-fe[2005-2008].codfw.wmnet,ms-fe[1005-1008].eqiad.wmnet,mw[2135-2147,2150-2212,2214-2290].codfw.wmnet,mw[1221-1235,1238-1258,1261-1290,1293-1348].eqiad.wmnet,mwdebug[2001-2002].codfw.wmnet,mwdebug[1001-1002].eqiad.wmnet,mwlog2001.codfw.wmnet,mwlog1001.eqiad.wmnet,mwmaint2001.codfw.wmnet,mwmaint1002.eqiad.wmnet,mx[1001,2001].wikimedia.org,ncredir[2001-2002].codfw.wmnet,ncredir[1001-1002].eqiad.wmnet,neon.eqiad.wmnet,netbox[1001,2001].wikimedia.org,netboxdb2001.codfw.wmnet,netboxdb1001.eqiad.wmnet,netflow2001.codfw.wmnet,netflow1001.eqiad.wmnet,netflow3001.esams.wmnet,netflow4001.ulsfo.wmnet,netmon[1002,2001].wikimedia.org,notebook[1003-1004].eqiad.wmnet,ores[2001-2009].codfw.wmnet,ores[1001-1009].eqiad.wmnet,orespoolcounter[2003-2004].codfw.wmnet,orespoolcounter[1003-1004].eqiad.wmnet,oresrdb[2001-2002].codfw.wmnet,oresrdb[1001-1002].eqiad.wmnet,pc[2007-2010].codfw.wmnet,pc[1007-1010].eqiad.wmnet,people1001.eqiad.wmnet,phab2001.codfw.wmnet,phab[1001,1003].eqiad.wmnet,ping2001.codfw.wmnet,ping1001.eqiad.wmnet,planet2001.codfw.wmnet,planet1001.eqiad.wmnet,poolcounter[2003-2004].codfw.wmnet,poolcounter[1004-1005].eqiad.wmnet,prometheus[2003-2004].codfw.wmnet,prometheus[1003-1004].eqiad.wmnet,proton[2001-2002].codfw.wmnet,proton[1001-1002].eqiad.wmnet,puppetboard2001.codfw.wmnet,puppetboard1001.eqiad.wmnet,puppetdb2002.codfw.wmnet,puppetdb1002.eqiad.wmnet,puppetmaster[2001-2002].codfw.wmnet,puppetmaster[1001-1003].eqiad.wmnet,pybal-test[2001-2003].codfw.wmnet,rdb[2003-2006].codfw.wmnet,rdb[1005-1006,1009-1010].eqiad.wmnet,registry[2001-2002].codfw.wmnet,registry[1001-1002].eqiad.wmnet,releases2001.codfw.wmnet,releases1001.eqiad.wmnet,relforge[1001-1002].eqiad.wmnet,restbase[2009-2020].codfw.wmnet,restbase[1016-1027].eqiad.wmnet,restbase-dev[1004-1006].eqiad.wmnet,rpki2001.codfw.wmnet,rpki1001.eqiad.wmnet,scandium.eqiad.wmnet,scb[2001-2006].codfw.wmnet,scb[1001-1004].eqiad.wmnet,schema[2001-2002].codfw.wmnet,schema[1001-1002].eqiad.wmnet,seaborgium.wikimedia.org,serpens.wikimedia.org,sessionstore[2001-2003].codfw.wmnet,sessionstore[1001-1003].eqiad.wmnet,snapshot[1005-1009].eqiad.wmnet,sodium.wikimedia.org,stat[1004-1007].eqiad.wmnet,theemin.codfw.wmnet,thorium.eqiad.wmnet,thumbor[2001-2004].codfw.wmnet,thumbor[1001-1004].eqiad.wmnet,torrelay1001.wikimedia.org,tungsten.eqiad.wmnet,vega.codfw.wmnet,wdqs[2001-2006].codfw.wmnet,wdqs[1003-1010].eqiad.wmnet,weblog1001.eqiad.wmnet,webperf[2001-2002].codfw.wmnet,webperf[1001-1002].eqiad.wmnet,wtp[2001-2020].codfw.wmnet,wtp[1025-1048].eqiad.wmnet,xhgui2001.codfw.wmnet,xhgui1001.eqiad.wmnet
0.0% (0/1391) success ratio (< 100.0% threshold) for command: 'ls -1 /etc/apt/p...|wikimedia.pref"'. Aborting.
0.0% (0/1391) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.

Change 561884 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] apt::pin: manage the apt preferences directory and purge unmanaged resources

https://gerrit.wikimedia.org/r/561884

Change 561884 merged by Jbond:
[operations/puppet@production] apt::pin: manage the apt preferences directory and purge unmanaged resources

https://gerrit.wikimedia.org/r/561884

This ticket actually sounds like it was completed but is still open. ?

This ticket actually sounds like it was completed but is still open. ?

Yes your right thanks, closing