This server is now active in production

In T243226#5824821, @Krenair wrote:

Hi alex,

hi alex,

>>! In T243226#5817998, @Krenair wrote:

am guessing this is just us needing to get a new puppetmaster with buster instead of stretch

I created a quick ruby script

I believe this was caused by a number oif issues which have since been fixed

• not fully merging labs changes
• not passing the sha1 correctly
• more sha1 checks
• running puppet-merge from multiple hosts

Looking at the code it only accepts SHA256withECDSA which seems to be the only supported type in FIDO. The error message indicates the key being sent is a EC key

great, ill close this then :)

Looks like this is caused by PUP-8067, i have created a new CR

What server was this so i can attempt to recreate?

only managed files now exist in /etc/apt/prefrences.d

• removed /etc/apt/preferences.d/t.pref and /etc/apt/preferences.d/t2.pref as pined version is no longer available
• remove mitaka_stretch_nojessiebpo_clientpackages.pref and mitaka_stretch_nojessiebpo_clientpackages.pref as component openstack-mitaka-jessie no longer avalible and systems are jessie
• remove jessie_mitaka_pinning_nova_compute_kvm.pref and jessie_mitaka_pinning_nova_compute.pref package is not installed but version specified is allready most prefered via jessie-wikimedia/openstack-mitaka-jessie

remaining unmanaged files under /etc/apt/prefrences.d/

• remove python3{-,_}{ldap3,pyasn}.pref and nethogs.pref as package available in jessie-wikimedia/openstack-mitaka-jessie

• remove /etc/apt/preferences.d/reprepro.pref from install servers as package is now in jessie-wikimedia/backports
• remove /etc/apt/preferences.d/go.pref on kubeernetes servers as it refered to jessie and theses machines are stretch
• remove /etc/apt/preferences.d/git_from_backports.pref from contint as git provided by component/ci
• remove /etc/apt/preferences.d/libapache2_mod_security2.pref from fermium as package is now in jessie-wikimedia/backports
• remove as package is now provided by stretch-wikimedia/thirdparty/elasticsearch-curator5 and ping is maintained via generic pin[1]
• sudo cumin 'cloudvirt[2001-2003]-dev.codfw.wmnet,cloudservices2002-dev.wikimedia.org' 'rm /etc/apt/preferences.d/mitaka_stretch_nojessiebpo.pref' as systems are no longer jessie
• sudo cumin 'scb[2001-2006].codfw.wmnet,scb[1001-1004].eqiad.wmnet' 'rm -f /etc/apt/preferences.d/python3_tornado.pref' as now provided by jessie-wikimedia/backports
• remove /etc/apt/preferences.d/ruby-json.pref from elnath as system is spare

remove the followinf old files

• /etc/apt/preferences.d/puppet_all.pref
• /etc/apt/preferences.d/facter.pref
• /etc/apt/preferences.d/smartmontools.pref

In T214605#5756945, @jbond wrote:

I have generated a list of managed sources files with the following snippet

from pypuppetdb import connect
from pypuppetdb.QueryBuilder import RegexOperator
from os.path import basename
found_files = set()
db = connect()
query_regex = RegexOperator('title', '^\/etc\/apt\/sources.list.d\/')
resources = db.resources('file', query=query_regex)
for resource in resources:
  found_files.add(basename(resource.name))
  print('|'.join(found_files))

The on cumin ran the following to find any unmanaged files

$ sudo cumin 'A:all' 'ls -1 /etc/apt/sources.list.d | grep -Ev "getenvoy-jessie.list|component-puppet5.list|thirdparty-ceph-nautilus-buster.list|wikimedia-spicerack.list|thirdparty-cloudera.list|wikimedia.list|thirdparty-k8s.list|component-facter3.list|component-pyall.list|wikimedia-cassandra311.list|ffmpeg-vp9.list|wikimedia-curator.list|thirdparty-grafana.list|thirdparty-ci.list|openstack-ocata-stretch.list|wikimedia-cassandra22.list|wikimedia-php72.list|wikimedia-thumbor.list|wikimedia-openjdk8.list|wikimedia-elastic.list|openstack-mitaka-jessie.list|component-ci.list|thirdparty-confluent.list|wikimedia-zookeeper349.list|amd-rocm271.list|stretch-node10.list|thirdparty-tor.list|wikimedia-node10.list|openjdk-8.list|debian-backports.list|buster-cergen.list|package-build-deb-src.list|stretch-wikimedia-thirdparty-oath.list|jenkins-thirdparty-ci.list" || true'

This resulted in no output

In T241277#5762041, @jcrespo wrote:

@CDanis Is this something you plan to work on? Otherwise, who do you need help with? I am trying to triage the importance of this ticket.

I have generated a list of managed sources files with the following snippet

I queried the puppetdb using pypuppetdb and there is only one file in sources.list.d not managed by puppet

In T236954#5749658, @colewhite wrote:

The changesets look great and appear to do the right thing.
The only other thing I could think of doing is to have yamllint run but not fail the build until we've cleaned up a bit. That way we can check against the linter as we go and not break everybody's workflow simultaneously.

In T238833#5749025, @Ottomata wrote:

Just so that you aren't caught off guard
file {'/srv/private/secret/secrets/certificate':

I think this idea would work with the existing puppetization. Since puppet is managing this as well as '/srv/private', I think it will know to use the more specific rule in the right place. I could be wrong though....

This information has been documented in https://wikitech.wikimedia.org/wiki/User:Jbond/Encryption

This is completed https://grafana-next.wikimedia.org/d/spring_boot_21/spring-boot-statistics

I created a quick CR to test out yamllint and another to make sure it only runs against changed files. I change added the following to the config to the one you provided however there are still a lot of violations

In T240324#5737046, @jcrespo wrote:

I am trying to triage, this seems like an internal "nice to have" feature, like T178575 but not something that probably requires immediate action. Maybe it should have lower priority in the context of Operations, but will not touch it if @jbond plans to work on it soon.

This is now complete all ops productions servers are on puppet 5

Have been trying to document some of the SSL findings and it appears to me that conftool client is currently preforming no SSL validations. Specificity

I have now updated the relevant CA files in the private repo

ema, confirmed that the traffic servers did need a restart which has now been preformed

• 'trafficserver' restarted on all cache-ats nodes
• puppetdb and postgresql restarted on puppetdb[12]002
• postgresql restarted on netbox[12]001
• stunnel4 service restarted on all service using rsync warped with stunnel
• rsyslog restarted every where that has kafka shipping
• uwsgi-netbox restarted on netbox servers other references are from systemd timers and no action needed
• varnishkafka-webrequest and varnishkafka-eventlogging restarted on effected servers
• no restart required
  • mysqld_exporter_config.py is ran by puppet
  • backup_mariadb.py ran by cron
  • check_mariadb.py ran by icinga
  • debmonitor::client either run as an apt hook or via cron