When we do this it may be worth considering overriding <Property name="baseDir">/tmp/logs</Property> in WEB-INF/classes/log4j2.xml. We do override this in /etc/cas/config/log4j2.xml however it seems when the service is (re)started the value in the embedded file is used for a period until the config file in /etc/cas is read. you can see this by listing values in /tmp/logs

I managed to get 6.1 built however there seems to be an issue with u2f support

Looking at the list of supported ticketing registries we have the following options

This seems to be related to PUP-8187 the puppet master --compile option in puppet version 5.5 doesn't correctly support pson and would ultimately be resolved by T236481 or T236373. This should only effect nodes which have a catalogue that sends binary data which most commonly happens when a file has the source parameter set to some binary content. There is an experimental switch simlar to future_parser and ill try to test this and add support to pcc next week

I think this should be fixed now please reopen if there is still an issue

This was actully fixed by the following change https://gerrit.wikimedia.org/r/c/operations/puppet/+/550657. however @Vgutierrez still reports that the file causes issues so further investigation is still warranted

thanks Andrew. Please re-open if any further issues are observed

@Krenair I have also ran into the certificate issue while looking at labtestpuppetmaster. i have a patch out at the moment which would update the web config to use the certs under puppet config print ssldir --section master instead of puppet config print ssldir --section agent which is currently being used. This will mean that one will need to generate puppetmaster certs signed by the CA they manage. currently;

• cloud-puppetmaster-01.cloudinfra.eqiad.wmflabs has a client certificate it uses to talk to cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs they:
  • live under /var/lib/puppet/ssl
  • are signed by the CA on cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs
• cloud-puppetmaster-01.cloudinfra.eqiad.wmflabs is also the CA for [most] clients in openstack
  • Its certificates live under /var/lib/puppet/server/ssl
  • signs certs with the CA from cloud-puppetmaster-01.cloudinfra.eqiad.wmflabs

Hi Andrew

@aborrero The current script references a file which is only avalible in the CA directory and therefore only available on the puppet master front end servers. I have created a CR to update this to use the localcacert location. I have also disabled the uyrllib warning

When you say it was removed, can you expand a bit. I see puppet_compiler/templates/run_wrapper.erb was removed which made reference to NUM_THREADS. however im not familure with that script or how its used. It is still possible to use NUM_THREADS environment variable from the compiler hosts e.g. CHANGE=546452 NODES=cp5001.eqsin.wmnet,cp5007.eqsin.wmnet BUILD_NUMBER=1 NUM_THREADS=2 puppet-compiler. Further the options is available if you schedule a job via the web interface

awesome thanks

what is still left to change in order to complete this?

What is the status of this considering we are now [i plan to drop the old puppetdb's completely next week] on puppetdb6

suspect this can be closed but reopen if im wrong

All puppet masters have been upgraded

The ops production puppet is now all running puppet 5. upgrades to wmcs are tracked in https://phabricator.wikimedia.org/T235218

This is now implemented

This has now been implmented

This is complete, please reopen if further issues/improvements

This is now complte

new puppetdbs aren ow live

complet

i think this is resolved now, please reopen if there is still an issue

@Eevans moritz mentioned there maybe some cassandra consideration to take into account and you could enlighten me as to what they are :)

I attempted a patch for this upstream although its not quite working yet

Think this is complete now

In T237259#5634722, @Joe wrote:

As far as etcd is concerned, a rolling restart should be enough to ensure the new CA is picked up. I will take care of that.

base::expose_puppet_certs - This can be ignored as it relates to the client key pairs and not the CA public certificate which is the file which is changing

In T236277#5631496, @CDanis wrote:

In terms of identifying services that use keys issued by the puppet CA -- is it wrong to think that the following would be a complete list?

• keys created using cergen
• users of base::expose_puppet_certs
• the few users we have that are referencing either puppet_ssldir() or manually hardcoding the /var/lib/puppet/ssl directory

I'd think it should be relatively straightforward to find all such users just by querying puppetdb.

In T236277#5631271, @akosiaris wrote:

IMPORTANT: The puppet CA cert (and correspondingly key), is used as a "master" (a failsafe in case the actual host key is not around) key for bacula backups. That is, if we lose it we won't be able to restore backups for hosts that no longer are around. This is documented under https://wikitech.wikimedia.org/wiki/Bacula#Restore_from_a_non-existent_host_(missing_private_key).

In T236277#5631150, @jbond wrote:

We also need to consider /usr/local/share/ca-certificates/Puppet_Internal_CA.crt which is linked to /etc/ssl/certs/Puppet_Internal_CA.pem. This cert is used for validating other services which rely on the puppetca. Once the new certificate is in place it is likely that we will need to restart/reload any deamons that use theses certificates so they have the new CA in memory. The major pain point here is likely MySQL

We also need to consider /usr/local/share/ca-certificates/Puppet_Internal_CA.crt which is linked to /etc/ssl/certs/Puppet_Internal_CA.pem. This cert is used for validating other services which rely on the puppetca. Once the new certificate is in place it is likely that we will need to restart/reload any deamons that use theses certificates so they have the new CA in memory. The major pain point here is likely MySQL

The volatile URI has been moved both puppetmasters

In T235218#5628537, @Krenair wrote:

So I've started making cloud-puppetmaster-04 ready to be a backend, it's got puppet errors about not being able to find the geoipupdate package. The only buster instance in labs with it installed is jbond-pmaster-buster.puppet.eqiad.wmflabs, which gets it from 500 http://deb.debian.org/debian buster/contrib amd64 Packages. Looking at that instance a little more closely that will be coming from this first line of /etc/apt/sources.list: deb http://deb.debian.org/debian/ buster main non-free contrib, however the file appears unpuppetised and on cloud-puppetmaster-04 the line is just deb http://deb.debian.org/debian/ buster main. @jbond, did you add that contrib component yourself or are we missing something? Where do production puppetmasters get it from?

Module has now been removed

Closing please reopen if further issues observed