@Peter this has been deployed now, please allow upto 30 minutes for the change to fully propagate, if you are still seeing issues after that please reopen this ticket

nodes with changes

there are a few nodes which have diffes like the following. This is caused when an attribute has a value of undef

Hello All,

wiki has been updated https://wikitech.wikimedia.org/w/index.php?title=Mailman&type=revision&diff=1837413&oldid=1826755

I think this is a great idea. As to which host, the cumin server makes sense to me or perhaps bastion? The user is a bit of a pain, it would be nice if we could have a global config e.g. /etc/ripe-atlas-tools/config but a quick look at the code suggests all config is per user. Long term it would be nice to send a PR to add support for a global config, short term perhaps use the user atlas. If we install the the ripe-atlas package with either --user or in a virtualenv we can then create some wrapper scripts e.g.

@alexhollender i have no added your wmf access to your ldap account please let me know if you are still unable to access resources

Hi Greg,

@greg are you able to approve this access request

@alexhollender what is your shell username i don't see anything obvious

@zeljkofilipin I have [now] set the subscription model to Require approval. I leave it to the admins to change the other privacy settings. @Jrbranaa should have recived the admin password via email so should be able to share that with you

Hello Jean-Rene,

Hello Greg,

I tried running the followin command on the server however the Current Cache policy remains as WriteThrough

@Urbanecm creating this un-triaged task with the Operations tag should be enough to bring it to the attention of the clinic duty, this task must have slipped through the gaps sorry about that. In relation to your specific issue. i just tried re-running your command at it worked without issue

@Nuria are you able to approve @Urbanecm access to researchers and analytics-privatedata-users

In T102099#5433561, @jcrespo wrote:

Hi, I am bit disconnected about the planning of deployment of this- Once all hosts (or all hosts that are planned above being migrated, is the puppet line supposed to go on the profile (or role) or on base.pp with some exclussions? It is not clear based on the ticket description and comments, or I may have missed it as it is a long ticket :-D.

When i started I believe someone from OIT created an office wiki page based of a standard template. There is also an Ops specific page which i think is referenced in the first wiki.

When this alerted today and ended up going right down the rabbit hole. Anyway i think i have found out some information.

The lua hack seems to have worked. I have again updated the config to send some canary servers to puppetmaster1003. So far i have seen no errors in the puppetdb log. Further via puppetdb i can see that the puppet reports and facts are all refreshing.

I have disabled puppetmaster1003 for now, unfortunately from reading the PUP-8901 It seems the advice from puppetlabs is to always upgrade all puppetmasters and the puppetdb in tandem

It seems when that the severs using the new puppet master cause the following stack trace when they try reach the 'store report' phase

Seems to correlate well with when i enabled the canary puppet master and when i started adding more canary hosts

@EBernhardson im not sure where the data above came from but i suspect a typo somewhere 2620:0:860::ed1a::1 dose not exist anywhere in the puppet repo. however 2620:0:860:ed1a::1 (notice the missing ':') is valid and is routed to the lvs eqiad servers. That said it does look like most lvs services are in the 2620:0:860:ed1a:: prefix an the servers i have found in the 2620:0:861:1:: prefix seem to be real machines so i suspect that the ipv6 address may be incorrect

Thanks brandon, Ill take a look at removing the ones SLAAC addresses from puppet this week. One of them, at least, was added by me and was what led me down this rabbit hole :)

Our current set up seems to use SLACC without any of the privacy extensions as such the lower 64 bits of the ipv6 address are composed of the mac address of the interface. It seems this would be fairly simple to script the AAAA records and then im not sure if we need to do much configuration. Although the conversion then turns to what do we do with all the hosts that currently have interface::add_ip6_mapped

Answering my own question, as NIC's can and do change in a server lifetime using SLAAC like this is undesirable as we dont want an IP change if we change a NIC

I think this is a really good idea. further after a bit of investigation i think any arbitrary string can be used. Later versions of the puppet documentation even use git rev-parse as an example command. also r10k has a script in there repo which has the sha1, git server and last git log message as the version

Hi all,

Sorry for the interuption, this issues ws caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/527064 when i removed the config entry at the common level. I have now added a default of 0 canary_hosts in labs.yaml and comon/profile/puppetmaster/frontend.yaml. Please reopen if you still see issues

Have a script that periodically generates a hiera file with that information for each host, to be merged by the puppetmasters with the public tree as we already do with the private repo

In case this needs rolling back the the issue can be fixed in 4.8 with the following patch https://phabricator.wikimedia.org/P8772

yes, closing thanks

puppetmaster1003 has now been build and is running puppet 5.5

Here is another example. puppetmaster1003.eqiad.wmnet is in both 'Hosts that have no differences' and 'Hosts that fail to compile when the change is applied'

In T224977#5265178, @Vgutierrez wrote:

After checking https://puppet-compiler.wmflabs.org/compiler1001/16855/ change error/warning logs for hosts marked as "fail to compile when the change is applied" it looks like two warnings are being interpreted as errors:

Warning: Unknown variable: '::restricted_to'. at /srv/jenkins-workspace/puppet-compiler/16855/change/src/modules/profile/manifests/ldap/client/labs.pp:5:72
Warning: Unknown variable: '::restricted_from'. at /srv/jenkins-workspace/puppet-compiler/16855/change/src/modules/profile/manifests/ldap/client/labs.pp:6:76

This error is triggered slower on some machines vs others

• cp2026 (cache::upload): takes a bit longer to die ~ 5 minutes
• cp1085 (cache::text): dies instantly
• cp1075 (cache::text): dies instantly
• cp1076 (cache::upload): dies instantly

Unfortunately this had to be downgraded again. We saw the following error with the new version

I have now pushed a patch so so that mtail uses -logs /dev/stdin instead of -logfds 0. would you like me to upgrade everything to mtail_3.0.0~rc24.1-1+wmf1_amd64.deb again?

below are all the hosts with puppetdb package installed

Investigating further this is due to how populate_puppetdb adds entries to the database. p`opulate_puppetdb` loops through the site.pp manifest picks one host from every role and compiles its catalog locally so that the result is sent to the puppetdb. This means that the puppet DB is populated and the other calls to query_resource have enough data for them succeed.