Puppet broken everywhere on labs
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	yuvipanda
	Dec 25 2014, 12:02 AM

Description

Puppet failing on all labs hosts, with:

yuvipanda@shinken-01:~$ sudo puppet agent -tv
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 400 on SERVER: 'find' is already in the '/certificate_request' ACL
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': Error 400 on SERVER: 'find' is already in the '/certificate_request' ACL
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Error 400 on SERVER: 'find' is already in the '/certificate_request' ACL Could not retrieve file metadata for puppet://virt1000.wikimedia.org/plugins: Error 400 on SERVER: 'find' is already in the '/certificate_request' ACL
Info: Loading facts in /var/lib/puppet/lib/facter/lldp.rb
Info: Loading facts in /var/lib/puppet/lib/facter/projectgid.rb
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ec2id.rb
Info: Loading facts in /var/lib/puppet/lib/facter/apt.rb
Info: Loading facts in /var/lib/puppet/lib/facter/default_gateway.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_config_dir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/physicalcorecount.rb
Info: Loading facts in /var/lib/puppet/lib/facter/meminbytes.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: 'find' is already in the '/certificate_request' ACL
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 400 on SERVER: 'find' is already in the '/certificate_request' ACL

Event Timeline

yuvipanda created this task.Dec 25 2014, 12:02 AM

yuvipanda raised the priority of this task from to Needs Triage.

yuvipanda triaged this task as Unbreak Now! priority.

yuvipanda updated the task description. (Show Details)

yuvipanda added projects: Puppet, acl*sre-team, Cloud-Services.

yuvipanda set Security to None.

yuvipanda added subscribers: yuvipanda, coren.

Doesn't affect instances that have a non virt1000 puppetmaster - self puppetmasters with latest ops/puppet clone and deployment-prep instances both work fine.

Turns out it was a stray line in /etc/puppet/auth.conf in virt1000. Offending lines were:

path /certificate_request
auth any
method find, save
allow *
method find, save

The duplicated method definition was causing the problem. I removed it and it was fixed.

I'm not actually sure how that extra line came about. Digging some more, I discovered that that particular file used to be managed via puppet, but was removed as redundant in https://gerrit.wikimedia.org/r/#/c/137886/. The file wasn't explicitly removed, and stayed on without causing too much trouble.

Still a mystery as to how the extra line came about, but that's perhaps ok. What's life without a few mysterious puppet failures on Christmas? :)

@Joe think I can remove that file now?

Things are recovering now.

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:56 PM

Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptJun 7 2017, 6:56 PM

Puppet broken everywhere on labsClosed, ResolvedPublicActions

Description

Event Timeline

Puppet broken everywhere on labs
Closed, ResolvedPublic
Actions