Page MenuHomePhabricator

Puppet failure on deploy-1006.devtools.eqiad1.wikimedia.cloud - Not authorized to call search on /file_metadata/volatile/GeoIP
Closed, ResolvedPublic

Description

Have been getting these for quite a while, decided to investigate just now.

brennen@deploy-1006:~$ sudo run-puppet-agent
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deploy-1006.devtools.eqiad1.wikimedia.cloud
Info: Applying configuration version '(57c50ad730) Btullis - Increase the heap for the mapreduce history service on an-master1003'
Error: /Stage[main]/Geoip::Data::Puppet/File[/usr/share/GeoIP]: Failed to generate additional resources using 'eval_generate': Error 500 on SERVER: Server Error: Not authorized to call search on /file_metadata/volatile/GeoIP with {:rest=>"volatile/GeoIP", :recurse=>true, :max_files=>0, :links=>"manage", :checksum_type=>"sha256", :source_permissions=>"ignore"}
Error: /Stage[main]/Geoip::Data::Puppet/File[/usr/share/GeoIP]: Could not evaluate: Could not retrieve file metadata for puppet:///volatile/GeoIP: Error 500 on SERVER: Server Error: Not authorized to call find on /file_metadata/volatile/GeoIP with {:rest=>"volatile/GeoIP", :links=>"manage", :checksum_type=>"sha256", :source_permissions=>"ignore"}

At a glance, this doesn't match anything on Portal:Cloud VPS/Admin/Runbooks/Cloud VPS alert Puppet failure on.

Event Timeline

Change #1026193 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] mediawiki/geoip: make loading geoip data from puppetserver optional

https://gerrit.wikimedia.org/r/1026193

Change #1026193 merged by Dzahn:

[operations/puppet@production] mediawiki/geoip: make loading geoip data from puppetserver optional

https://gerrit.wikimedia.org/r/1026193

@brennen Thanks for bringing this up. It was an older issue for me but this made me actually merge my change. I had waited a while because it's mediawki::common. But it's deployed now and I see no changes anywhere in prod.

We could now try to set profile::mediawiki::common::load_geoip_data_from_puppetserver to False in Hiera for just this instance and see if it fixes it.

Dzahn changed the task status from Open to In Progress.Jul 18 2024, 8:03 PM
Dzahn claimed this task.

doing that.. hold on

Added the Hiera key on the instance and it did indeed fix the puppet run.

Then moved it from instance level to the "puppet prefix" level for deploy*.

Info: Caching catalog for deploy-1006.devtools.eqiad1.wikimedia.cloud
Info: Applying configuration version '(873e9d5226) Jesse Hathaway - pcc-db1002: add hiera data'
Notice: Applied catalog in 9.85 seconds