Page MenuHomePhabricator

Hiera is not properly configured on Nodepool instances
Closed, ResolvedPublic

Description

As part of magically provisioning dependencies on Nodepool instances (eg https://gerrit.wikimedia.org/r/#/c/274675/ ) I have found out Hiera was falling due to empty files (fixed: T128846: DIB images have /puppet/hieradata files that are empty).

While debugging, I also found that Hiera lookup is broken on Nodepool instances. Namely it is missing the fact for labsproject:

$ /puppet/utils/hiera_lookup --fqdn=`hostname --fqdn` labs project

$ facter labsproject

$ 

The hiera configuration file references ::labsproject and since it is empty, a wild range of Hiera hierarchy is not traversed. Relevant conf from hiera.yaml

:hierarchy:
  - "labs/hosts/%{::hostname}"
  - "labs/%{::labsproject}/host/%{::hostname}"
  - "labs/%{::labsproject}/common"
  - "labs"
  - "secret/%{::labsproject}"
  - "private/%{::labsproject}"
  - common
  - "secret/common"
  - "private/common"

Using hiera_lookup in verbose mode (-v):

jenkins@ci-jessie-wikimedia-45448:~$ /puppet/utils/hiera_lookup --fqdn=`hostname --fqdn` labsproject -v
DEBUG: 2016-03-07 16:38:25 +0000: Looking up labsproject

DEBUG: 2016-03-07 16:38:25 +0000: Loading info from hosts/ci-jessie-wikimedia-45448 for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: The source is: hosts/ci-jessie-wikimedia-45448
DEBUG: 2016-03-07 16:38:25 +0000: Cannot find datafile /puppet/hieradata/hosts/ci-jessie-wikimedia-45448.yaml, skipping
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in 
DEBUG: 2016-03-07 16:38:25 +0000: Loading info from regex/ci-jessie-wikimedia-45448.eqiad.wmflabs for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: Regex match going on - using regex.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in /puppet/hieradata/regex.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading file /puppet/hieradata/regex.yaml
<snip Scanning label foo>
DEBUG: 2016-03-07 16:38:25 +0000: Loading info from eqiad/labs for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: The source is: eqiad/labs
DEBUG: 2016-03-07 16:38:25 +0000: Cannot find datafile /puppet/hieradata/eqiad/labs.yaml, skipping
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in 
DEBUG: 2016-03-07 16:38:25 +0000: Loading info from eqiad for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: The source is: eqiad
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in /puppet/hieradata/eqiad.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading file /puppet/hieradata/eqiad.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading info from private/eqiad/labs for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: The source is: eqiad/labs
DEBUG: 2016-03-07 16:38:25 +0000: Cannot find datafile /puppet/hieradata/eqiad/labs.yaml, skipping
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in 
DEBUG: 2016-03-07 16:38:25 +0000: Loading info from private/eqiad for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: The source is: eqiad
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in /puppet/hieradata/eqiad.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading file /puppet/hieradata/eqiad.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading info from common for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: The source is: common
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in /puppet/hieradata/common.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading file /puppet/hieradata/common.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading info from private/common for labsproject
DEBUG: 2016-03-07 16:38:25 +0000: The source is: common
DEBUG: 2016-03-07 16:38:25 +0000: Searching for labsproject in /puppet/hieradata/common.yaml
DEBUG: 2016-03-07 16:38:25 +0000: Loading file /puppet/hieradata/common.yaml

jenkins@ci-jessie-wikimedia-45448:~$

Facter is unahppy:

$ facter hostname realm labsproject labsprojectfrommetadata
hostname => ci-jessie-wikimedia-45448
labsproject => nil
labsprojectfrommetadata => nil
realm => nil

It seems labsproject and realm are populated from puppet.git manifests/realm.pp which we might not include.

The labsprojectfrommetadata would be shipped via modules/base/lib/facter/labsprojectfrommetadata.rb:

Facter.add(:labsprojectfrommetadata) do

And we can't really include ::base because it ships too much unrelated stuff that ends up breaking puppet provisioning on the very lightweight image.

Event Timeline

hashar created this task.Mar 7 2016, 4:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 7 2016, 4:46 PM

You can add facts by passing in environment variables:

$ FACTER_MYVARNAME=blah puppet apply

This will cause Facter to load MYVARNAME as a facter variable.

hashar added a comment.Mar 7 2016, 5:57 PM
$ facter labsproject

$ FACTER_LABSPROJECT=foobar facter labsproject
foobar
$

Super nice hack @mobrovac ! I guess that is a fair workaround to populate facts :-} Thank you

Change 275786 had a related patch set uploaded (by Hashar):
dib: set $labsproject

https://gerrit.wikimedia.org/r/275786

Change 275786 merged by jenkins-bot:
dib: set $labsproject

https://gerrit.wikimedia.org/r/275786

hashar triaged this task as High priority.Mar 8 2016, 11:56 AM
hashar moved this task from Backlog to In-progress on the Continuous-Integration-Scaling board.
hashar claimed this task.Mar 8 2016, 1:53 PM

operations/puppet.git has a hiera utility /utils/hiera_lookup but it does not support labs anymore so I had to fix it:

https://gerrit.wikimedia.org/r/276345
https://gerrit.wikimedia.org/r/276346

Change 276346 had a related patch set uploaded (by Hashar):
hiera_lookup: recognize labs project and site

https://gerrit.wikimedia.org/r/276346

Change 276345 had a related patch set uploaded (by Hashar):
hiera_lookup: support 'labs' realm

https://gerrit.wikimedia.org/r/276345

hashar closed this task as Resolved.Mar 10 2016, 12:11 PM

I have used the updated /utils/hiera_lookup on a Nodepool instance and it seems to work.

$ cat labs/contintcloud/common.yaml 
---
# install dev packages for services
service::configuration::use_dev_pkgs: true
ci-jessie-wikimedia-48049:~$ PUPPETDIR=/puppet ./hiera_lookup --fqdn=ci-jessie-wikimedia-48049.contintcloud.eqiad.wmflabs service::configuration::use_dev_pkgs
true

Been fixed by setting $::labsproject in puppet via https://gerrit.wikimedia.org/r/#/c/275786/

Change 276345 merged by Alexandros Kosiaris:
hiera_lookup: support 'labs' realm

https://gerrit.wikimedia.org/r/276345

Change 276346 merged by Filippo Giunchedi:
hiera_lookup: recognize labs project and site

https://gerrit.wikimedia.org/r/276346

Last patch landed in puppet.git so that is definitely fixed.