WMCS standalone puppet master does not lookup cherry picked hiera change
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Mar 16 2021, 9:50 AM

Description

deployment-prep uses a standalone puppetmaster, when cherry picking a patch that affects hieradata the change is not reflected.

I have cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/672658/ on deployment-puppetmaster04.deployment-prep.eqiad.wmflabs which adds:

hieradata/common/profile/ci/slave/labs/common.yaml

profile::java::java_packages:
    - version: 11
      variant: jre-headless

The new value is not found, it ends up to fallback on the default: []

Puppet hierarchy is:

/etc/puppet/hiera.yaml

version: 5
defaults:
  datadir: /etc/puppet/hieradata
  data_hash: yaml_data
hierarchy:
  - name: 'Http Yaml'
    data_hash: cloudlib::httpyaml
    uri: "http://puppetmaster.cloudinfra.wmflabs.org:8100/v1/%{::labsproject}/node/%{facts.fqdn}"
  - name: "cloud hierarchy"
    paths:
      - "cloud/%{::wmcs_deployment}/%{::labsproject}/hosts/%{::hostname}.yaml"
      - "cloud/%{::wmcs_deployment}/%{::labsproject}/common.yaml"
      - "cloud/%{::wmcs_deployment}.yaml"
      - "cloud.yaml"
  - name: "Secret hierarchy"
    path: "%{::labsproject}.yaml"
    datadir: "/etc/puppet/secret/hieradata"
  - name: "Private hierarchy"
    paths:
      - "labs/%{::labsproject}/common.yaml"
      - "%{::labsproject}.yaml"
      - "labs.yaml"
    datadir: "/etc/puppet/private/hieradata"
  - name: "Common hierarchy"
    path: "common.yaml"
  - name: "Secret Common hierarchy"
    path: "common.yaml"
    datadir: "/etc/puppet/secret/hieradata"
  - name: "Private Common hierarchy"
    path: "common.yaml"
    datadir: "/etc/puppet/private/hieradata"

If I look it up on deployment-puppetmaster04:

sudo puppet lookup --compile --explain --node deployment-deploy01.deployment-prep.eqiad.wmflabs profile::java::java_packages

Warning: Undefined variable 'puppetmaster'; \n   (file & line not available)
Warning: Unknown variable: 'profile::backup::host::jobdefaults'. (file: /etc/puppet/modules/backup/manifests/set.pp, line: 10, column: 33)
Warning: Unknown variable: 'profile::backup::host::jobdefaults'. (file: /etc/puppet/modules/backup/manifests/set.pp, line: 10, column: 33)

Searching for "profile::java::java_packages"
  Global Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppet/hiera.yaml"

    Hierarchy entry "Http Yaml"

      URI "http://puppetmaster.cloudinfra.wmflabs.org:8100/v1/deployment-prep/node/deployment-deploy01.deployment-prep.eqiad.wmflabs"
        Original uri: "http://puppetmaster.cloudinfra.wmflabs.org:8100/v1/%{::labsproject}/node/%{facts.fqdn}"
        No such key: "profile::java::java_packages"

    Hierarchy entry "cloud hierarchy"

      Path "/etc/puppet/hieradata/cloud/eqiad1/deployment-prep/hosts/deployment-deploy01.yaml"
        Original path: "cloud/%{::wmcs_deployment}/%{::labsproject}/hosts/%{::hostname}.yaml"
        Path not found

      Path "/etc/puppet/hieradata/cloud/eqiad1/deployment-prep/common.yaml"
        Original path: "cloud/%{::wmcs_deployment}/%{::labsproject}/common.yaml"
        No such key: "profile::java::java_packages"

      Path "/etc/puppet/hieradata/cloud/eqiad1.yaml"
        Original path: "cloud/%{::wmcs_deployment}.yaml"
        No such key: "profile::java::java_packages"

      Path "/etc/puppet/hieradata/cloud.yaml"
        Original path: "cloud.yaml"
        Found key: "profile::java::java_packages" value: []

Or in other words, it does not seem to search /etc/hieradata/common.

Apparently caused by the migration to hiera 5 https://gerrit.wikimedia.org/r/c/operations/puppet/+/615159 by @jbond , previously we had:

hiera.yaml

defaults:
  datadir: /etc/puppet/hieradata
:hierarchy:
  ...
  - "common"
  ...

I guess if the patch is merged, the lookup will be found via puppetmaster.cloudinfra.wmflabs.org

The workaround for the cherry-pick to work would be to use the generic /etc/puppet/hieradata/cloud.yaml instead of /etc/puppet/hieradata/common/profile/ci/slave/labs/common.yaml .

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	contint: remove erroneous hiera setting for labs	operations/puppet	production	+0 -3

Customize query in gerrit

Related Objects

Mentioned In: T269354: Switch Jenkins servers to Java 11

Event Timeline

hashar created this task.Mar 16 2021, 9:50 AM

Restricted Application edited projects, added cloud-services-team (Kanban); removed cloud-services-team. · View Herald TranscriptMar 16 2021, 9:50 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

hashar updated the task description. (Show Details)Mar 16 2021, 9:52 AM

taavi added projects: Beta-Cluster-Infrastructure, Puppet.Mar 16 2021, 9:52 AM

taavi subscribed.

@hashar tl;dr the you should place the value in hieradata/cloud.yaml to set a sane default for all cloud projects and hieradata/cloud/eqiad1/deployment-prep/common.yaml to override in deployment-prep

This is not related to the hiera5 switch but more related the differences between cloud-vps and prod hiera lookups, specificity cloud vps hiera dose not use wmflib::expand_path, in hiera3 this was the equivalent of the nuyaml3 expand_path config which never existed in labs, in fact labs was still using nuyaml vs nuyaml3 which potentially didn't even support expand_paths (before my time)

hashar mentioned this in T269354: Switch Jenkins servers to Java 11.Mar 16 2021, 4:25 PM

I just misunderstood how hiera lookup works on WMCS. The bulk of it is that parameters specific to profiles are never looked up since commons directory is not part of the hierarchy.

My use case is that for hosts having profile::ci::slave::labs::common, I would like to set:

profile::java::java_packages:
    - version: 11
      variant: jre-headless

Maybe I can just set it project wide for the integration and puppet-diffs project since there are no other use cases for Java.

For deployment-prep that would surely cause some troubles here and there when java_packages is already defined or when some profile already set it to use Java 8. I guess I can do it on a per host based (solely deployment-deploy01 instance would need it).

Change 673286 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] contint: remove erroneous hiera setting for labs

https://gerrit.wikimedia.org/r/673286

gerritbot added a project: Patch-For-Review.Mar 18 2021, 2:50 PM