Currently hiera default values are either defined as a parameter to the lookup function or in hiera. When the defaults are defined in hiera we need to add them in two seoperate places:

• production: in the correct file, as per the wmflib::expand_path behaviour, under hieradata/common
• cloud: add them to cloud.yaml

This is ne area where we see lots of small issues like the following:

1. users updates a core module/profile to add a new parameter
2. they add a default value to the production hiera structure
3. merge
4. every thing works on production
5. some time later its noticed that puppet is broken on all cloud hosts with a message simlar to
   • "unable to find value for profile::someprofile::someparameter`
6. add the value to cloud.yaml, merge
7. cloud vps hosts working again

A simple way to resolve this would be to ensure that the cloud and profuction hiera configueration both took the default values from the same location. there are a number of ways to achive this

• add an entry to the cloud hiera.yaml config to add thesame wmflib::expand_bath lookup to happen at lower level of the hieararcy then cloud.yaml
• make use of in module hiera data. this moves the default values from hieradata/common to modules/profile/data. this adds the default values directly to the profile module and decouples them from either production or cloud hiera configurations
• Make use of automatic parameter lookups and put the default directly in the profile manifest

My personal opinion is that we should switch to automatic parameter lookups however this would be a big change, not necessarily from a technical PoV (we actually use automatic parameter lookups now but that's a separate issue). however the use of the lookup function is engrained in the policy and likely many engineers so it could be a difficult cultural change. We have also tried to completely disable it in the past so it has previously caused some issues. It would also be a big change, likely a lot of manual work and difficult to review.

With this in mind i think ne of the other two solutions would offer an easier path forward and we can punt the automatic parameter lookup debate to another day.