Since moving to puppet7 puppet runs have started to have sporadic faliures with error messages such as
/Stage[main]/Base::Screenconfig/File[/root/.screenrc] Could not evaluate: Could not retrieve information from environment production source(s) puppet:///modules/base/screenrc
Theses errors happen at the time of puppet-merge and are a by product of using g10k. when we preform puppet merger to the puppet servers we update a repo in /srv/git/puppet and then use g10k to sync the data to /srv/puppet_code. during this time there is a short period of time when there is no data in /srv/puppet_code and as such we see theses errors. We did not have this issue on with puppet masters as puppet-merge updates the puppetcode directory directly, this has its own issue in that agents could get the wrong file or a partially written file.
We should investigate if we can improve g10k to make it atomic we could also consider removing g10k, this is not currently used however i think if we can keep it it opens up some possibilities for future improvements