user story:
Recently passive failover phab host phab2002 had to be reimaged (T377374) to move it to a new VLAN.
So everything stayed the same as before and the same production role(phabricator) was applied before and after.
The reimaging cookbook we use expects that initial puppet runs on reimaged systems just work and if they don't the entire
cookbook ends with a failure message.
This is fine for a lot of services but it's often an issue for any services that are deployed by scap.
The procedure needed here is usually more like:
- run puppet once to trigger a command that bootstraps scap itself
- if this fails (T257317, T257317#10212656) ) run the scap bootstrap command manually, (scap deploy --init ,T363415#9762416 , T257317#9851645)
- run puppet again (sometimes run into a dependency issue which this ticket is about in specific)
- deploy using scap
- run puppet again
..or. trying to avoid these:
- have a [[ https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/phabricator/migration.pp | "migration" class ]]which is applied first to new hosts, before the actual production class is applied.
The latter won't work for reimages of existing hosts though unless we first remove the production role, add the migration role, reimage and then revert the process.
One part of this is a puppet / scap dependency issue, which manifests like this:
Oct 16 19:14:56 phab2002 puppet-agent[3358]: (/Stage[main]/Phabricator::Config/Scap::Target[phabricator/deployment]/Package[phabricator/deployment]) Provider scap3 is not functional on this host Oct 16 19:14:56 phab2002 puppet-agent[3358]: (/Stage[main]/Phabricator::Config/File[/srv/phab]) Dependency Package[phabricator/deployment] has failures: true Oct 16 19:14:56 phab2002 puppet-agent[3358]: (/Stage[main]/Phabricator::Config/File[/srv/phab]) Skipping because of failed dependencies Oct 16 19:14:56 phab2002 puppet-agent[3358]: (/Stage[main]/Phabricator::Config/File[/srv/phab/phabricator/scripts/]) Skipping because of failed dependencies Oct 16 19:14:56 phab2002 puppet-agent[3358]: (/Stage[main]/Phabricator::Config/File[/srv/phab/phabricator/scripts/mail/]) Skipping because of failed dependencies Oct 16 19:14:56 phab2002 puppet-agent[3358]: (Class[Phabricator::Config]) Unscheduling all events on Class[Phabricator::Config] Oct 16 19:14:56 phab2002 puppet-agent[3358]: (/Stage[main]/Phabricator/File[/usr/local/bin/arc]) Skipping because of failed dependencies Oct 16 19:14:56 phab2002 puppet-agent[3358]: (/Stage[main]/Phabricator::Vcs/File[/usr/local/bin/git-http-backend]) Skipping because of failed dependencies Oct 16 19:14:56 phab2002 puppet-agent[3358]: (/Stage[main]/Phabricator::Vcs/File[/usr/libexec]) Skipping because of failed dependencies Oct 16 19:14:56 phab2002 puppet-agent[3358]: (/Stage[main]/Phabricator::Vcs/File[/usr/libexec/phabricator-ssh-hook.sh]) Skipping because of failed dependencies Oct 16 19:14:56 phab2002 puppet-agent[3358]: (/Stage[main]/Phabricator::Tools/Package[python3-mysqldb]) Skipping because of failed dependencies Oct 16 19:14:56 phab2002 puppet-agent[3358]: (/Stage[main]/Phabricator::Tools/Package[python3-pymysql]) Skipping because of failed dependencies Oct 16 19:14:56 phab2002 puppet-agent[3358]: (/Stage[main]/Phabricator::Tools/File[/etc/phabtools.conf]) Skipping because of failed dependencies Oct 16 19:14:56 phab2002 puppet-agent[3358]: (/Stage[main]/Phabricator::Tools/File[/srv/dumps]) Skipping because of failed dependencies .. followed by many many other things all failing because of failed dependencies
As discussed with @jnuche on IRC "that is triggered by phabricator::config when it tries to init the phabricator/deployment` repo.
and at that point puppet should have installed scap to be able to work correctly, but it hasn't yet".
We have tracked down that puppet tries to install scap from the scap::target class and "we should be able to solve the order problem and make sure scap gets installed before the stuff in phabricator::config runs".