I think that we have various possible improvements to avoid what happen in your related task:

1. Puppet CI could catch that the role assigned in site.pp doesn' exists

2. The reimage cookbook could check that a host has a role applied in site.pp before starting (but in this case it would have not caught the issue as it was assigned)

3. The reimage cookbook could try to be smarter to report a complete puppet failure (already tracked in T338990 )

4. The install_console command should be common knowledge and part of everyone's onboarding.

Thanks for the links, very helpful. I'll file a ticket for 1.

As for 4, I think we need to be careful about what we expect out of the user. If we have a WMF-specific command or process that is very important for the troubleshooting process, it should be called out as often as possible. Think of it like Linux rescue mode's Press CTRL-S to drop into a shell.

Change 956082 had a related patch set uploaded (by Bking; author: Bking):

[operations/cookbooks@master] sre.hosts.reimage: Suggest install-console for troubleshooting

https://gerrit.wikimedia.org/r/956082

@Volans ACK, working on it as a Friday project now. Sorry for the delay.

Change 956082 merged by Bking:

[operations/cookbooks@master] sre.hosts.reimage: Suggest install-console for troubleshooting

https://gerrit.wikimedia.org/r/956082