While working on T366555 I noticed burrow doesn't start at boot but only at the first puppet run:
Jun 4 09:14:09 kafkamon1003 puppet-agent[1160]: Applying configuration version '(f56f683d75) Muehlenhoff - gerrit: move java_home hiera setting to role' Jun 4 09:14:23 kafkamon1003 puppet-agent[1160]: (/Stage[main]/Profile::Kafka::Monitoring/Profile::Kafka::Burrow[main-eqiad]/Burrow[main-eqiad]/File[/var/run/burrow]/ensure) created (corrective) Jun 4 09:14:24 kafkamon1003 puppet-agent[1160]: (/Stage[main]/Profile::Kafka::Monitoring/Profile::Kafka::Burrow[main-eqiad]/Burrow[main-eqiad]/Systemd::Service[burrow-main-eqiad]/Service[burrow-main-eqiad]/ensure) ensure changed 'stopped' to 'running' (corrective) Jun 4 09:14:24 kafkamon1003 puppet-agent[1160]: (/Stage[main]/Profile::Kafka::Monitoring/Profile::Kafka::Burrow[main-eqiad]/Burrow[main-eqiad]/Systemd::Service[burrow-main-eqiad]/Service[burrow-main-eqiad]) Unscheduling refresh on Service[burrow-main-eqiad] Jun 4 09:14:25 kafkamon1003 puppet-agent[1160]: (/Stage[main]/Profile::Kafka::Monitoring/Profile::Kafka::Burrow[jumbo-eqiad]/Burrow[jumbo-eqiad]/Systemd::Service[burrow-jumbo-eqiad]/Service[burrow-jumbo-eqiad]/ensure) ensure changed 'stopped' to 'running' (corrective) Jun 4 09:14:25 kafkamon1003 puppet-agent[1160]: (/Stage[main]/Profile::Kafka::Monitoring/Profile::Kafka::Burrow[jumbo-eqiad]/Burrow[jumbo-eqiad]/Systemd::Service[burrow-jumbo-eqiad]/Service[burrow-jumbo-eqiad]) Unscheduling refresh on Service[burrow-jumbo-eqiad] Jun 4 09:14:25 kafkamon1003 puppet-agent[1160]: (/Stage[main]/Profile::Kafka::Monitoring/Profile::Kafka::Burrow[logging-eqiad]/Burrow[logging-eqiad]/Systemd::Service[burrow-logging-eqiad]/Service[burrow-logging-eqiad]/ensure) ensure changed 'stopped' to 'running' (corrective) Jun 4 09:14:25 kafkamon1003 puppet-agent[1160]: (/Stage[main]/Profile::Kafka::Monitoring/Profile::Kafka::Burrow[logging-eqiad]/Burrow[logging-eqiad]/Systemd::Service[burrow-logging-eqiad]/Service[burrow-logging-eqiad]) Unscheduling refresh on Service[burrow-logging-eqiad]
The solution is to enable the relative systemd units and make sure systemd-tmpfiles or the systemd unit itself creates/manages the /var/run directory, not puppet