Today, while upgrading mtail across all cache hosts, we were surprised to find that systemd attempted to start mtail.service. On cache hosts, mtail.service cannot start properly as we already run a mtail instance started by varnishmtail.service binding on port 3903 (mtail's default). We do explicitly instruct puppet to ensure that mtail.service is stopped, but that is not enough to prevent package upgrades from taking initiatives and starting services up.
More in general, there are 4 events that I can think of which trigger service startups:
- root running systemctl start $service
- systemd starting an enabled service at boot time
- puppet saying ensure => running
- debian package installation / upgrades
There are probably more possible triggers (socket activation and who knows what else), but I do not think they're particularly relevant here.
In the specific case of mtail above, @MoritzMuehlenhoff suggested masking the unit with systemctl mask. That has the effect of stopping all 4 triggers above, and would work fine with mtail. However, I've started wondering about other cases, such as for example pybal.service. In pybal's case, we do want (1) and (2), while we explicitly avoid (3). At the moment, we have taken no measures against (4), although I think it would be desirable to do so (we do not necessarily want to immediately switch traffic back to a primary LVS server upon pybal upgrade).
I'm opening this task to collect thoughts on the subject and as a tracking task for work going in the direction of stopping unwanted service startups initiated (4).