Page MenuHomePhabricator
Create Task
Maniphest T211027

puppet (systemd::service) attempts to start manually masked units
Closed, ResolvedPublic
Actions

Description

While provisioning restbase hardware in codfw (T209615) I came across the fact that puppet attempts to start manually masked systemd units (i.e. systemctl mask <unit> on the system, not ensure => masked in puppet). AFAIK this behavior as seen on restbase hardware is new, in the sense that previously (on jessie? the new hardware is stretch) puppet would honor masked status and don't start such units. (the unit cassandra-a eventually fails but for unrelated reasons)

root@restbase2015:~# systemctl mask cassandra-a
Created symlink /etc/systemd/system/cassandra-a.service → /dev/null.
root@restbase2015:~# puppet agent --test
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for restbase2015.codfw.wmnet
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1543853078'
Notice: /Stage[main]/Cassandra/Cassandra::Instance[a]/Systemd::Service[cassandra-a]/Service[cassandra-a]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Cassandra/Cassandra::Instance[a]/Systemd::Service[cassandra-a]/Service[cassandra-a]: Unscheduling refresh on Service[cassandra-a]
Notice: /Stage[main]/Cassandra/Cassandra::Instance[b]/Systemd::Service[cassandra-b]/Service[cassandra-b]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Cassandra/Cassandra::Instance[b]/Systemd::Service[cassandra-b]/Service[cassandra-b]: Unscheduling refresh on Service[cassandra-b]
Notice: /Stage[main]/Cassandra/Cassandra::Instance[c]/Systemd::Service[cassandra-c]/Service[cassandra-c]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Cassandra/Cassandra::Instance[c]/Systemd::Service[cassandra-c]/Service[cassandra-c]: Unscheduling refresh on Service[cassandra-c]
Notice: Applied catalog in 13.10 seconds
root@restbase2015:~# systemctl status cassandra-a
● cassandra-a.service - distributed storage system for structured data
   Loaded: loaded (/lib/systemd/system/cassandra-a.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2018-12-03 16:05:13 UTC; 31s ago
 Main PID: 80047 (code=exited, status=3)

Dec 03 16:04:56 restbase2015 systemd[1]: Started distributed storage system for structured data.
Dec 03 16:05:13 restbase2015 systemd[1]: cassandra-a.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Dec 03 16:05:13 restbase2015 systemd[1]: cassandra-a.service: Unit entered failed state.
Dec 03 16:05:13 restbase2015 systemd[1]: cassandra-a.service: Failed with result 'exit-code'.
root@restbase2015:~#

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Dec 3 2018, 4:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 3 2018, 4:08 PM
fgiunchedi mentioned this in T209615: rack/setup/install restbase201[3-8].codfw.wmnet.Dec 3 2018, 4:35 PM
MoritzMuehlenhoff subscribed.Dec 3 2018, 5:33 PM
fgiunchedi added a comment.Dec 4 2018, 10:43 AM

Looks like this is working as intended for systemd provider (/usr/lib/ruby/vendor_ruby/puppet/provider/service/systemd.rb)

def enable
  self.unmask
  systemctl_change_enable(:enable)
end

def mask
  self.disable
  systemctl_change_enable(:mask)
end

def unmask
  systemctl_change_enable(:unmask)
end

def restartcmd
  [command(:systemctl), "restart", @resource[:name]]
end

def startcmd
  self.unmask
  [command(:systemctl), "start", @resource[:name]]
end

def stopcmd
  [command(:systemctl), "stop", @resource[:name]]
end

def statuscmd
  [command(:systemctl), "is-active", @resource[:name]]
end
fgiunchedi updated the task description. (Show Details)Dec 4 2018, 10:46 AM
Dzahn subscribed.EditedDec 5 2018, 5:20 PM

https://tickets.puppetlabs.com/browse/PUP-1253

https://github.com/puppetlabs/puppet/pull/3141

"If a service is masked, it is deemed to also be disabled. If a service is
masked and changed to enabled, it will first be unmasked since the
standard 'systemctl enable' command does not properly unmask the command
first."

Is this the reason for the change in behaviour? https://github.com/puppetlabs/puppet/pull/4770/files/7a5176c88f261402a8e73d926034e390f189a1d0#r56208804

fgiunchedi renamed this task from puppet (systemd::service) attempts to start masked units to puppet (systemd::service) attempts to start manually masked units.Dec 7 2018, 8:32 AM
fgiunchedi updated the task description. (Show Details)
fgiunchedi added a comment.Dec 7 2018, 8:34 AM
In T211027#4800887, @Dzahn wrote:

https://tickets.puppetlabs.com/browse/PUP-1253

https://github.com/puppetlabs/puppet/pull/3141

"If a service is masked, it is deemed to also be disabled. If a service is
masked and changed to enabled, it will first be unmasked since the
standard 'systemctl enable' command does not properly unmask the command
first."

Is this the reason for the change in behaviour? https://github.com/puppetlabs/puppet/pull/4770/files/7a5176c88f261402a8e73d926034e390f189a1d0#r56208804

I'm not sure! I've clarified the task though to mention this is about manually masked units as opposed to instructing puppet to mask the units. The latter would do the right thing I think, though I can see how it is impractical to change unit status as masked in puppet, as opposed to "just" issue systemctl mask on the host

herron triaged this task as Medium priority.Jan 2 2019, 9:47 PM
herron added a project: Puppet.
fgiunchedi mentioned this in T214166: Improve cassandra JBOD integration post-reimage.Jan 18 2019, 1:50 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:48 PM
Aklapper added a project: Infrastructure-Foundations.Jun 21 2021, 8:59 PM
jbond added a parent task: T294906: Puppet Improvements.Nov 3 2021, 12:02 PM
jbond edited projects, added Puppet-Core; removed Puppet.Jun 12 2023, 5:06 PM
jbond subscribed.

Looks like this is working as intended for systemd provider (/usr/lib/ruby/vendor_ruby/puppet/provider/service/systemd.rb)

although if you run systemctl restart $service manually on a masked service you get Failed to restart apache2.service: Unit apache2.service is masked. so puppet must be doing something extra. this is a bit strange.

LSobanski closed this task as Resolved.Mon, Apr 27, 2:42 PM
LSobanski claimed this task.
LSobanski subscribed.

This should no longer be an issue with a systemd define in place. Please reopen if needed.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits