zuul-merger git-daemon process is not started properly by systemd ?
Closed, ResolvedPublic

Description

When deploying the zuul-merger and git-daemon services on contint1001 and contint2001 ( https://gerrit.wikimedia.org/r/336807 ), the zuul-merger service spawned properly (it has a systemd unit) but the git-daemon one did not.

When I logged on one of the machine the state was:

# systemctl status git-daemon
● git-daemon.service - LSB: git-daemon service
   Loaded: loaded (/etc/init.d/git-daemon)
   Active: active (exited) since Fri 2017-02-10 03:23:56 UTC; 6h ago

Feb 10 03:23:56 contint1001 systemd[1]: Started LSB: git-daemon service.

But there were no service running. The status command exited 0 and hence puppet would not try to start it.

I had to first stop it which has lead to status exiting non zero:

# systemctl stop git-daemon
# systemctl status git-daemon; echo $?
● git-daemon.service - LSB: git-daemon service
   Loaded: loaded (/etc/init.d/git-daemon)
   Active: inactive (dead) since Fri 2017-02-10 09:42:48 UTC; 19s ago
  Process: 140970 ExecStop=/etc/init.d/git-daemon stop (code=exited, status=0/SUCCESS)

Feb 10 03:23:56 contint1001 systemd[1]: Started LSB: git-daemon service.
Feb 10 09:42:29 contint1001 systemd[1]: Started LSB: git-daemon service.
Feb 10 09:42:48 contint1001 systemd[1]: Stopping LSB: git-daemon service...
Feb 10 09:42:48 contint1001 systemd[1]: Stopped LSB: git-daemon service.
3

Then start it and it worked just fine this time:

# systemctl status git-daemon; echo $?
● git-daemon.service - LSB: git-daemon service
   Loaded: loaded (/etc/init.d/git-daemon)
   Active: active (running) since Fri 2017-02-10 09:43:22 UTC; 2s ago
  Process: 140970 ExecStop=/etc/init.d/git-daemon stop (code=exited, status=0/SUCCESS)
  Process: 141186 ExecStart=/etc/init.d/git-daemon start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/git-daemon.service
           └─141191 /usr/lib/git-core/git-daemon --user=gitdaemon --pid-file=/var/run/git-daemon.pid --detach --reuseaddr --verbose --export-all --forbid-override=receiv...

Feb 10 09:43:22 contint1001 systemd[1]: Started LSB: git-daemon service.
Feb 10 09:43:22 contint1001 git-daemon[141191]: Ready to rumble
0

hashar created this task.Feb 10 2017, 9:47 AM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptFeb 10 2017, 9:47 AM
hashar edited the task description. (Show Details)Feb 10 2017, 9:50 AM
Stashbot added a subscriber: Stashbot.

Mentioned in SAL (#wikimedia-operations) [2017-02-10T09:51:53Z] <hashar> Reenabling puppet and zuul-merger on contint1001 and contint2001. The git-daemon is running now T140297 T150936. The 'systemctl status git-daemon' thought that the service was running when it was not (filled T157785 )

The puppet service definition at https://github.com/wikimedia/operations-puppet/blob/cfbdecb/modules/contint/manifests/zuul/git_daemon.pp#L25-L31 :

service { 'git-daemon':
  ensure     => running,
  enable     => true,
  hasrestart => true,
  subscribe  => File['/etc/default/git-daemon'],
  require    => Package['git-daemon-sysvinit'],
}

Change 336978 had a related patch set uploaded (by Hashar):
contint: git-daemon service is 'sysvinit'

https://gerrit.wikimedia.org/r/336978

Change 336978 merged by Jcrespo:
contint: git-daemon service is 'sysvinit'

https://gerrit.wikimedia.org/r/336978

Patch failed with:

Error: Failed to apply catalog: Parameter provider failed on Service[git-daemon]: Invalid service provider 'sysvinit' at /etc/puppet/modules/contint/manifests/zuul/git_daemon.pp:32
hashar changed the title from "zuul-merger git-daemon process is not start properly by systemd ?" to "zuul-merger git-daemon process is not started properly by systemd ?".Feb 17 2017, 9:32 AM

Change 340496 had a related patch set uploaded (by Paladox):
[operations/puppet] Zuul: Make sure git-daemon starts after installing it

https://gerrit.wikimedia.org/r/340496

hashar added a comment.Mar 9 2017, 9:24 AM

From my comment on the hack https://gerrit.wikimedia.org/r/#/c/340496/

The git-daemon-sysvinit package is just some sysvinit boiler plate around /usr/lib/git-core/git-daemon:

$ dpkg -L git-daemon-sysvinit
/etc/default/git-daemon
/etc/init.d/git-daemon
/usr/share/doc/git-daemon-sysvinit/copyright
/usr/share/doc/git-daemon-sysvinit/changelog.gz
/usr/share/doc/git-daemon-sysvinit/README.Debian
/usr/share/doc/git-daemon-sysvinit/changelog.Debian.gz
/usr/share/git-core/sysvinit/sentinel

The actual daemon /usr/lib/git-core/git-daemon is provided by the git package. So instead we should remove git-daemon-sysvinit and provide a systemd service / base::service_unit . That will be more resilient.

Change 342128 had a related patch set uploaded (by Hashar):
[operations/puppet] (WIP) contint: migrate git-daemon to systemd

https://gerrit.wikimedia.org/r/342128

Change 342128 merged by Muehlenhoff:
[operations/puppet] contint: migrate git-daemon to systemd

https://gerrit.wikimedia.org/r/342128

hashar closed this task as "Resolved".Mar 10 2017, 4:26 PM
hashar claimed this task.

Solved. We no more use git-daemon-sysvinit and now run the daemon using systemd. I did the migration of the production machines (had to manually remove the package).

Change 340496 abandoned by Paladox:
Zuul: Make sure git-daemon starts after installing it

https://gerrit.wikimedia.org/r/340496