zuul-merger git-daemon process is not started properly by systemd ?
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Feb 10 2017, 9:47 AM

Description

When deploying the zuul-merger and git-daemon services on contint1001 and contint2001 ( https://gerrit.wikimedia.org/r/336807 ), the zuul-merger service spawned properly (it has a systemd unit) but the git-daemon one did not.

When I logged on one of the machine the state was:

# systemctl status git-daemon
● git-daemon.service - LSB: git-daemon service
   Loaded: loaded (/etc/init.d/git-daemon)
   Active: active (exited) since Fri 2017-02-10 03:23:56 UTC; 6h ago

Feb 10 03:23:56 contint1001 systemd[1]: Started LSB: git-daemon service.

git-daemon status.png (102×577 px, 25 KB)

But there were no service running. The status command exited 0 and hence puppet would not try to start it.

I had to first stop it which has lead to status exiting non zero:

# systemctl stop git-daemon
# systemctl status git-daemon; echo $?
● git-daemon.service - LSB: git-daemon service
   Loaded: loaded (/etc/init.d/git-daemon)
   Active: inactive (dead) since Fri 2017-02-10 09:42:48 UTC; 19s ago
  Process: 140970 ExecStop=/etc/init.d/git-daemon stop (code=exited, status=0/SUCCESS)

Feb 10 03:23:56 contint1001 systemd[1]: Started LSB: git-daemon service.
Feb 10 09:42:29 contint1001 systemd[1]: Started LSB: git-daemon service.
Feb 10 09:42:48 contint1001 systemd[1]: Stopping LSB: git-daemon service...
Feb 10 09:42:48 contint1001 systemd[1]: Stopped LSB: git-daemon service.
3

Then start it and it worked just fine this time:

# systemctl status git-daemon; echo $?
● git-daemon.service - LSB: git-daemon service
   Loaded: loaded (/etc/init.d/git-daemon)
   Active: active (running) since Fri 2017-02-10 09:43:22 UTC; 2s ago
  Process: 140970 ExecStop=/etc/init.d/git-daemon stop (code=exited, status=0/SUCCESS)
  Process: 141186 ExecStart=/etc/init.d/git-daemon start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/git-daemon.service
           └─141191 /usr/lib/git-core/git-daemon --user=gitdaemon --pid-file=/var/run/git-daemon.pid --detach --reuseaddr --verbose --export-all --forbid-override=receiv...

Feb 10 09:43:22 contint1001 systemd[1]: Started LSB: git-daemon service.
Feb 10 09:43:22 contint1001 git-daemon[141191]: Ready to rumble
0

git-daemon-status-OK.png (209×635 px, 49 KB)

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
Zuul: Make sure git-daemon starts after installing it	operations/puppet	production	+3 -0
contint: migrate git-daemon to systemd	operations/puppet	production	+43 -35
contint: git-daemon service is 'sysvinit'	operations/puppet	production	+1 -0

Customize query in gerrit

Related Objects

Mentioned In: rEWLE399b38512a4b: Update patch set 2
T150936: Phase out scandium.eqiad.wmnet
Mentioned Here: T150936: Phase out scandium.eqiad.wmnet

Event Timeline

hashar created this task.Feb 10 2017, 9:47 AM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptFeb 10 2017, 9:47 AM

hashar updated the task description. (Show Details)Feb 10 2017, 9:50 AM

Mentioned in SAL (#wikimedia-operations) [2017-02-10T09:51:53Z] <hashar> Reenabling puppet and zuul-merger on contint1001 and contint2001. The git-daemon is running now T140297 T150936. The 'systemctl status git-daemon' thought that the service was running when it was not (filled T157785 )

The puppet service definition at https://github.com/wikimedia/operations-puppet/blob/cfbdecb/modules/contint/manifests/zuul/git_daemon.pp#L25-L31 :

service { 'git-daemon':
  ensure     => running,
  enable     => true,
  hasrestart => true,
  subscribe  => File['/etc/default/git-daemon'],
  require    => Package['git-daemon-sysvinit'],
}

Change 336978 had a related patch set uploaded (by Hashar):
contint: git-daemon service is 'sysvinit'

https://gerrit.wikimedia.org/r/336978

gerritbot added a project: Patch-For-Review.Feb 10 2017, 10:06 AM

hashar removed a parent task: T150936: Phase out scandium.eqiad.wmnet.Feb 13 2017, 8:10 PM

Change 336978 merged by Jcrespo:
contint: git-daemon service is 'sysvinit'

https://gerrit.wikimedia.org/r/336978

Patch failed with:

Error: Failed to apply catalog: Parameter provider failed on Service[git-daemon]: Invalid service provider 'sysvinit' at /etc/puppet/modules/contint/manifests/zuul/git_daemon.pp:32

@hashar sysvinit is not a valid provider in service puppet code, see https://docs.puppet.com/puppet/latest/types/service.html#service-attribute-provider and https://docs.puppet.com/puppet/latest/types/service.html#service-providers please.

heres a systemd file for git-daemon http://pkgs.fedoraproject.org/cgit/rpms/git.git/commit/?id=906d8479f79ee89ec399fa5988b11fa037f18d38 and https://gist.github.com/zinovyev/5828aa70db3ba96169ddabd5d8bdd9ed

hashar renamed this task from zuul-merger git-daemon process is not start properly by systemd ? to zuul-merger git-daemon process is not started properly by systemd ?.Feb 17 2017, 9:32 AM

Change 340496 had a related patch set uploaded (by Paladox):
[operations/puppet] Zuul: Make sure git-daemon starts after installing it

https://gerrit.wikimedia.org/r/340496

From my comment on the hack https://gerrit.wikimedia.org/r/#/c/340496/

The git-daemon-sysvinit package is just some sysvinit boiler plate around /usr/lib/git-core/git-daemon:

$ dpkg -L git-daemon-sysvinit
/etc/default/git-daemon
/etc/init.d/git-daemon
/usr/share/doc/git-daemon-sysvinit/copyright
/usr/share/doc/git-daemon-sysvinit/changelog.gz
/usr/share/doc/git-daemon-sysvinit/README.Debian
/usr/share/doc/git-daemon-sysvinit/changelog.Debian.gz
/usr/share/git-core/sysvinit/sentinel

The actual daemon /usr/lib/git-core/git-daemon is provided by the git package. So instead we should remove git-daemon-sysvinit and provide a systemd service / base::service_unit . That will be more resilient.

Change 342128 had a related patch set uploaded (by Hashar):
[operations/puppet] (WIP) contint: migrate git-daemon to systemd

https://gerrit.wikimedia.org/r/342128

Change 342128 merged by Muehlenhoff:
[operations/puppet] contint: migrate git-daemon to systemd

https://gerrit.wikimedia.org/r/342128

Solved. We no more use git-daemon-sysvinit and now run the daemon using systemd. I did the migration of the production machines (had to manually remove the package).

Change 340496 abandoned by Paladox:
Zuul: Make sure git-daemon starts after installing it