Page MenuHomePhabricator

"systemctl restart zuul" does not gracefully restart Zuul scheduler
Open, Needs TriagePublic

Description

On contint2001 I did a systemctl restart zuul which I would expect to gracefully stop Zuul by issuing SIGUSR1 and start it once it is terminated:

/lib/systemd/system/zuul.service
[Service]
ExecStart=/srv/deployment/zuul/venv/bin/zuul-server -d -c /etc/zuul/zuul-server.conf
ExecStop=/bin/kill -USR1 $MAINPID
TimeoutStopSec=infinity

And of course systemd terminated it instead. From journalctl:

Jul 03 12:40:57 contint2001 systemd[1]: Stopping Zuul Server...
Jul 03 12:40:57 contint2001 systemd[1]: zuul.service: Succeeded.
Jul 03 12:40:57 contint2001 systemd[1]: Stopped Zuul Server.
Jul 03 12:40:57 contint2001 systemd[1]: Started Zuul Server.

The service is marked as active since 12:40:57

In the Zuul debug.log:

2020-07-03 12:40:57,866 DEBUG zuul.Scheduler: Run handler sleeping
2020-07-03 12:40:57,874 DEBUG zuul.Scheduler: Prepare to exit
2020-07-03 12:40:57,874 DEBUG zuul.Scheduler: Waiting for exit
2020-07-03 12:41:09,904 DEBUG zuul.MergeClient: Connecting to gearman at 127.0.0.1:4730
2020-07-03 12:41:09,906 DEBUG zuul.MergeClient: Waiting for gearman
2020-07-03 12:41:09,914 INFO zuul.Server: Starting scheduler

Looks like restart does not use ExecStop :/

Event Timeline

hashar created this task.Jul 3 2020, 12:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 3 2020, 12:51 PM
hashar added a comment.Jul 3 2020, 1:13 PM

The SIGUSR1 did get emitted that is what causes the Scheduler to emit Prepare to exit and Waiting for exit. The issue is that after commands in ExecStop have been executed, systemd immediately kills all processes in the control group. So the kill SIGUSR1 has to be synchronously for the main pid to exit :\