Page MenuHomePhabricator

restbase - some nodes missing systemd unit for service
Closed, ResolvedPublic

Description

on restbase1004 and restbase1005 puppet fails because it can't start restbase service. on restbase1003 it works though. i could confirm /etc/init.d/restbase is identical on nodes and also git status in /usr/lib/restbase/deploy/restbase seems to be identical.

it's trying to use systemctl but see this difference:

@restbase1003:/usr/lib/restbase# systemctl list-units --type=service | grep restbase.service
restbase.service                                              loaded active exited  LSB: REST storage API and backend orchestration layer


@restbase1005:/etc/systemd/system# systemctl list-units --type=service | grep restbase.service
@restbase1005:/etc/systemd/system#

makes you think it should be in /etc/systemd/restbase/restbase.conf but i don't see that on 1003 either.

Event Timeline

Dzahn created this task.Feb 19 2015, 1:49 AM
Dzahn raised the priority of this task from to Needs Triage.
Dzahn updated the task description. (Show Details)
Dzahn added a subscriber: Dzahn.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 19 2015, 1:49 AM
Dzahn added a comment.EditedFeb 19 2015, 1:55 AM
root@restbase1005:/etc/systemd# systemctl status restbase
● restbase.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)
root@restbase1003:/etc/systemd# systemctl status restbase
● restbase.service - LSB: REST storage API and backend orchestration layer
   Loaded: loaded (/etc/init.d/restbase)
   Active: active (exited) since Wed 2015-02-18 02:36:15 UTC; 23h ago
  Process: 61392 ExecStop=/etc/init.d/restbase stop (code=exited, status=0/SUCCESS)
  Process: 61392 ExecStop=/etc/init.d/restbase stop (code=exited, status=0/SUCCESS)
  Process: 61400 ExecStart=/etc/init.d/restbase start (code=exited, status=0/SUCCESS)

Feb 18 02:36:10 restbase1003 restbase[61400]: Started RESTBase server, by default listening on port 7231
Feb 18 02:36:10 restbase1003 restbase[61409]: 
Feb 18 02:36:10 restbase1003 restbase[61409]: module.js:340
Feb 18 02:36:10 restbase1003 restbase[61409]: throw err;
Feb 18 02:36:10 restbase1003 restbase[61409]: ^
Feb 18 02:36:10 restbase1003 restbase[61409]: Error: Cannot find module '/usr/lib/restbase/deploy/restbase/server.js'
Feb 18 02:36:10 restbase1003 restbase[61409]: at Function.Module._resolveFilename (module.js:338:15)
Feb 18 02:36:10 restbase1003 restbase[61409]: at Function.Module._load (module.js:280:25)
Feb 18 02:36:10 restbase1003 restbase[61409]: at Function.Module.runMain (module.js:497:10)
Dzahn updated the task description. (Show Details)Feb 19 2015, 1:56 AM
Dzahn set Security to None.
Dzahn updated the task description. (Show Details)
Dzahn added a comment.Feb 19 2015, 2:06 AM
root@restbase1003:/run/systemd/generator.late# head -n1 restbase.service 
# Automatically generated by systemd-sysv-generator
root@restbase1005:/run/systemd/generator.late# head -n1 restbase.service
head: cannot open ‘restbase.service’ for reading: No such file or directory

so the generator did not generate it? systemctl daemon-reload should run systemd-sysv-generator, but that didn't make it appear either. in fact, generator.late is completely empty on 1005 but has a whole bunch of files on 1003

Dzahn added a comment.Feb 19 2015, 2:19 AM

Fixed it. Just needed some patience after running systemctl daemon-reload. It did in fact recreate the files and the file for restbase appeared. After that i could start the service and:

< icinga-wm> RECOVERY - puppet last run on restbase1005 is OK: OK:
< icinga-wm> RECOVERY - puppet last run on restbase1004 is OK:

18:11 < mutante> !log restbase1004/1005 systemctl daemon-reload to run systemd-sysv-generator to make it create missing unit for restbase and unbreak puppet running the service

18:18 < mutante> !log restbase1004 - starting restbase service, running puppet

btw, maybe we should monitor the actual service running not just puppet run

Dzahn closed this task as Resolved.Feb 19 2015, 2:20 AM
Dzahn claimed this task.

Thanks a lot for investigating & fixing this, @Dzahn!

Dzahn added a subscriber: Joe.Feb 19 2015, 6:07 PM