Page MenuHomePhabricator

restbase is not listening on port 7231 on aqs*
Closed, ResolvedPublic

Description

aqs100* are setup and ready and restbase is started up on them but it is not listening on the TCP port it advertises itself as listening to:

akosiaris@aqs1001:~$ sudo grep restbase /var/log/syslog
Oct  6 09:03:20 aqs1001 restbase[9019]: Started RESTBase server, by default listening on port 7231
akosiaris@aqs1001:~$ sudo lsof -i -n -P |grep 7231
akosiaris@aqs1001:~$

Also

sudo systemctl status restbase
* restbase.service - LSB: REST storage API and backend orchestration layer
   Loaded: loaded (/etc/init.d/restbase)
   Active: active (running) since Tue 2015-10-06 09:03:25 UTC; 9min ago
  Process: 8970 ExecStop=/etc/init.d/restbase stop (code=exited, status=0/SUCCESS)
  Process: 9019 ExecStart=/etc/init.d/restbase start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/restbase.service
           |-9025 /bin/sh -c /usr/bin/nodejs /usr/lib/restbase/deploy/restbase/server.js -c /etc/restbase/config.yaml 2>&1 | logger -i -t restbase
           |-9027 /usr/bin/nodejs /usr/lib/restbase/deploy/restbase/server.js -c /etc/restbase/config.yaml
           `-9028 logger -i -t restbase

Oct 06 09:03:20 aqs1001 restbase[9019]: Started RESTBase server, by default listening on port 7231

Event Timeline

akosiaris raised the priority of this task from to High.
akosiaris updated the task description. (Show Details)
akosiaris added projects: Services, Analytics.
akosiaris subscribed.

Yeah, it is busted. I was trying to deploy Friday but failed. Analytics and Services folks are going to have to look into it. There is some issue where restbase isn't starting up properly in systemd.

Change 243955 had a related patch set uploaded (by Mobrovac):
AQS RESTBase: Use full path to the server module script

https://gerrit.wikimedia.org/r/243955

Change 243955 merged by Ottomata:
AQS RESTBase: Use full path to the server module script

https://gerrit.wikimedia.org/r/243955

The patch fixes the issue - after running Puppet and restarting RESTBase on aqs1001 the service is up and running. @Ottomata, please apply the change to the other two nodes and restart RESTBase there as well so we can close this.

AQS is now up && running, resolving.

Dzahn subscribed.

We still have 3 CRITs in Icinga for "Restbase endpoints health" on aqs and there was a comment next to them linking to this ticket.

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=endpoints

We still have 3 CRITs in Icinga for "Restbase endpoints health" on aqs and there was a comment next to them linking to this ticket.

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=endpoints

This is a different issue. Opened T115588: Automatic monitoring not working for AQS to track it, resolving this one.