Page MenuHomePhabricator

restbase is not listening on port 7231 on aqs*
Closed, ResolvedPublic

Description

aqs100* are setup and ready and restbase is started up on them but it is not listening on the TCP port it advertises itself as listening to:

akosiaris@aqs1001:~$ sudo grep restbase /var/log/syslog
Oct  6 09:03:20 aqs1001 restbase[9019]: Started RESTBase server, by default listening on port 7231
akosiaris@aqs1001:~$ sudo lsof -i -n -P |grep 7231
akosiaris@aqs1001:~$

Also

sudo systemctl status restbase
* restbase.service - LSB: REST storage API and backend orchestration layer
   Loaded: loaded (/etc/init.d/restbase)
   Active: active (running) since Tue 2015-10-06 09:03:25 UTC; 9min ago
  Process: 8970 ExecStop=/etc/init.d/restbase stop (code=exited, status=0/SUCCESS)
  Process: 9019 ExecStart=/etc/init.d/restbase start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/restbase.service
           |-9025 /bin/sh -c /usr/bin/nodejs /usr/lib/restbase/deploy/restbase/server.js -c /etc/restbase/config.yaml 2>&1 | logger -i -t restbase
           |-9027 /usr/bin/nodejs /usr/lib/restbase/deploy/restbase/server.js -c /etc/restbase/config.yaml
           `-9028 logger -i -t restbase

Oct 06 09:03:20 aqs1001 restbase[9019]: Started RESTBase server, by default listening on port 7231

Event Timeline

akosiaris created this task.Oct 6 2015, 9:14 AM
akosiaris raised the priority of this task from to High.
akosiaris updated the task description. (Show Details)
akosiaris added projects: Services, Analytics.
akosiaris added a subscriber: akosiaris.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 6 2015, 9:14 AM

Yeah, it is busted. I was trying to deploy Friday but failed. Analytics and Services folks are going to have to look into it. There is some issue where restbase isn't starting up properly in systemd.

Change 243955 had a related patch set uploaded (by Mobrovac):
AQS RESTBase: Use full path to the server module script

https://gerrit.wikimedia.org/r/243955

mobrovac claimed this task.Oct 6 2015, 5:01 PM

Change 243955 merged by Ottomata:
AQS RESTBase: Use full path to the server module script

https://gerrit.wikimedia.org/r/243955

The patch fixes the issue - after running Puppet and restarting RESTBase on aqs1001 the service is up and running. @Ottomata, please apply the change to the other two nodes and restart RESTBase there as well so we can close this.

mobrovac set Security to None.
mobrovac removed a subscriber: gerritbot.
mobrovac closed this task as Resolved.Oct 7 2015, 2:17 PM

AQS is now up && running, resolving.

Dzahn reopened this task as Open.Oct 14 2015, 2:27 PM
Dzahn added a subscriber: Dzahn.

We still have 3 CRITs in Icinga for "Restbase endpoints health" on aqs and there was a comment next to them linking to this ticket.

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=endpoints

mobrovac closed this task as Resolved.Oct 15 2015, 2:05 PM

We still have 3 CRITs in Icinga for "Restbase endpoints health" on aqs and there was a comment next to them linking to this ticket.
https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=endpoints

This is a different issue. Opened T115588: Automatic monitoring not working for AQS to track it, resolving this one.