Page MenuHomePhabricator

citoid dead on sca1001
Closed, ResolvedPublic0 Estimated Story Points

Description

Is dead. doesn't come up on restart, has errors in logs.

Event Timeline

yuvipanda raised the priority of this task from to High.
yuvipanda updated the task description. (Show Details)
yuvipanda added a project: Citoid.
yuvipanda added subscribers: yuvipanda, mobrovac.

The error comes from the fact that the current version checked out on the server is not up-to-date, thus having problems loading the config.

However, we are running into issues when deploying the changes. Concretely, the checkout stage fails because trebuchet is not able to find the submodule's newest ref (which is present in a normal check-out), as seen in P402 . Citoid's deployment config (in role::deployment) seems to be alright, the contents in /srv/deployment/citoid/deploy on tin as well.

Change 197114 had a related patch set uploaded (by Mobrovac):
Update Citoid dependencies

https://gerrit.wikimedia.org/r/197114

Change 197114 merged by Mobrovac:
Update Citoid dependencies

https://gerrit.wikimedia.org/r/197114

So after some investigation it became obvious the trebuchet could not update the git submodule for citoid. The error was

output: fatal: reference is not a tree: 2cd246b99fd2f407b92c33a079a1120f0abe7117

The repository on tin had the commit but would not advertise it. The solution was to

tin:/srv/deployment/citoid/deploy/src$ git update-server-info

The good news is that on very first failed restart on a host, trebuchet refused correctly to restart the service on the other one so the service had 0 downtime (hence my marking of the task as not blocking anything)

The problem has now changed into

Error: Cannot find module 'bunyan'

which is being look into by @mobrovac

mobrovac claimed this task.

The problem has now changed into

Error: Cannot find module 'bunyan'

which is being look into by @mobrovac

This was due to unmet node dependencies. After updating them and restarting, the service is alive and kicking on both hosts, thus closing.