Page MenuHomePhabricator

Parsoid refuses to start on beta cluster
Closed, ResolvedPublic

Description

We noticed the Parsoid service on Beta-Cluster-Infrastructure deployment-parsoid05 refuses to start.

The Jenkins job apparently manage to install all the dependencies and does start the service but it dies right away:

From /data/project/parsoid/parsoid.log:

module.js:340
    throw err;
          ^
Error: Cannot find module '/srv/deployment/parsoid/parsoid/api/server.js'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:902:3

There is no /api/ directory available.


Related configuration files that are provided by puppet:

/etc/init/parsoid.conf:

# vim: set ft=upstart:

#####################################################################
### THIS FILE IS MANAGED BY PUPPET
### puppet:///files/misc/parsoid.upstart
#####################################################################

description "Parsoid HTTP service"

start on (local-filesystem and net-device-up IFACE!=lo)
stop on runlevel [!2345]

# up ulimit -n a bit
limit nofile 10000 10000

setuid "parsoid"
setgid "parsoid"

env DEFAULTFILE=/etc/default/parsoid

# Basic built-in defaults. Overridden by whatever
# is defined in the DEFAULTFILE defined above.
env VCAP_APP_PORT="8000"
env NODE_PATH="/var/lib/parsoid/deploy/node_modules"
env PARSOID_BASE_PATH="/var/lib/parsoid/deploy/src"
env PARSOID_LOG_FILE="/dev/null"
env PARSOID_SETTINGS_FILE="../conf/wmf/localsettings.js"

respawn

# wait 60 seconds for a graceful restart before killing the master
kill timeout 60

script
    if [ -f "$DEFAULTFILE" ] ; then
        . "$DEFAULTFILE"
    fi
    DAEMON_ARGS="-c $PARSOID_SETTINGS_FILE"
    chdir "$PARSOID_BASE_PATH"
    exec /usr/bin/nodejs api/server.js $DAEMON_ARGS < /dev/null >> "$PARSOID_LOG_FILE" 2>&1
end script

/etc/default/parsoid:

#####################################################################
### THIS FILE IS MANAGED BY PUPPET
### puppet:///templates/misc/parsoid.default
#####################################################################

# File where the parsoid daemon will write stdin and stdout to
PARSOID_LOG_FILE=/data/project/parsoid/parsoid.log

# Checkout of mediawiki/services/parsoid

PARSOID_BASE_PATH=/srv/deployment/parsoid/parsoid


# Checkout of mediawiki/services/parsoid/deploy

NODE_PATH=/srv/deployment/parsoid/parsoid/node_modules


# Checkout of mediawiki/services/parsoid/deploy

PARSOID_SETTINGS_FILE=/srv/deployment/parsoid/deploy/conf/wmf/betalabs.localsettings.js

Event Timeline

hashar raised the priority of this task from to Medium.
hashar updated the task description. (Show Details)
hashar added subscribers: Krenair, Luke081515, hashar and 2 others.

Most probably caused by https://gerrit.wikimedia.org/r/#/c/249138/ T115665: Reorg parsoid repo

It renamed a bunch of files and the puppet configuration file hasn't been updated to reflect the new files layout.

PARSOID_BASE_PATH and NODE_PATH in /etc/default/parsoid are wrong.

Similar problem with mw-vagrant. It is looking for /vagrant/srv/parsoid/src/api/server.js. It doesn't exist but there's /vagrant/srv/parsoid/src/bin/server.js.

I am going to add a symlink for now so these continue to work. Separately, we'll update puppet (for both beta and production) -- I need to understand how the puppet updates and deploy ordering works.

For puppet, manifests/role/parsoid.pp vary the config with:

class role::parsoid::beta {
    ...
    # For beta, override NODE_PATH:
    $parsoid_node_path = '/srv/deployment/parsoid/deploy/node_modules'
    # Also override PARSOID_SETTINGS_FILE
    $parsoid_settings_file = '/srv/deployment/parsoid/deploy/conf/wmf/betalabs.localsettings.js'

    # Checkout of mediawiki/services/parsoid
    $parsoid_base_path = '/srv/deployment/parsoid/deploy/src'
    ...
}

The puppet variables are then used to craft the /etc/default/parsoid file via an .erb template.

templates/misc/parsoid.default.erb:

#####################################################################
### THIS FILE IS MANAGED BY PUPPET
### puppet:///templates/misc/parsoid.default
#####################################################################

# File where the parsoid daemon will write stdin and stdout to
PARSOID_LOG_FILE=<%= @parsoid_log_file %>

# Checkout of mediawiki/services/parsoid
<% if @parsoid_base_path then %>
PARSOID_BASE_PATH=<%= @parsoid_base_path %>
<% else %>
# Override in puppet using parsoid_base_path
#PARSOID_BASE_PATH
<% end %>

# Checkout of mediawiki/services/parsoid/deploy
<% if @parsoid_node_path then %>
NODE_PATH=<%= @parsoid_node_path %>
<% else %>
# Override in puppet using parsoid_node_path
#NODE_PATH=
<% end %>

# Checkout of mediawiki/services/parsoid/deploy
<% if @parsoid_settings_file then %>
PARSOID_SETTINGS_FILE=<%= @parsoid_settings_file %>
<% else %>
# Override in puppet using parsoid_settings_file
#PARSOID_SETTINGS_FILE=/etc/parsoid/settings.js
<% end %>

https://gerrit.wikimedia.org/r/#/c/249392/ is the parsoid patch to add the symlink
https://gerrit.wikimedia.org/r/#/c/249399/ is the puppet change (marked it WIP so it doesn't get deployed prematurely)

Change 249406 had a related patch set uploaded (by Mobrovac):
Parsoid: Upstart: start the service as bin/server.js

https://gerrit.wikimedia.org/r/249406

Change 249406 merged by jenkins-bot:
Parsoid: Upstart: start the service as bin/server.js

https://gerrit.wikimedia.org/r/249406

root@deployment-parsoid05:~# curl http://localhost:8000/_version|python -m json.tool
{
	    "name": "parsoid",
	    "sha": "47b8013ae7b15428122579bfff4943191c1866db",
	    "version": "0.4.1-git"
}

Seems fine now.

ssastry claimed this task.
ssastry removed a project: Patch-For-Review.
ssastry set Security to None.