Page MenuHomePhabricator

VisualEditor (RESTBase) broken in vagrant
Closed, ResolvedPublic

Description

After booting Vagrant today, RESTBase wasn't responding (getting HTTP 503 Service Unavailable on all requests to dev.wiki.local.wmftest.net:8080/api/rest_v1, e.g. when loading VisualEditor).

vagrant provision & reload didn't resolve this.

When running vagrant git-update, there were two errors:

==> Updating zotero_translation_server ...
[*] Updating repo in /vagrant/srv/zotero/translation-server ...
From https://gerrit.wikimedia.org/r/p/mediawiki/services/zotero/translation-server
   508676c..6a4a828  master     -> origin/master
First, rewinding head to replay your work on top of it...
error: The following untracked working tree files would be overwritten by checkout:
defaults/preferences/defaults.js
Please move or remove them before you can switch branches.
Aborting
could not detach HEAD
 ***** GIT PULL failed in /vagrant/srv/zotero/translation-server for branch 'master'
[*] Restarting service zotero ...
zotero stop/waiting
zotero start/running, process 9427

==> Updating zotero_translators ...
[*] Updating repo in /vagrant/srv/zotero/translators ...
fatal: ref HEAD is not a symbolic ref
expr: syntax error
Previous HEAD position was 6fc0f92... Add Scap3 configuration
Switched to branch 'master'
Your branch and 'origin/master' have diverged,
and have 7 and 316 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)
First, rewinding head to replay your work on top of it...

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: empty ident name (for <vagrant@mediawiki-vagrant.dev>) not allowed
You need to set your committer info first
 ***** GIT PULL failed in /vagrant/srv/zotero/translators for branch 'master'

Looks like part of the update process is making a commit, but git (since recently?) by default no longer allows making commits without a config (there is an override option though).

This means the restbase git-update won't finish due to failing dependencies. I don't know this would help. I guess if that update finishes maybe it'll restart the service properly.

Restbase log:

 at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:7:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at RBSQLite.setup (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/index.js:184:20)\n    at makeRBSQLite (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/index.js:201:15)\n    at module.exports (/vagrant/srv/restbase/sys/table.js:16:65)","levelPath":"fatal/startup","msg":"Message not supplied","time":"2016-04-19T00:20:26.000Z","v":0}
{"name":"restbase","hostname":"mediawiki-vagrant","pid":1994,"level":50,"message":"worker 3260 died (1), restarting.","levelPath":"error/service-runner/master","msg":"worker 3260 died (1), restarting.","time":"2016-04-19T00:20:28.022Z","v":0}
{"name":"restbase","hostname":"mediawiki-vagrant","pid":3269,"level":60,"err":{"message":"Cannot find module '/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/binding/node-v46-linux-x64/node_sqlite3.node'","name":"Error","stack":"Error: Cannot find module '/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/binding/node-v46-linux-x64/node_sqlite3.node'\n    at Function.Module._resolveFilename (module.js:325:15)\n    at Function.Module._load (module.js:276:25)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/sqlite3.js:4:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/lib/clientWrapper.js:4:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:7:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at RBSQLite.setup (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/index.js:184:20)\n    at makeRBSQLite (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/index.js:201:15)\n    at module.exports (/vagrant/srv/restbase/sys/table.js:16:65)","code":"MODULE_NOT_FOUND"},"stack":"Error: Cannot find module '/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/binding/node-v46-linux-x64/node_sqlite3.node'\n    at Function.Module._resolveFilename (module.js:325:15)\n    at Function.Module._load (module.js:276:25)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/sqlite3.js:4:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/lib/clientWrapper.js:4:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:7:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at RBSQLite.setup (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/index.js:184:20)\n    at makeRBSQLite (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/index.js:201:15)\n    at module.exports (/vagrant/srv/restbase/sys/table.js:16:65)","levelPath":"fatal/startup","msg":"Message not supplied","time":"2016-04-19T00:20:35.704Z","v":0}
{"name":"restbase","hostname":"mediawiki-vagrant","pid":1994,"level":50,"message":"worker 3269 died (1), restarting.","levelPath":"error/service-runner/master","msg":"worker 3269 died (1), restarting.","time":"2016-04-19T00:20:37.727Z","v":0}
{"name":"restbase","hostname":"mediawiki-vagrant","pid":3278,"level":60,"err":{"message":"Cannot find module '/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/binding/node-v46-linux-x64/node_sqlite3.node'","name":"Error","stack":"Error: Cannot find module '/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/binding/node-v46-linux-x64/node_sqlite3.node'\n    at Function.Module._resolveFilename (module.js:325:15)\n    at Function.Module._load (module.js:276:25)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/sqlite3.js:4:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/lib/clientWrapper.js:4:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:7:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at RBSQLite.setup (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/index.js:184:20)\n    at makeRBSQLite (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/index.js:201:15)\n    at module.exports (/vagrant/srv/restbase/sys/table.js:16:65)","code":"MODULE_NOT_FOUND"},"stack":"Error: Cannot find module '/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/binding/node-v46-linux-x64/node_sqlite3.node'\n    at Function.Module._resolveFilename (module.js:325:15)\n    at Function.Module._load (module.js:276:25)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/sqlite3.js:4:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/lib/clientWrapper.js:4:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:7:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at RBSQLite.setup (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/index.js:184:20)\n    at makeRBSQLite (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/index.js:201:15)\n    at module.exports (/vagrant/srv/restbase/sys/table.js:16:65)","levelPath":"fatal/startup","msg":"Message not supplied","time":"2016-04-19T00:20:44.794Z","v":0}
{"name":"restbase","hostname":"mediawiki-vagrant","pid":1994,"level":50,"message":"worker 3278 died (1), restarting.","levelPath":"error/service-runner/master","msg":"worker 3278 died (1), restarting.","time":"2016-04-19T00:20:46.811Z","v":0}
{"name":"restbase","hostname":"mediawiki-vagrant","pid":3287,"level":60,"err":{"message":"Cannot find module '/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/binding/node-v46-linux-x64/node_sqlite3.node'","name":"Error","stack":"Error: Cannot find module '/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/binding/node-v46-linux-x64/node_sqlite3.node'\n    at Function.Module._resolveFilename (module.js:325:15)\n    at Function.Module._load (module.js:276:25)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/sqlite3.js:4:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/lib/clientWrapper.js:4:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:7:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at RBSQLite.setup (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/index.js:184:20)\n    at makeRBSQLite (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/index.js:201:15)\n    at module.exports (/vagrant/srv/restbase/sys/table.js:16:65)","code":"MODULE_NOT_FOUND"},"stack":"Error: Cannot find module '/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/binding/node-v46-linux-x64/node_sqlite3.node'\n    at Function.Module._resolveFilename (module.js:325:15)\n    at Function.Module._load (module.js:276:25)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/node_modules/sqlite3/lib/sqlite3.js:4:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/lib/clientWrapper.js:4:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at Object.<anonymous> (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:7:15)\n    at Module._compile (module.js:409:26)\n    at Object.Module._extensions..js (module.js:416:10)\n    at Module.load (module.js:343:32)\n    at Function.Module._load (module.js:300:12)\n    at Module.require (module.js:353:17)\n    at require (internal/module.js:12:17)\n    at RBSQLite.setup (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/index.js:184:20)\n    at makeRBSQLite (/vagrant/srv/restbase/node_modules/restbase-mod-table-sqlite/index.js:201:15)\n    at module.exports (/vagrant/srv/restbase/sys/table.js:16:65)","levelPath":"fatal/startup","msg":"Message not supplied","time":"2016-04-19T00:20:53.483Z","v":0}
{"name":"restbase","hostname":"mediawiki-vagrant","pid":1994,"level":50,"message":"worker 3287 died (1), restarting.","levelPath":"error/service-runner/master","msg":"worker 3287 died (1), restarting.","time":"2016-04-19T00:20:55.511Z","v":0}

Similar:

Event Timeline

Krinkle created this task.Apr 19 2016, 12:16 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 19 2016, 12:16 AM
Krinkle updated the task description. (Show Details)Apr 19 2016, 12:19 AM
Krinkle updated the task description. (Show Details)Apr 19 2016, 12:22 AM
Krinkle updated the task description. (Show Details)Apr 19 2016, 12:32 AM
Krinkle updated the task description. (Show Details)Apr 19 2016, 12:36 AM

I manually removed defaults/preferences/defaults.js (presumably left behind by a previous version of the puppet manifest), and re-ran vagrant git-update. It still fails on zotero/translations but that doesn't affect restbase's update.

Regardless, RESTBase still fails with HTTP 503. Another around of vagrant provision and vagrant reload (all without errors) also did not bring back a working RESBase install.

@Krinkle May be vagrant ssh, sudo service restbase stop; cd /vagrant/srv/restbase; rm -rf ./node_modules; npm i; sudo service restbase start would help as a quick fix. Zotero issue needs fixing anyway though.

I had the same issues with Zotero and Zotero Translators but RESTbase worked fine for me, FWIW.

I manually removed defaults/preferences/defaults.js (presumably left behind by a previous version of the puppet manifest), and re-ran vagrant git-update.

That's because we now have a symlink in the repo to make it work with Scap3 in production.

It still fails on zotero/translations but that doesn't affect restbase's update.

For the fix of the translators repo cf T133038: vagrant git-update might fail if git username is not configured.

Regardless, RESTBase still fails with HTTP 503. Another around of vagrant provision and vagrant reload (all without errors) also did not bring back a working RESBase install.

I'm unable to reporduce this error. Silly question: have you git pull-ed in your Vagrant repo dir before doing the update?

@Krinkle May be vagrant ssh, sudo service restbase stop; cd /vagrant/srv/restbase; rm -rf ./node_modules; npm i; sudo service restbase start would help as a quick fix.

That's the equivalent of vagrant git-update --force.

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptApr 19 2016, 6:02 PM
Krinkle closed this task as Resolved.May 24 2016, 12:14 AM
Krinkle claimed this task.

@Krinkle May be vagrant ssh, sudo service restbase stop; cd /vagrant/srv/restbase; rm -rf ./node_modules; npm i; sudo service restbase start would help as a quick fix. Zotero issue needs fixing anyway though.

I was having a very similar error to this "Error loading data from server: HTTP 503". These instructions worked, but I also had to chown -R dgarry srv earlier in the process since I was getting errors trying to modify files. Thanks for helping!

Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptMay 19 2017, 12:46 PM