I'm a software developer working in archives.
Wed, Sep 11
Thanks again for everyone's help.
@bd808 thank you! I only was doing the ssh-within-ssh because it worked for me in the past. But I've got ProxyJump working fine now, and can access ws-web, so I appreciate you taking the time to explain this solution.
Tue, Sep 10
Thanks for getting wikistream back working again. I'm still unable to ssh in however.
Hmm, looking at the docs perhaps it's no longer possible to ssh to bastion.wmflabs.org and then ssh on to ws-web? Should I follow the instructions at this page to set up bast1002.wikimedia.org as a proxy?
Thanks for the quick response! Here you go:
Dec 19 2018
You're right, sorry for the false alarm.
Nov 27 2018
The root partition on ws-web was full again. For some reason daemon.log in /var/log/ doesn't rotate properly. I've adjusted the code to squash excessive logging when running in production. Hopefully that will help in the future. You should find the service is running again.
Apr 20 2018
In the comment that you copy and pasted I did the second option of creating a systemd override. The trouble was that my override was overidden.
I recently noticed that wikistream was down (again) and I needed to update /lib/systemd/system/varnish.service again because something (Puppet?) overwrote my Varnish config change. Varnish needs to run on port 80, but something keeps resetting it to 6081, which is the default. Perhaps this happens when varnish is upgraded. I have no clue, but it's annoying to notice it's down again only to see the same problem has repeated.
Apr 15 2018
I'm seeing a Varnish error at wikistream.wmflabs.org again. The backend on ws-web is working fine. I'm not quite sure what's going on upstream from the app. Can someone help?
Jan 27 2018
Jan 26 2018
I modified /lib/systemd/system/varnish.service to have varnish listen on port 80, since that's how it ran for quite a while (years?) without any trouble until recently. Thanks for the information about systemd, I was looking around in /etc/init and /etc/init.d.
Jan 24 2018
Varnish is running on ws-web, which proxies the wikistream app that listens on localhost:3000. A while ago (maybe a year, or more?) I configured Varnish to listen on 0.0.0.0:80 in /etc/default/varnish. But sudo service varnish start no longer seems to be reading that for reasons I do not understand. Instead Varnish is listening on port 6081.
Jan 23 2018
Dec 31 2017
It is still maintained, at least by me when I notice it is down. It appeared that /var/log on ws-web filled up with lots of daemon.log.x files (9 GBs worth). I believe this caused the process to stop, and for it to not be able to restart.
Nov 4 2017
I'm a bit confused about how to get npm install to run under kubernetes with the docker-registry.tools.wmflabs.org/toollabs-nodejs-base:latest image. Is there an example of that somewhere?
Oct 18 2017
Ok, thanks for the details, this helps a lot. So it sounds like the way to move forward is to start using Kubernetes which I wanted to learn about anyway, and now I have a reason.
Sep 7 2017
Yes, I'm able to log in now. Thanks so much!
Sep 2 2017
I don't know if T103148 is related or not. Maybe my homedir on ws-web was somehow modified?
I can see that the ssh-key for my laptop's id_rsa.pub and the id_rsa.pub in my homedir on bastion.wmflabs.org are present in that list.
I don't know if this is related but I'm unable to ssh from bastion.wmflabs.org to ws-web any longer. I notice the service is currently down and wanted to see what was wrong.
Aug 6 2017
The memory leak is a result of using the old version of node with the Cheerio library. If node can be upgraded that would help me. v0.10 is end-of-life and is no longer maintained.
Jul 8 2017
I actually have tried upping the memory. It just takes longer to run out of memory. I would like to upgrade Node if possible.
Jun 1 2017
Nov 14 2016
Sorry I didn't see https://phabricator.wikimedia.org/T146154 before I opened this.
Jan 31 2016
Sorry about that. I really had no idea it was such a large file when I started to grep it :-) I have since deleted it, which probably wasn't so nice to NFS either...oops!
May 17 2015
As a current user of Freebase Suggest, which is going away apparently on June 30th, it would be super to have a Wikidata based alternative. I suspect (but do not know) that there are other Freebase Suggest users who will need to migrate to something, and having a path to Wikidata would be good for Wikidata entities being used outside of the Wikipedia ecosystem.
Jan 25 2015
Yes, this seemed to work: