Thanks for looking at this @Bstorm. Yes I would be willing to test as long as browser support is something you want to have the image have.

I don't know if anything changed in NFS but I upgraded the Node image in the k8s config and it is running now!

Honestly, I'm finding it hard to find any information about how to run Node processes on ToolsForge. So for now I will run this process on my own server until I can figure out how to convince the Kubernetes cluster to run it. The process was running for over a year with no problems and then I discovered almost by accident that it had failed and now will not restart. It was good while it lasted!

If you would like to test and see the errors you can:

Thanks! It is Toolforge related. I looked for the Toolforge tag and couldn't seem to find it for some reason. Maybe I was looking for Toolsforge ...

Thanks for the work @bd808 !

@yuvipanda would you be open to me using your wikistream Tools project as a place to move wikistream into? It doesn't look like the space is currently being used? If you add me as a member of the project I think I can take it from there.

I seem to remember moving this to run at WMF as Toolforge was transitioning its architecture so it wasn't available at the time. I maintain another service in Toolforge and have found it to be a supportive environment. So I will go ahead and request the new tool and udpate this ticket when it's available there.

I cannot honestly. If you think it is worth preserving please feel free to migrate the service. It really is a minimal node process that requires no database access at all. Otherwise I'll run it myself somewhere else.

If there are instructions on how to create the new instance and update DNS appropriately so that wikistream.wmflabs.org points to it that would be appreciated. I won't be able to determine whether I can get this done in the next few weeks without more information.

Ahh, yes that works for me. Thank you for clarifying!

I'm sorry to have to reopen this. But I've been notified that wikistream.wmflabs.org is down again and I no longer seem to ssh with the the ProxyJump configuration? Here's the log of my ssh attempt:

Thanks again for everyone's help.

@bd808 thank you! I only was doing the ssh-within-ssh because it worked for me in the past. But I've got ProxyJump working fine now, and can access ws-web, so I appreciate you taking the time to explain this solution.

Thanks for getting wikistream back working again. I'm still unable to ssh in however.

Hmm, looking at the docs perhaps it's no longer possible to ssh to bastion.wmflabs.org and then ssh on to ws-web? Should I follow the instructions at this page to set up bast1002.wikimedia.org as a proxy?

Thanks for the quick response! Here you go:

You're right, sorry for the false alarm.

The root partition on ws-web was full again. For some reason daemon.log in /var/log/ doesn't rotate properly. I've adjusted the code to squash excessive logging when running in production. Hopefully that will help in the future. You should find the service is running again.

In the comment that you copy and pasted I did the second option of creating a systemd override. The trouble was that my override was overidden.

I recently noticed that wikistream was down (again) and I needed to update /lib/systemd/system/varnish.service again because something (Puppet?) overwrote my Varnish config change. Varnish needs to run on port 80, but something keeps resetting it to 6081, which is the default. Perhaps this happens when varnish is upgraded. I have no clue, but it's annoying to notice it's down again only to see the same problem has repeated.

I'm seeing a Varnish error at wikistream.wmflabs.org again. The backend on ws-web is working fine. I'm not quite sure what's going on upstream from the app. Can someone help?

I modified /lib/systemd/system/varnish.service to have varnish listen on port 80, since that's how it ran for quite a while (years?) without any trouble until recently. Thanks for the information about systemd, I was looking around in /etc/init and /etc/init.d.

Varnish is running on ws-web, which proxies the wikistream app that listens on localhost:3000. A while ago (maybe a year, or more?) I configured Varnish to listen on 0.0.0.0:80 in /etc/default/varnish. But sudo service varnish start no longer seems to be reading that for reasons I do not understand. Instead Varnish is listening on port 6081.

It is still maintained, at least by me when I notice it is down. It appeared that /var/log on ws-web filled up with lots of daemon.log.x files (9 GBs worth). I believe this caused the process to stop, and for it to not be able to restart.

I'm a bit confused about how to get npm install to run under kubernetes with the docker-registry.tools.wmflabs.org/toollabs-nodejs-base:latest image. Is there an example of that somewhere?

Ok, thanks for the details, this helps a lot. So it sounds like the way to move forward is to start using Kubernetes which I wanted to learn about anyway, and now I have a reason.

Yes, I'm able to log in now. Thanks so much!

I don't know if T103148 is related or not. Maybe my homedir on ws-web was somehow modified?

I can see that the ssh-key for my laptop's id_rsa.pub and the id_rsa.pub in my homedir on bastion.wmflabs.org are present in that list.

I don't know if this is related but I'm unable to ssh from bastion.wmflabs.org to ws-web any longer. I notice the service is currently down and wanted to see what was wrong.

The memory leak is a result of using the old version of node with the Cheerio library. If node can be upgraded that would help me. v0.10 is end-of-life and is no longer maintained.

I actually have tried upping the memory. It just takes longer to run out of memory. I would like to upgrade Node if possible.

Sorry I didn't see https://phabricator.wikimedia.org/T146154 before I opened this.

Sorry about that. I really had no idea it was such a large file when I started to grep it :-) I have since deleted it, which probably wasn't so nice to NFS either...oops!

As a current user of Freebase Suggest, which is going away apparently on June 30th, it would be super to have a Wikidata based alternative. I suspect (but do not know) that there are other Freebase Suggest users who will need to migrate to something, and having a path to Wikidata would be good for Wikidata entities being used outside of the Wikipedia ecosystem.

Yes, this seemed to work: