Looks like elasticsearch has stopped.
It was being configured by puppet using the same roles as the labs cluster. Role still seems to be in place but nothing is running on 9200. Investigation is needed.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T198498 Wikifactmine-pipeline daily import job backing up | |||
| Open | None | T198690 Investigate why Wikifactmine ElasticSearch has stopped |
Event Timeline
Comment Actions
@Aklapper It's a Project Grants funded project which has two tools on Toolforge and well as a project on CloudVPS. I thought it had a component project for it but I guess it never did.
I'm not actually actively working on it now. I guess whoever is doing the work on it from https://meta.wikimedia.org/wiki/Grants:Project/ScienceSource may want one.
Comment Actions
[offtopic] @Tarrow: If you or someone would like to have a workboard and project tag, please follow https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects - thanks!
Comment Actions
Separate, but related, https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/447564/ by @EBernhardson proposes to remove support for ElasticSearch 2.x in the puppet roles. We think that wikifactmine is the only cluster left running 2.x today.
Comment Actions
@bd808 @EBernhardson I think the puppet role is good to be removed now. I've unapplied it from horizon and removed the references on heira.
Comment Actions
I can't ssh into any of these VMs at all at the moment. They're in a fairly precarious situation if not running puppet.
Comment Actions
These three elastic search nodes have local firewalls (ferm + iptables) that seem not to be managed by puppet. That means that they're probably blocking whatever ports you would normally need to communicate. I edited ferm by hand to allow bastion access so that you should be able to ssh in and see what's going on.