https://logstash-beta.wmflabs.org/goto/3da590c69d2896cf4d4cd227616fcd29 is one of them, but you should check the fatalmonitor board for more.

@MarcoAurelio Thanks for the report!

Our celery worker died three days ago, probably due to out-of-memory. It's not the first time we've seen this with beta ORES. I'll repurpose this task to find a long-term fix.

Dear @awight; thanks for your quick response. Yesterday @Krenair was discussing at -releng that there were a number of Puppet errors on all Beta Cluster machines such as full disks, missing or wrong hieradata, etc. Maybe worth having a look at those as well? Nota bene: I hardly know puppet stuff so apologies if I'm wrong. Regards.

It looks like we might need more memory on sca03 (or whatever beta cluster node we're deploying to). Maybe it's time to make our beta node look like ores-staging (which has 16 GB of memory)

Alternatively, we could also reduce the # of workers from 8 to 4. I think we could still handle beta-capacity with that.

Looking at /srv/log/ores/app.log, we've been down for at least 2 weeks. Any useful evidence has been rotated out of logs at this point. Let's let it crash again and try to check the diagnostics sooner?

Mentioned in SAL (#wikimedia-releng) [2018-01-09T08:51:51Z] <Amir1> ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki (T184276)