@PeterBowman What -mem X setting are you using for your job? Have you tried increasing that limit?

https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Allocating_additional_memory

@bd808 Thanks, increasing -mem value from 5g to 8g made my jobs respond again, I'll try to adjust it to something in between. I was hoping to ultimately avoid this - unable to create native thread seemed unrelated to the amount of memory (IMO, in spite of the OOM error class) and 5g was already high, compare with the default vmem size as specified in that link. I never had to increase this limit in over 3 years and something changed just a few days ago, so I wanted to leave my feedback here. Please feel free to close this task if it's OK to raise -mem that high.

Edit: currently using -mem 6g.

something changed just a few days ago

We have had several Java users notice that the version of Java we have on Stretch job grid needs more -mem allocated than the Trusty job grid did. I can not think of any software related change in Toolforge in the past days/weeks that would have made this worse. Depending on what your bot is doing it may be actually just processing more data in a run related to some change in the rate of content in the wikis you are interacting with too. That can be tricky to determine for some bots.