Page MenuHomePhabricator

Wikimetrics is trying to use more than 10 connections at the same time
Closed, ResolvedPublic5 Estimated Story Points

Description

I'm tagging DBAs on this after a conversation with @Marostegui in -cloud. I did some more digging and logging and I found that I think the wikimetrics user is restricted to 10 connections at the same time across all wikis, not just for one wiki. I ran show processlist fast enough to catch the moment it ran out of connections:

IdUserHostdbCommandTimeStateInfoProgress
168140821s5226110.68.23.232:32886ilowiki_pQuery0initshow processlist0.000
168141098s5226110.68.23.232:32912maiwiki_pSleep3NULL0.000
168141099s5226110.68.23.232:32914fiwiki_pSleep3NULL0.000
168141100s5226110.68.23.232:32916napwiki_pSleep3NULL0.000
168141103s5226110.68.23.232:32918dsbwiki_pSleep3NULL0.000
168141104s5226110.68.23.232:32920cebwiki_pSleep3NULL0.000
168141106s5226110.68.23.232:32922pmswiki_pSleep3NULL0.000
168141107s5226110.68.23.232:32924fawiki_pSleep3NULL0.000
168141108s5226110.68.23.232:32926ugwiki_pSleep3NULL0.000
168141109s5226110.68.23.232:32928snwiki_pSleep3NULL0.000

And in the log I have errors from all the other wikis it tries to access. I'll try to solve this by messing with the code. I should be able to close the connection to one wiki before moving on to the next one, it's not running in parallel like I thought. So I'm taking care of this, just cc-ing DBAs so they're in the loop since I'll be on paternity leave any moment.

Event Timeline

Thanks @Milimetric for spending time on fixing this and for understanding the implications of going from 10 to 100 as max allowed connections :-)
cc @madhuvishy so she is aware that in the end the tool might be started again (hopefully!)

Nuria set the point value for this task to 5.Aug 11 2017, 3:02 PM