Investigate memory leak in precached
Closed, ResolvedPublic
Actions

Description

We just had another minor outage of ores-web-03. It looks like the precached utility was using 3.1GB of memory!

USER	PID	%CPU	%MEM	VSZ	RSS	TTY	STAT	START	TIME
www-data	24019	13.5	78.6	3947668	3187824	?	Ssl	Sep12	2210:14

/srv/ores/venv/bin/python /srv/ores/config/submodules/ores/utility precached stream.wikimedia.org https://ores.wmflabs.org --config /srv/ores/config/config

Event Timeline

Halfak created this task.Sep 23 2016, 8:49 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 23 2016, 8:49 PM

Happened again just a moment ago. It looks like the same story with precached.

I noted in meeting that I suspect that there's something going on with metrics collection.

Halfak claimed this task.Oct 17 2016, 4:49 PM

Halfak triaged this task as High priority.

Looks like this could be useful: https://pythonhosted.org/Pympler/muppy.html

I worked out that there was a datastructure that would grow slowly over time. I just submitted https://github.com/wiki-ai/ores/pull/170 that should address the issue.

Halfak moved this task from Active to Review on the Scoring-platform-team (Current) board.Oct 17 2016, 8:57 PM

Ladsgroup moved this task from Review to Done on the Scoring-platform-team (Current) board.Oct 18 2016, 2:03 PM

Halfak closed this task as Resolved.Oct 24 2016, 7:33 PM

Investigate memory leak in precached Closed, ResolvedPublicActions

Description

Event Timeline

Investigate memory leak in precached
Closed, ResolvedPublic
Actions