Page MenuHomePhabricator

Investigate memory leak in precached
Closed, ResolvedPublic

Description

We just had another minor outage of ores-web-03. It looks like the precached utility was using 3.1GB of memory!

USERPID%CPU%MEMVSZRSSTTYSTATSTARTTIME
www-data2401913.578.639476683187824?SslSep122210:14

/srv/ores/venv/bin/python /srv/ores/config/submodules/ores/utility precached stream.wikimedia.org https://ores.wmflabs.org --config /srv/ores/config/config

Event Timeline

Halfak created this task.Sep 23 2016, 8:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 23 2016, 8:49 PM

Happened again just a moment ago. It looks like the same story with precached.

I noted in meeting that I suspect that there's something going on with metrics collection.

Halfak claimed this task.Oct 17 2016, 4:49 PM
Halfak triaged this task as High priority.

I worked out that there was a datastructure that would grow slowly over time. I just submitted https://github.com/wiki-ai/ores/pull/170 that should address the issue.

Halfak closed this task as Resolved.Oct 24 2016, 7:33 PM