Page MenuHomePhabricator

Check drafttopic model memory usage
Closed, ResolvedPublic


Simple check:

  1. Start python interpreter.
  2. Check memory usage. Record RES.
  3. Load drafttopic model. Use it to make a prediction.
  4. Check memory usage. Record RES.
  5. Post results here.

ORES check:

  1. Load ORES web and celery workers as configured from ores-wmflabs-deploy
  2. Check memory usage (both celery and web workers)
  3. Add drafttopic model to configuration and restart ORES service
  4. Use drafttopic model to make a prediction.
  5. Check memory usage (both celery and web workers)

Related Objects

Event Timeline

Halfak triaged this task as Medium priority.Apr 16 2018, 2:30 PM
Halfak created this task.

Simple check:

Not loaded

halfak    5029  7.0  0.8 480916 69776 pts/1    Sl+  17:13   0:01 python


halfak    5029 32.4  6.6 1036444 521708 pts/1  S+   17:13   0:10 python

Note that making a prediction did not change memory usage at all.

Slightly different results when running under ORES, it seems that performing drafttopic scoring does have a one-time effect on RES for each thread.

Not configured:

awight   28404  0.0  3.4 745336 272956 pts/6   S+   15:48   0:00 python

Configured but not yet scored:

awight   31781  0.0  7.6 1079512 598680 pts/6  S+   15:53   0:00 python

After scoring:

awight   31820  0.0  7.6 1081632 600860 pts/6  S+   15:53   0:00 python

Reading your notes, should we expect a ~2x increase in per-worker memory usage (based on a naive interpretation of RSS?)

I'm pretty certain this won't cause 2x overall memory usage, although it is the upper bound. Some of RSS ends up being copy-on-write shared across processes, but we can't predict what proportion that will be. I think we should check memory usage on the canary box before doing the full deployment.

We're running really low on memory for the celery workers so I'm reducing concurrency from 16 to 14.

OK everything looks good. We're running nicely on See :)