Page MenuHomePhabricator

Don't load models into memory on web workers
Closed, ResolvedPublic

Description

The web workers don't actually use the model to do any scoring. Let's keep the metadata handy and ditch the actual models on the web nodes.

Event Timeline

Halfak created this task.Jul 5 2016, 6:50 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 5 2016, 6:50 PM
Ladsgroup triaged this task as High priority.Jul 8 2016, 6:06 AM

Looking at this on my laptop with the wmflabs deploy config

model setuwsgicelery
all532MB920MB
half contexts disabled178MB153MB
no dam/gf models426MB675MB
10x enwiki_wp101126MB1487MB

I tried writing a little script to get a sense for how our models took up memory.

>>> import glob
>>> from revscoring import ScorerModel
>>> # RES Check 1
... 
>>> from revscoring.languages import *
>>> from revscoring.languages import arabic, czech, dutch, english, estonian, french, german, hebrew, hungarian, indonesian, italian, norwegian, persian, polish, portuguese, spanish, swedish, russian, ukrainian, vietnamese
>>> # RES Check 2
... 
>>> models = [ScorerModel.load(open(path)) for path in glob.glob("submodules/*/models/*.model")]
>>> # RES Check 3
... 
>>> len(models)
42

RES memory checks:

  1. 58.932 MB
  2. 302.808 MB
  3. 1153.012 MB
Halfak closed this task as Resolved.Aug 2 2016, 9:47 PM