The web workers don't actually use the model to do any scoring. Let's keep the metadata handy and ditch the actual models on the web nodes.
Description
Description
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | None | T139408 [Epic] ORES refactor: Scoring structure | |||
Resolved | Halfak | T139407 Don't load models into memory on web workers |
Event Timeline
Comment Actions
Looking at this on my laptop with the wmflabs deploy config
model set | uwsgi | celery |
all | 532MB | 920MB |
half contexts disabled | 178MB | 153MB |
no dam/gf models | 426MB | 675MB |
10x enwiki_wp10 | 1126MB | 1487MB |
Comment Actions
I tried writing a little script to get a sense for how our models took up memory.
>>> import glob >>> from revscoring import ScorerModel >>> # RES Check 1 ... >>> from revscoring.languages import * >>> from revscoring.languages import arabic, czech, dutch, english, estonian, french, german, hebrew, hungarian, indonesian, italian, norwegian, persian, polish, portuguese, spanish, swedish, russian, ukrainian, vietnamese >>> # RES Check 2 ... >>> models = [ScorerModel.load(open(path)) for path in glob.glob("submodules/*/models/*.model")] >>> # RES Check 3 ... >>> len(models) 42
RES memory checks:
- 58.932 MB
- 302.808 MB
- 1153.012 MB