The web workers don't actually use the model to do any scoring. Let's keep the metadata handy and ditch the actual models on the web nodes.
I tried writing a little script to get a sense for how our models took up memory.
>>> import glob >>> from revscoring import ScorerModel >>> # RES Check 1 ... >>> from revscoring.languages import * >>> from revscoring.languages import arabic, czech, dutch, english, estonian, french, german, hebrew, hungarian, indonesian, italian, norwegian, persian, polish, portuguese, spanish, swedish, russian, ukrainian, vietnamese >>> # RES Check 2 ... >>> models = [ScorerModel.load(open(path)) for path in glob.glob("submodules/*/models/*.model")] >>> # RES Check 3 ... >>> len(models) 42
RES memory checks:
- 58.932 MB
- 302.808 MB
- 1153.012 MB