⚓ T243451 Deploy ORES -- Late Jan 2020

Subject	Repo	Branch	Lines +/-
New draft topic models with 50d vectors.	mediawiki/services/ores/deploy	master	+3 -3
Updates for revscoring 2.6.5	research/ores/wheels	master	+36 -36
Adds topic models for ar, cs, ko, and vi.	mediawiki/services/ores/deploy	master	+22 -2

Status	Assigned	Task
Resolved	Halfak	T243451 Deploy ORES -- Late Jan 2020
Resolved	Halfak	T235181 Build WikiProject directory topic models for ar, cs, and kowiki
Resolved	Halfak	T235183 Experiment with different vector lengths for ar, cs, en, and kowiki topic models.
Resolved	Halfak	T235184 Generate word vectors for ar, cs, en, and ko using FastText
Resolved	Isaac	T242013 Implement native NN model in revscoring
Resolved	Halfak	T235187 Create labeled data for topic models in ar, cs, kowiki
Resolved	Isaac	T236713 Improve drafttopic training data pipeline
Resolved	Isaac	T240273 Extract cross-wiki WikiProject tags
Resolved	Halfak	T240286 Re-train English Wikipedia topic model using new WikiProject Taxonomy
Resolved	Halfak	T240276 Restructure WikiProject directory to be better
Resolved	kevinbazira	T240282 Improve WikiProject template --> WikiProject mapping
Resolved	Isaac	T241270 Add wikidata features to topic models
Resolved	kevinbazira	T242345 Implement English pronoun count features in topic models
Resolved	Halfak	T243108 Add new vectors to deployment assets
Resolved	Halfak	T243107 Retrain enwiki drafttopic models on supervised vectors
Resolved	Halfak	T243522 Reduce memory footprint of topic models

Halfak created this task.Jan 22 2020, 8:42 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 22 2020, 8:42 PM

Halfak added subtasks: T235181: Build WikiProject directory topic models for ar, cs, and kowiki, T242345: Implement English pronoun count features in topic models, T243108: Add new vectors to deployment assets, T243107: Retrain enwiki drafttopic models on supervised vectors, T235184: Generate word vectors for ar, cs, en, and ko using FastText.Jan 22 2020, 8:43 PM

Change 566595 had a related patch set uploaded (by Halfak; owner: Halfak):
[mediawiki/services/ores/deploy@master] Adds topic models for ar, cs, ko, and vi.

https://gerrit.wikimedia.org/r/566595

gerritbot added a project: Patch-For-Review.Jan 22 2020, 8:44 PM

Change 566595 merged by Accraze:
[mediawiki/services/ores/deploy@master] Adds topic models for ar, cs, ko, and vi.

https://gerrit.wikimedia.org/r/566595

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2020, 10:10 PM

Halfak moved this task from Parked to Pending deployment on the Machine-Learning-Team (Active Tasks) board.Jan 23 2020, 3:32 PM

I failed on the deploy to beta. Looks like memory usage is far too high. I'm investigating.

I'm investigating memory usage. I'm working from a python terminal on my dev laptop. Essentially, I'm tracking VSZ and RSS while running commands.

Before loading anything:

VSZ: 35600
RSS: 9340

After from revscoring import Model:

VSZ: 495752
RSS: 76216

After enwiki = Model.load(open("models/enwiki.articletopic.gradient_boosting.model"))

VSZ: 1010852
RSS: 567348

After arwiki = Model.load(open("models/arwiki.articletopic.gradient_boosting.model"))

VSZ: 1385732
RSS: 941856

After enwiki2 = Model.load(open("models/enwiki.articletopic.gradient_boosting.model"))

VSZ: 1464596
RSS: 1020768

This is higher memory usage than I think we are really prepared for. After loading all of the models, it ends up being about 3x as much memory as we needed before. As we can see from the final load, that memory gets shared relatively straightforwardly, but it is still too much.

I wonder if we can use gensim's memory-map mode to get around this. Alternatively, we can reduce the dimensions of our vectors or reduce the size of the vocabulary.

Halfak mentioned this in T243522: Reduce memory footprint of topic models.Jan 23 2020, 4:03 PM

Change 567120 had a related patch set uploaded (by Halfak; owner: Halfak):
[research/ores/wheels@master] Updates for revscoring 2.6.5

https://gerrit.wikimedia.org/r/567120

gerritbot added a project: Patch-For-Review.Jan 24 2020, 7:08 PM

Change 567120 merged by Accraze:
[research/ores/wheels@master] Updates for revscoring 2.6.5

https://gerrit.wikimedia.org/r/567120

Maintenance_bot removed a project: Patch-For-Review.Jan 24 2020, 9:10 PM

Change 567143 had a related patch set uploaded (by Halfak; owner: Halfak):
[mediawiki/services/ores/deploy@master] New draft topic models with 50d vectors.

https://gerrit.wikimedia.org/r/567143

gerritbot added a project: Patch-For-Review.Jan 24 2020, 9:23 PM

Change 567143 merged by Halfak:
[mediawiki/services/ores/deploy@master] New draft topic models with 50d vectors.

https://gerrit.wikimedia.org/r/567143

Maintenance_bot removed a project: Patch-For-Review.Jan 24 2020, 10:10 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-03T21:01:24Z] <halfak@deploy1001> Started deploy [ores/deploy@50a101a]: T243451

Mentioned in SAL (#wikimedia-operations) [2020-02-03T21:14:09Z] <halfak@deploy1001> Finished deploy [ores/deploy@50a101a]: T243451 (duration: 12m 47s)

akosiaris mentioned this in T242705: ORES uwsgi consumes a large amount of memory and CPU when shutting down (as part of a restart).Feb 5 2020, 8:13 AM

Halfak moved this task from Pending deployment to Completed on the Machine-Learning-Team (Active Tasks) board.Feb 5 2020, 4:26 PM

Halfak closed this task as Resolved.Feb 5 2020, 4:27 PM

Halfak claimed this task.

Halfak closed subtask T243522: Reduce memory footprint of topic models as Resolved.

Halfak closed subtask T243108: Add new vectors to deployment assets as Resolved.

Halfak closed subtask T243107: Retrain enwiki drafttopic models on supervised vectors as Resolved.

Halfak closed subtask T242345: Implement English pronoun count features in topic models as Resolved.

Halfak closed subtask T235184: Generate word vectors for ar, cs, en, and ko using FastText as Resolved.

Halfak closed subtask T235181: Build WikiProject directory topic models for ar, cs, and kowiki as Resolved.