Page MenuHomePhabricator

Deploy drafttopic model to production ORES
Closed, ResolvedPublic

Details

Related Gerrit Patches:
mediawiki/services/ores/deploy : masterProvision the drafttopic model

Related Objects

Event Timeline

Halfak created this task.Sep 20 2017, 4:55 PM
Restricted Application added a project: Scoring-platform-team. · View Herald TranscriptSep 20 2017, 4:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Halfak triaged this task as Medium priority.Oct 9 2017, 9:13 PM
Halfak renamed this task from Deploy topic model to ORES to Deploy drafttopic model to ORES.Apr 16 2018, 2:26 PM
Halfak added a project: drafttopic-modeling.
Restricted Application added a project: Product-Analytics. · View Herald TranscriptApr 18 2018, 11:48 PM

Change 432000 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/services/ores/deploy@master] [WIP] Provision the drafttopic model

https://gerrit.wikimedia.org/r/432000

Change 432000 merged by Awight:
[mediawiki/services/ores/deploy@master] Provision the drafttopic model

https://gerrit.wikimedia.org/r/432000

awight renamed this task from Deploy drafttopic model to ORES to Deploy drafttopic model to production ORES.May 28 2018, 11:26 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-04T20:31:40Z] <awight@deploy1001> Started deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336

Mentioned in SAL (#wikimedia-operations) [2018-06-04T20:34:14Z] <awight@deploy1001> Finished deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (duration: 02m 35s)

Mentioned in SAL (#wikimedia-operations) [2018-06-04T20:40:10Z] <awight@deploy1001> Started deploy [ores/deploy@d77e52c]: l ores2001.codfw.wmnet ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)

Mentioned in SAL (#wikimedia-operations) [2018-06-04T20:40:15Z] <awight@deploy1001> Started deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)f

Mentioned in SAL (#wikimedia-operations) [2018-06-04T20:40:38Z] <awight@deploy1001> Finished deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)f (duration: 00m 22s)

Mentioned in SAL (#wikimedia-operations) [2018-06-04T20:48:39Z] <awight@deploy1001> Started deploy [ores/deploy@65e979f]: ores2001 canary of drafttopic; T176336 (take 3 after bumping revision)

Mentioned in SAL (#wikimedia-operations) [2018-06-04T20:50:26Z] <awight@deploy1001> Finished deploy [ores/deploy@65e979f]: ores2001 canary of drafttopic; T176336 (take 3 after bumping revision) (duration: 01m 47s)

Mentioned in SAL (#wikimedia-operations) [2018-06-06T10:15:14Z] <awight@deploy1001> Started deploy [ores/deploy@bf182e2]: ORES canary deployment to ores2002.codfw.wmnet; T176336

Mentioned in SAL (#wikimedia-operations) [2018-06-06T10:15:21Z] <awight@deploy1001> Finished deploy [ores/deploy@bf182e2]: ORES canary deployment to ores2002.codfw.wmnet; T176336 (duration: 00m 06s)

Mentioned in SAL (#wikimedia-operations) [2018-06-06T10:15:41Z] <awight@deploy1001> Started deploy [ores/deploy@65e979f]: ORES canary deployment to ores2002.codfw.wmnet; T176336

Mentioned in SAL (#wikimedia-operations) [2018-06-06T10:19:25Z] <awight@deploy1001> Finished deploy [ores/deploy@65e979f]: ORES canary deployment to ores2002.codfw.wmnet; T176336 (duration: 03m 44s)

awight added a subscriber: awight.Jun 6 2018, 10:22 AM

Drafttopic code is live on ores2002, the word2vec file has cloned successfully after applying a workaround. I'll monitor memory for a few hours, it's taken an expected dip and will hopefully stabilize.

ores2002 looks very much like ores2001 and seems to be in a stable garbage-collection cycle. Minimum headroom has dropped from c. 20GiB to 14GiB. CPU and other metrics are unchanged, however "scores errored" went from 0.005/min to 0.5 or so, which is annoying.

awight added a comment.Jun 6 2018, 2:29 PM

I've run the workaround creating ~deploy-service/.gitconfig on all production hosts.

Mentioned in SAL (#wikimedia-operations) [2018-06-06T15:01:08Z] <awight@deploy1001> Started deploy [ores/deploy@65e979f]: ORES: new draft topic model; T176336

Mentioned in SAL (#wikimedia-operations) [2018-06-06T15:26:45Z] <awight@deploy1001> Finished deploy [ores/deploy@65e979f]: ORES: new draft topic model; T176336 (duration: 25m 37s)

awight closed this task as Resolved.Jun 6 2018, 3:28 PM
awight claimed this task.
awight moved this task from Pending deployment to Done on the Scoring-platform-team (Current) board.