Page MenuHomePhabricator
Create Task
Maniphest T197097

Migrate models to LFS
Closed, ResolvedPublic
Actions

Description

Plan and migrate all model files into git-lfs, to make development and distribution more convenient.

Related Objects
Search...

StatusSubtypeAssignedTask
 DuplicateNoneT197180 Try to increase ORES deployment parallelism
 ResolvedLadsgroupT197096 [Epic] Use LFS for large ORES files
 ResolvedLadsgroupT197097 Migrate models to LFS

Event Timeline

awight created this task.Jun 13 2018, 11:03 AM
awight moved this task from Unsorted to Maintenance/cleanup on the Machine-Learning-Team board.Jun 20 2018, 2:48 PM
• Vvjjkkii renamed this task from Migrate models to LFS to e4aaaaaaaa.Jul 1 2018, 1:04 AM
• Vvjjkkii triaged this task as High priority.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from e4aaaaaaaa to Migrate models to LFS.Jul 2 2018, 1:55 PM
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
CommunityTechBot added a subscriber: Aklapper.
Ladsgroup claimed this task.Aug 20 2018, 1:20 PM
Ladsgroup edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptAug 20 2018, 1:20 PM
Ladsgroup added a comment.Aug 20 2018, 5:38 PM

I'm reshaping all of the repos to use git lfs. Took a back up https://analytics.wikimedia.org/datasets/archive/public-datasets/all/ores/20180820/ and also will keep another backup in my laptop

Ladsgroup moved this task from Parked to Pending deployment on the Machine-Learning-Team (Active Tasks) board.Aug 20 2018, 7:07 PM

All of the repos with model files now are using git lfs. Deploying this to prod is blocked on T192042

Ladsgroup mentioned this in T192042: Create gerrit mirrors for all github-based ORES repos.Aug 20 2018, 9:36 PM
Stashbot subscribed.Aug 22 2018, 10:30 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-22T10:30:25Z] <Amir1> deploying ores gerrit:454283 to beta (T197097)

Ladsgroup moved this task from Incoming to In progress on the User-Ladsgroup board.Aug 22 2018, 6:08 PM
Stashbot added a comment.Aug 22 2018, 8:05 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-22T20:05:55Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@8ff4da1]: Update ORES to use git lfs for model T197097

Stashbot added a comment.Aug 22 2018, 8:37 PM

Mentioned in SAL (#wikimedia-operations) [2018-08-22T20:37:37Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@8ff4da1]: Update ORES to use git lfs for model T197097 (duration: 31m 42s)

Ladsgroup moved this task from Pending deployment to Completed on the Machine-Learning-Team (Active Tasks) board.Aug 22 2018, 8:41 PM
awight added a comment.EditedAug 23 2018, 1:00 AM

Something for the documentation—I got the same "smudge" error as you reported earlier, but on a different repo and object. A git submodule sync seems to have solved everything in my case (though I'm still waiting an hour plus or minus two hours to download the objects ;-)

Ladsgroup closed this task as Resolved.Nov 1 2018, 8:00 PM
hashar added a project: git-lfs.Thu, Apr 4, 8:25 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL