Page MenuHomePhabricator
Create Task
Maniphest T197096

[Epic] Use LFS for large ORES files
Closed, ResolvedPublic
Actions

Description

  • Models
  • Datasets
  • Reference data such as embeddings

We need to plan these migrations, they probably involve rewriting repository history among other nastiness.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Bump wheels to HEADLadsgroupmediawiki/services/ores/deploymaster+2 -2
Enable LFS for research/ores/wheelsLadsgroupAll-Projectsrefs/meta/config+5 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

awight created this task.Jun 13 2018, 11:01 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 13 2018, 11:01 AM
awight added a subtask: T192617: Consider committing all non-private datasets to the repos.Jun 13 2018, 11:02 AM
awight added a parent task: T197180: Try to increase ORES deployment parallelism.Jun 13 2018, 8:16 PM
awight edited projects, added: Machine-Learning-Team (Active Tasks); removed: Machine-Learning-Team.Jun 13 2018, 11:29 PM
Halfak edited projects, added: Machine-Learning-Team; removed: Machine-Learning-Team (Active Tasks).Jun 18 2018, 1:48 PM
awight moved this task from Unsorted to Maintenance/cleanup on the Machine-Learning-Team board.Jun 20 2018, 2:48 PM
• Vvjjkkii renamed this task from [Epic] Use LFS for large ORES files to f4aaaaaaaa.Jul 1 2018, 1:04 AM
• Vvjjkkii triaged this task as High priority.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from f4aaaaaaaa to [Epic] Use LFS for large ORES files.Jul 2 2018, 1:55 PM
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
CommunityTechBot added a subscriber: Aklapper.
Ladsgroup claimed this task.Aug 20 2018, 1:19 PM
Ladsgroup edited projects, added: Machine-Learning-Team (Active Tasks); removed: Machine-Learning-Team.
Ladsgroup subscribed.

Already on it.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptAug 20 2018, 1:19 PM
Ladsgroup moved this task from Parked to Pending deployment on the Machine-Learning-Team (Active Tasks) board.Aug 22 2018, 11:44 AM
Ladsgroup moved this task from Incoming to In progress on the User-Ladsgroup board.Aug 22 2018, 6:08 PM
Ladsgroup moved this task from Pending deployment to Parked on the Machine-Learning-Team (Active Tasks) board.Aug 22 2018, 8:41 PM
Ladsgroup added a comment.Aug 22 2018, 8:45 PM

I put this back to active because I want to turn wheels to git lfs as well, since it's growing really quickly and it's already 220MB since it's binary files and git grows horribly with files it can't delta.

gerritbot subscribed.Aug 27 2018, 7:34 PM

Change 455629 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[All-Projects@refs/meta/config] Enable LFS for research/ores/wheels

https://gerrit.wikimedia.org/r/455629

gerritbot added a project: Patch-For-Review.Aug 27 2018, 7:34 PM
gerritbot added a comment.Aug 30 2018, 6:25 PM

Change 455629 merged by 20after4:
[All-Projects@refs/meta/config] Enable LFS for research/ores/wheels

https://gerrit.wikimedia.org/r/455629

awight added a comment.EditedOct 10 2018, 11:26 PM

This task should be deduped or linked to T171619: [Epic] ORES should use a git large file plugin for storing serialized binaries

Ladsgroup closed subtask T197605: Confirm that continuous integration is compatible with LFS as Resolved.Nov 1 2018, 7:59 PM
Ladsgroup closed subtask T197097: Migrate models to LFS as Resolved.
Ladsgroup moved this task from Parked to Pending deployment on the Machine-Learning-Team (Active Tasks) board.Nov 7 2018, 6:52 PM

Now I rewrote whole history of wheels, that was the last bit. Now I'm going to deploy it.

gerritbot added a comment.Nov 7 2018, 7:02 PM

Change 472230 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/services/ores/deploy@master] Bump wheels to HEAD

https://gerrit.wikimedia.org/r/472230

Ladsgroup mentioned this in rORESDEPLOY25dfa4f5488a: Bump wheels to HEAD.Nov 7 2018, 7:05 PM
gerritbot added a comment.Nov 7 2018, 7:06 PM

Change 472230 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Bump wheels to HEAD

https://gerrit.wikimedia.org/r/472230

Stashbot mentioned this in T191842: Deployment git server can't supply ORES hosts in parallel.Nov 7 2018, 9:03 PM
Stashbot subscribed.

Mentioned in SAL (#wikimedia-operations) [2018-11-07T21:03:28Z] <ladsgroup@deploy1001> Started deploy [ores/deploy@25dfa4f]: T191842 T197096

Stashbot added a comment.Nov 7 2018, 9:20 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-07T21:20:52Z] <ladsgroup@deploy1001> Finished deploy [ores/deploy@25dfa4f]: T191842 T197096 (duration: 17m 24s)

Ladsgroup moved this task from Pending deployment to Completed on the Machine-Learning-Team (Active Tasks) board.Nov 7 2018, 9:21 PM
Ladsgroup closed this task as Resolved.Dec 17 2018, 12:39 PM
Halfak closed subtask T192617: Consider committing all non-private datasets to the repos as Declined.Apr 2 2019, 9:20 PM
hashar added a project: git-lfs.Apr 4 2024, 8:25 AM
Maintenance_bot removed a project: Patch-For-Review.Apr 4 2024, 8:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits