Page MenuHomePhabricator

Build ORES dependencies and store objects in repo
Closed, DeclinedPublic

Description

Custom built packages should be hosted in WMF repos.

This task is done when the dependencies are merged into a repo that we can safely pull from in production.

Event Timeline

awight raised the priority of this task from to Needs Triage.
awight updated the task description. (Show Details)
awight added a subscriber: Aklapper.

I'd say let's just keep it in wiki-ai/ for now?

As for debs, in labs we can just store them in an aptly setup.

@Halfak that's about writing the package files, etc. This is actually building the .debs and putting them in a place somewhere.

@awight, what's the status here? Anything I can help with?

It's me! I will try to get more of this going next week - I'm trying to
import all of them into gerrit...

We've been having a lot of trouble with sklearn. I was looking for alternatives to .deb. It looks like Python Wheels may be a nice alternative. They encapsulate a pre-compiled/pre-organized set of files streamlined for installation. The best part is that you can install without executing any arbitrary python code.

I was able to compile a whl on my laptop (Trusty 14.04) for sklearn, scp that file to stat1003.eqiad.wmnet (Trusty 14.04) and then install the whl without issue.

(3.4)[halfak@stat1003: ~/Downloads]
$ time pip install scikit_learn-0.17.dev0-cp34-cp34m-linux_x86_64.whl 
Unpacking ./scikit_learn-0.17.dev0-cp34-cp34m-linux_x86_64.whl
Installing collected packages: scikit-learn
Successfully installed scikit-learn
Cleaning up...

real	0m1.391s
user	0m1.249s
sys	0m0.119s
(3.4)[halfak@stat1003: ~/Downloads]
$ python
Python 3.4.0 (default, Apr 11 2014, 13:05:11) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from sklearn.ensemble import RandomForestClassifier
>>> rf = RandomForestClassifier()
>>> rf
RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
            max_depth=None, max_features='auto', max_leaf_nodes=None,
            min_samples_leaf=1, min_samples_split=2,
            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,
            oob_score=False, random_state=None, verbose=0,
            warm_start=False)

@Halfak hey can you document the trouble you're having with sklearn? The deb build works AFAIK, you just need to grab the right repo, whatever it says in the card...

Sorry! I was confused. I thought we were still stick on that deb. What task/docs do I reference to find out the necessary details to start testing on labs?

I've already built the package and added to the repo server!

What's a repo server? How do I access it?

Also, does that mean that this task is done? :D

I'm on my way out of town, maybe @yuvipanda can help? This task is not
quite done cos we're trying to do git build package layout so that
maintenance upgrades are simpler...

Halfak set Security to None.

@yuvipanda
Can you confirm, is this task still a high priority for production deployment, and the next steps are to finish adapting for gbp? Or did you have a way around .debs entirely?

Halfak renamed this task from Build ORES support debs, store objects and sources in WMF repos to Build ORES dependencies and store objects in repo.Mar 2 2016, 11:32 PM
Halfak moved this task from Paused to Backlog on the Machine-Learning-Team (Active Tasks) board.
Halfak updated the task description. (Show Details)
Halfak added a project: ORES.

Looks like we can do bdist wheels just fine. @schana, would you be interested in picking this up?