Oct 22 2020
You can trash everything! Sorry for the delayed response.
Nov 29 2019
Hi @Aklapper, Thanks for asking! I think this got stuck in code review. I'm happy to step in and move it forward once folks have time to code review it.
Mar 18 2019
Feb 15 2019
I've been lurking on this issue, and just wanted chime in with one bit of information I learned through experience. I'm not sure what model of Dell server you are using, but there are some subtle issues with fitting GPUs in a 2U server (and maybe others). In my case, a Nvidia GeForce card (which I know you aren't considering) technically fit in my 2U server, but the power location for the card meant it extruded beyond the enclosure.
Jan 9 2018
That makes sense. There are plenty of other avenues I can explore without a GPU.
I have availability this week and next, but I think @Ottomata is right. It will be tough to do this work if it's attached to a production machine. How feasible is it to move it to one?
Jan 8 2018
@dr0ptp4kt Thanks for chiming in, and I hope the leave was restful!
Jan 6 2018
Hi all, I’m in a position to put a GPU to use now and am happy to help if I can. I want to make sure I understand where things stand: is the issue with the encumbered kernel driver or the open cl library? If it’s the former the situation is much more difficult.
Nov 28 2017
Hi all, I am reopening this. Hooray :)
Oct 25 2017
Yes! I think this is all set. I'm currently working on T174796 to create data needed for these instances.
Sep 13 2017
Sep 12 2017
Indeed, I now have Yarn access! Thanks @elukey!
Sep 2 2017
@JAllemandou, thanks for the pointers! I think there's a little confusion on this, though. I volunteered to productionize Navigation Vectors (see T174796). I'm happy to also work on clickstream once this is done, but I think it will take several months to wrap up Navigation Vectors because of my teaching commitments.
Aug 28 2017
Everything looks good now! Thanks for your quick help, @Ottomata! I'm going to close this ticket and get to work :)
Aug 25 2017
One follow-up: The Navigation Vectors project uses Hive queries, so I think I also need the analytics-privatedata-users role. Is this correct? If so, should I start a new ticket, or can that also be added to this ticket?
Aug 23 2017
Yes! I updated Help:SSH to indicate that DSA is being phased out.
@herron I am having some trouble logging in. I can get to bastion but not beyond. I'm suspicious that the key I gave you is a DSS key, not a DSA key. I requested a DSA key, but the public key starts with ssh-dss instead.
Aug 17 2017
@RobH, I've signed the L3. My wikitech username is "Shilad Sen" and my preferred shell username is shiladsen. I created a new public SSH key for the production environment, and it is below I think that should be everything!
Jul 12 2017
Also, I'll probably be using Docker images (we have a WikiBrain docker image). I presume that it's better to run the Docker image in a VM rather than on the host, but please let me know if that's not correct.
Apr 25 2017
I am happy to help with engineering on this if we can find a way to make that work. I've set up navigation-based word2vec pipelines in similar environments (PySpark, Oozie, etc.) in the past.
Apr 24 2017
Great! 24GB of memory and 4 cores would be great if that works for you.
Apr 21 2017
Good questions! The big files are statistical models. So they take a while to build (a day or two), but they can be easily recreated. I think your suggestion of swapping the VMs over time seems reasonable. My only thought is if we could have a little more wiggle room... perhaps 300GB.. that would substantially reduce the rate at which we had to turn over the images.
I think Aaron was saying that although 200GB would probably work right now, it would't hold Wikipedia for very long. 500GB would definitely last for 5 years. Somewhere in between those sizes would work for a few years. Two of the large stoarage VMs would be plenty initially.
Apr 20 2017
Just to follow up on this. Aaron's estimates are pretty accurate. The disk cached data structures require about 200GB for larger language editions right now. We would likely expand to 500GB over time (or if we require "more advanced" WikiBrain features). Is this possible?
Jan 26 2017
I have spent quite a bit of time on this over the past few years. I do have a service that I could make available as an endpoint. HOWEVER, from what I've seen in my projects a much better approach is combining the work of Ellery Wulczyn on navigation vectors (https://meta.wikimedia.org/wiki/Research:Wikipedia_Navigation_Vectors) with the "standard" content-based approaches from Wikipedia.
Apr 28 2015
Hello! I believe I listed the wrong Wikitech username. It should be "Shilad Sen" instead of just "Shilad". Are you able to change this? Sorry for the mistake!
Apr 25 2015
Apr 24 2015
As Aaron said, this software will push the limits of your largest VM (16GB). I'd feel much safer if I knew our system was sandboxed and had no possibility of affecting other software running on tools.wmflabs.org.