Page MenuHomePhabricator
Create Task
Maniphest T183410

Create base docker image for ORES
Closed, ResolvedPublic
Actions

Description

Port ORES service provisioning to the Docker framework. For the rough pass, it's good enough to emulate what's happening inside of Travis-CI:

https://github.com/wiki-ai/ores/blob/master/.travis.yml

Here are some ways you might be able to create a reference installation:

Validate dependencies and basic sanity by running:

pytest

You can validate your installation by running:

./utility test_api http://localhost:8080 --debug

Related Objects
Search...

Event Timeline

Zppix created this task.Dec 20 2017, 7:33 PM
awight renamed this task from Create base docker image and set env vars for Wiki-AI CI to Create base docker image for ORES.May 18 2018, 1:32 PM
awight removed awight as the assignee of this task.
awight updated the task description. (Show Details)
awight added a subscriber: Oriolsoriano.
awight updated the task description. (Show Details)May 18 2018, 1:35 PM
awight updated the task description. (Show Details)May 18 2018, 1:37 PM
Oriolsoriano added a comment.May 18 2018, 5:44 PM

Submitted a patch including an initial approach to the ORES service Dockerfile. Please, feel free to discuss anything about it :)
Don't know how your usual code-review flow, so sorry for the duplicity between here & github:
https://github.com/wiki-ai/ores/pull/251

Also added a Makefile. I typically add them to my projects solely for convenience purposes. I am aware it is very opinionated, you call.

Considerations:

  • Used 'python:3.5-slim' from python's oficial repo @ DockerHub as a base image. That's python 3.5.5 on top of Debian jessie. If this needs to be changed, please, let me know.
  • Looking at the travis file, there seems to be a redis server running. If that is a requirement, I'd suggest having that in another container.
  • In the case of requiring various services (as the redis), I'd suggest ddding a docker-compose file for dev purposes (it's convenient). If the trend to move to kubernetes pushes hard, minikube is also an option.
  • Why is nodejs needed along python?

Issues:

  • When testing the installation, there seems to be something wrong/missing:
$ ./utility test_api http://localhost:8080 --debug
2018-05-18 15:57:13,313 DEBUG:ores.utilities.test_api -- Requesting /
2018-05-18 15:57:13,326 DEBUG:requests.packages.urllib3.connectionpool -- Starting new HTTP connection (1): localhost
2018-05-18 15:57:13,330 DEBUG:requests.packages.urllib3.connectionpool -- http://localhost:8080 "GET / HTTP/1.1" 301 241
2018-05-18 15:57:14,010 DEBUG:requests.packages.urllib3.connectionpool -- http://localhost:8080 "GET /w/index.php HTTP/1.1" 301 0
2018-05-18 15:57:15,015 DEBUG:requests.packages.urllib3.connectionpool -- http://localhost:8080 "GET /wiki/Main_Page HTTP/1.1" 200 4419
2018-05-18 15:57:15,016 DEBUG:ores.utilities.test_api -- Requesting /ui/
2018-05-18 15:57:15,018 DEBUG:requests.packages.urllib3.connectionpool -- Starting new HTTP connection (1): localhost
2018-05-18 15:57:15,020 DEBUG:requests.packages.urllib3.connectionpool -- http://localhost:8080 "GET /ui/ HTTP/1.1" 404 201
Traceback (most recent call last):
  File "./utility", line 4, in <module>
    ores.main()
  File "/home/vagrant/src/ores/ores/ores.py", line 57, in main
    module.main(sys.argv[2:])
  File "/home/vagrant/src/ores/ores/utilities/test_api.py", line 35, in main
    make_request(ores_url, "/ui/")
  File "/home/vagrant/src/ores/ores/utilities/test_api.py", line 142, in make_request
    response.status_code, path, response.content)
AssertionError: Status code mismatch 404 for /ui/: b'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /ui/ was not found on this server.</p>\n</body></html>\n'

Beyond that, the image has all the depencies & is usable as it is.

awight added a project: Wikimania-Hackathon-2018.May 18 2018, 10:02 PM
awight mentioned this in rORES098f428e2154: T183410/docker base image (#251).May 19 2018, 2:41 PM
awight closed this task as Resolved.May 19 2018, 3:30 PM
awight assigned this task to Oriolsoriano.
awight mentioned this in T195073: Use docker for ORES travis-ci tests.
awight added a parent task: T195073: Use docker for ORES travis-ci tests.
awight edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.
awight moved this task from Parked to Completed on the Machine-Learning-Team (Active Tasks) board.
awight edited projects, added Wikimedia-Hackathon-2018; removed Wikimania-Hackathon-2018.May 19 2018, 5:23 PM
awight moved this task from Backlog to Project on the Wikimedia-Hackathon-2018 board.
Quiddity mentioned this in T195219: Wikimedia Hackathon 2018 Showcase.May 20 2018, 5:16 PM
Halfak awarded a token.May 21 2018, 8:31 AM
Halfak added a comment.May 21 2018, 8:40 AM

This is really great. I'll work with @awight to review. Thank you for your work. I hope we can turn this into a set of containers that we might be able to run in production. See https://github.com/wiki-ai/ores-wmflabs-deploy for a production-ish version of our system. ores_wsgi.py and ores_celery.py provide service-starters for web and compute workers respectively.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits