Page MenuHomePhabricator

Create base docker image for ORES
Closed, ResolvedPublic

Description

Port ORES service provisioning to the Docker framework. For the rough pass, it's good enough to emulate what's happening inside of Travis-CI:

https://github.com/wiki-ai/ores/blob/master/.travis.yml

Here are some ways you might be able to create a reference installation:

Validate dependencies and basic sanity by running:

pytest

You can validate your installation by running:

./utility test_api http://localhost:8080 --debug

Event Timeline

awight renamed this task from Create base docker image and set env vars for Wiki-AI CI to Create base docker image for ORES.May 18 2018, 1:32 PM
awight removed awight as the assignee of this task.
awight updated the task description. (Show Details)
awight added a subscriber: Oriolsoriano.

Submitted a patch including an initial approach to the ORES service Dockerfile. Please, feel free to discuss anything about it :)
Don't know how your usual code-review flow, so sorry for the duplicity between here & github:
https://github.com/wiki-ai/ores/pull/251

Also added a Makefile. I typically add them to my projects solely for convenience purposes. I am aware it is very opinionated, you call.

Considerations:

  • Used 'python:3.5-slim' from python's oficial repo @ DockerHub as a base image. That's python 3.5.5 on top of Debian jessie. If this needs to be changed, please, let me know.
  • Looking at the travis file, there seems to be a redis server running. If that is a requirement, I'd suggest having that in another container.
  • In the case of requiring various services (as the redis), I'd suggest ddding a docker-compose file for dev purposes (it's convenient). If the trend to move to kubernetes pushes hard, minikube is also an option.
  • Why is nodejs needed along python?

Issues:

  • When testing the installation, there seems to be something wrong/missing:
$ ./utility test_api http://localhost:8080 --debug
2018-05-18 15:57:13,313 DEBUG:ores.utilities.test_api -- Requesting /
2018-05-18 15:57:13,326 DEBUG:requests.packages.urllib3.connectionpool -- Starting new HTTP connection (1): localhost
2018-05-18 15:57:13,330 DEBUG:requests.packages.urllib3.connectionpool -- http://localhost:8080 "GET / HTTP/1.1" 301 241
2018-05-18 15:57:14,010 DEBUG:requests.packages.urllib3.connectionpool -- http://localhost:8080 "GET /w/index.php HTTP/1.1" 301 0
2018-05-18 15:57:15,015 DEBUG:requests.packages.urllib3.connectionpool -- http://localhost:8080 "GET /wiki/Main_Page HTTP/1.1" 200 4419
2018-05-18 15:57:15,016 DEBUG:ores.utilities.test_api -- Requesting /ui/
2018-05-18 15:57:15,018 DEBUG:requests.packages.urllib3.connectionpool -- Starting new HTTP connection (1): localhost
2018-05-18 15:57:15,020 DEBUG:requests.packages.urllib3.connectionpool -- http://localhost:8080 "GET /ui/ HTTP/1.1" 404 201
Traceback (most recent call last):
  File "./utility", line 4, in <module>
    ores.main()
  File "/home/vagrant/src/ores/ores/ores.py", line 57, in main
    module.main(sys.argv[2:])
  File "/home/vagrant/src/ores/ores/utilities/test_api.py", line 35, in main
    make_request(ores_url, "/ui/")
  File "/home/vagrant/src/ores/ores/utilities/test_api.py", line 142, in make_request
    response.status_code, path, response.content)
AssertionError: Status code mismatch 404 for /ui/: b'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /ui/ was not found on this server.</p>\n</body></html>\n'

Beyond that, the image has all the depencies & is usable as it is.

This is really great. I'll work with @awight to review. Thank you for your work. I hope we can turn this into a set of containers that we might be able to run in production. See https://github.com/wiki-ai/ores-wmflabs-deploy for a production-ish version of our system. ores_wsgi.py and ores_celery.py provide service-starters for web and compute workers respectively.