Page MenuHomePhabricator

Set up a docker-compose containing ORES services required to run tests
Closed, ResolvedPublic

Description

Build a docker-compose containing the services required to run ORES tests.
Services related to ORES seem to be:

  • ORES API
  • redis score cache
  • task queue (redis service)
  • ORES workers (using the same codebase as the API)

In order to validate that the setup is OK:

  • pytest
  • Run on the API service: ./utility applications.wsgi --config ./config/celery-test/
  • Run on a worker service: ./utility applications.celery --config ./config/celery-test/

Event Timeline

Minor update: we noticed that the existing Travis CI configuration only requires one redis service, and we point both the task queue and score cache at the same server.

Submitted a PR for this: https://github.com/wiki-ai/ores/pull/253

Add ORES docker-compose file for testing purposes

This starts up 3 services (containers):

  • ores-api
  • ores-worker
  • redis

The tests seem to pass: make test
ores-api runs: ./utility applications.wsgi --config ./config/celery-test/
ores-worker runs: ./utility applications.celery --config ./config/celery-test/
And visiting the url http://localhost:8080/v2/scores/testwiki/revid/641962088?features=true does return results.

Considerations:

  • For simplicity, both the task queue and the score cache are pointed at the same redis. It can be split into 2 when needed.
  • I was skeptical about adding changes to test config files without breaking anything, so I added a new set of config files 'celery-test-docker'. It adds a 'docker_celery' to scoring_systems, a 'docker_redis' to score_caches and sets ores to use 'docker_celery' as its scoring system.
  • Celery seems to complain very hard when the service is run as root (which makes sense). I set the env var C_FORCE_ROOT to true to get it to start. Probably the best would be adding the user to application is supposed to be run by to the docker image.

This - plus all the stuff that I might have missed - should suffice to setup a dockerized dev environment.
Additionally, it should make the transition to kubernetes easier :) (transitioning from docker-compose to kube is easy)

Vvjjkkii renamed this task from Set up a docker-compose containing ORES services required to run tests to iocaaaaaaa.Jul 1 2018, 1:09 AM
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
Mainframe98 renamed this task from iocaaaaaaa to Set up a docker-compose containing ORES services required to run tests.Jul 1 2018, 7:27 AM
Mainframe98 raised the priority of this task from High to Needs Triage.
Mainframe98 updated the task description. (Show Details)
Mainframe98 added a subscriber: Aklapper.
awight claimed this task.

Isn't it done?

Thanks for the nudge!