Quarry, unable to run tests following the README.md
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Abstract09
	May 16 2022, 8:28 PM

Description

Failed to run tests, following the readme here: https://github.com/wikimedia/analytics-quarry-web

Download and install Docker and docker-compose (already ships with docker on Windows and Mac)
Clone the Quarry repository
Run docker-compose up success, press Ctrl+C to stop.
Run docker-compose -f docker-compose-test.yml run --rm test

What happens?:
I have such errors:

WARN[0000] Found orphan containers ([web-worker-1 web-web-1 web-mywiki-1 web-db-1 web-redis-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/py/_error.py", line 65, in checked_call
    return func(*args, **kwargs)
PermissionError: [Errno 13] Permission denied: '/app/.tox/log'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/tox", line 11, in <module>
    load_entry_point('tox==2.5.0', 'console_scripts', 'tox')()
  File "/usr/lib/python3/dist-packages/tox/session.py", line 39, in main
    retcode = Session(config).runcommand()
  File "/usr/lib/python3/dist-packages/tox/session.py", line 337, in __init__
    config.logdir.ensure(dir=1)
  File "/usr/lib/python3/dist-packages/py/_path/local.py", line 524, in ensure
    return p._ensuredirs()
  File "/usr/lib/python3/dist-packages/py/_path/local.py", line 509, in _ensuredirs
    self.mkdir()
  File "/usr/lib/python3/dist-packages/py/_path/local.py", line 459, in mkdir
    py.error.checked_call(os.mkdir, fspath(p))
  File "/usr/lib/python3/dist-packages/py/_error.py", line 85, in checked_call
    raise cls("%s%r" % (func.__name__, args))
py.error.EACCES: [Permission denied]: mkdir('/app/.tox/log',)

I also tried docker-compose -f docker-compose-test.yml run --rm --abort-on-container-exit --exit-code-from test, and got

unknown flag: --abort-on-container-exit

What should have happened instead?:
No errors.

Software version (if not a Wikimedia wiki), browser information, screenshots, other information, etc.:
Docker version 20.10.15, build fd82621d35
Docker Compose version 2.5.0

Details

	Subject	Repo	Branch	Lines +/-
	README.md: update how to run tests	analytics/quarry/web	master	+13 -21

Customize query in gerrit

Related Objects

Mentioned Here: T295318: Fix Toolhub developer environment tooling to work on Linux hosts

Event Timeline

Abstract09 created this task.May 16 2022, 8:28 PM

Restricted Application added subscribers: RhinosF1, Aklapper. · View Herald TranscriptMay 16 2022, 8:28 PM

This doesn't sound like an issue in Quarry itself to me, but an issue with the local Docker setup? I do not think that the Quarry documentation should cover Docker Troubleshooting unrelated to Quarry itself...

@Aklapper: @rook confirmed they could reproduce via email too.

RhinosF1 added a subscriber: rook.May 16 2022, 8:59 PM

py.error.EACCES: [Permission denied]: mkdir('/app/.tox/log',)

This sounds a lot like T295318: Fix Toolhub developer environment tooling to work on Linux hosts

@Aklapper yes I think it is a bug in our docs for quarry.

Things, including testing things, were updated in 2021-09 I believe that is when blubber was introduced to quarry. I'm not sure if the line in the README is trying to do anything different than:
blubber .pipeline/blubber.yaml quarry-test | docker build --tag blubber-quarry:01 --file - . ; docker run blubber-quarry:01

(Docs on how to setup the blubber function found here: https://wikitech.wikimedia.org/wiki/Blubber/Download#Blubber_as_a_(micro)Service )

@bd808 does that seem right to you?

In T308493#7934300, @rook wrote:

@bd808 does that seem right to you?

That is basically what one would do to match how the pipelinelib tests work in Jenkins. My note about the error sounding similar to T295318 was based on that pattern matching.

Looking at the existing docker-compose config files and the readme again, it appears that currently a local custom Dockerfile is being used rather than the blubber generated one in the local dev & test setup. The error message about failing to write to /app/.tox/log still sounds like issues with host bind mount permissions matching the runtime permissions inside the container to me.

In T308493#7935097, @bd808 wrote:

That is basically what one would do to match how the pipelinelib tests work in Jenkins. My note about the error sounding similar to T295318 was based on that pattern matching.

Thanks!

It seems like we should have the local testing be the same as the jenkins testing. So dropping the docker-compose-test.yml file and updating the README.md with blubber instructions seems like a reasonable approach. I'm not certain that Jenkins isn't up to something else, but that would likely be an adventure for another ticket.

I'll take that route unless someone gets there first or wants to further discuss.

rook added a project: good first task.May 20 2022, 12:09 PM

rook added a project: Wikimedia-Hackathon-2022.May 21 2022, 2:55 PM

Change 794619 had a related patch set uploaded (by Jiyu; author: Jiyu):

[analytics/quarry/web@master] README.md: update how to run tests

https://gerrit.wikimedia.org/r/794619

gerritbot added a project: Patch-For-Review.May 21 2022, 5:07 PM

Abstract09 claimed this task.May 21 2022, 5:08 PM

Change 794619 merged by jenkins-bot:

[analytics/quarry/web@master] README.md: update how to run tests

https://gerrit.wikimedia.org/r/794619

rook closed this task as Resolved.May 21 2022, 6:00 PM