Page MenuHomePhabricator

fireflytools python Kubernetes pod stuck in CrashLoopBackOff state
Closed, ResolvedPublic

Assigned To
Authored By
bd808
Feb 19 2019, 12:49 AM
Referenced Files
F28252937: image.png
Feb 20 2019, 6:31 PM
F28252760: image.png
Feb 20 2019, 6:09 PM
F28248498: image.png
Feb 20 2019, 3:26 AM
F28246933: image.png
Feb 19 2019, 5:12 PM
F28245615: image.png
Feb 19 2019, 1:56 PM

Description

Traceback (most recent call last):
  File "/data/project/fireflytools/www/python/src/app.py", line 3, in <module>
    from flask import Flask, request, Response, render_template, redirect
ImportError: No module named 'flask'

Event Timeline

Mentioned in SAL (#wikimedia-cloud) [2019-02-19T00:49:58Z] <bd808> Stopped webservice; stuck in CrashLoopBackOff due to missing python dependency. $HOME/www/python/venv should be examined (T216461)

While trying to upgrade python, I am getting the error

image.png (69×926 px, 10 KB)

How can this be solved?
@bd808 , Please check whether you are facing the same issue as state by you in the task, I have recently installed flask module by using "pip install --upgrade flask" command.

While trying to upgrade python, I am getting the error

image.png (69×926 px, 10 KB)

How can this be solved?

It is a warning, not an error. The Docker image that we are currently using for python on the Toolforge Kubernetes cluster are based on a Debian Jessie base image. That base image provides Python 3.4. To upgrade to a newer version of python we will need to change the base image to Debian Stretch or newer. This is something that has to be done by the Toolforge admin team; it is not an upgrade that tool maintainers can do themselves.

@bd808 , Please check whether you are facing the same issue as state by you in the task, I have recently installed flask module by using "pip install --upgrade flask" command.

kubectl get pods will show you the state of your running pods. If the 'STATUS' column shows 'Running' rather than 'CrashLoopBackOff' then you have fixed the issue I reported here.

I think it's fixed.

image.png (79×729 px, 11 KB)

But when I select the link present in the page https://en.wikipedia.org/wiki/Special:LintErrors, it shows 404. You need to click the link for the word 'tool' in line 3 inorder to get that page. I don't know whether anything needs to be done for that link or not.

But when I select the link present in the page https://en.wikipedia.org/wiki/Special:LintErrors, it shows 404. You need to click the link for the word 'tool' in line 3 inorder to get that page. I don't know whether anything needs to be done for that link or not.

You have started the webservice using the default PHP 5.6 runtime:

$ webservice status
Your webservice of type php5.6 is running
$ cat service.manifest
# This file is used by toollabs infrastructure.
# Please do not edit manually at this time.
backend: kubernetes
distribution: Ubuntu
version: 2
web: php5.6

This tool actually seems to be built as a python uwsgi webservice with content in the $HOME/www/python/... directories. This means that you need to stop the php webservice container (webservice stop) and then start it again using the python3 + wsgi runtime (webservice --backend=kubernetes python start).

bd808 added a subscriber: Adithyak1997.

From the merged task at T216346: Facing an issue related to Toolforge Trusty:

I have recently got an email stating me to migrate Fireflytools to Kubernetes cluster. By referring to the page https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation, I started the migration. First I did the steps mentioned under the heading Move a grid engine webservice which I think has been successfully completed. Then I did the steps mentioned under Move a cron job after which I tried to do the steps mentioned under Rebuild virtualenv for python users. While doing those steps, the problem was that since there was no requirements.txt file, I was not able to upgrade to the latest version. Since the migration was only partially done, the tool is currently not working. I would like to know whether I can upgrade to latest version of Python without creating the requirements.txt file. The command qstat is also not working now.

From the merged task at T216346: Facing an issue related to Toolforge Trusty:

While doing those steps, the problem was that since there was no requirements.txt file, I was not able to upgrade to the latest version. Since the migration was only partially done, the tool is currently not working. I would like to know whether I can upgrade to latest version of Python without creating the requirements.txt file.

Yes, we just need to figure out which python dependencies are needed by the tool. One way to do that would be to get the pip freeze output from the old venv. I think this is actually what we are now working through in this task.

The command qstat is also not working now.

qstat does not print any output if not jobs owned by the invoking user are active on the grid. You can verify that by running qstat -u '*' or qstat -j '*' instead which will show you all of the jobs running on whichever grid the bastion you are accessing from is attached to.

image.png (71×1 px, 17 KB)

Does this have to do anything with the current issue?

image.png (71×1 px, 17 KB)

Does this have to do anything with the current issue?

No, this warning is expected. We use the same uwsgi server config for both the python2 and python3 containers. The uwsgi process gives this warning on startup, but it does not cause any problems for the service actually running.


It would be helpful if you posted error messages and other shell and log output as text rather than screenshots. Search engines can not parse the images, so these errors and whatever answers we come up with are difficult for others who may have the same issues to find when the search tools are not able to index all of the content. See the Phabricator Remarkup reference for examples of how to format code blocks.

Sorry. The whole problem is currently resolved. What I did was that I referred the file "/data/project/fireflytools/www/python/src/app.py" and installed all the required packages using the command "pip install <name_of_package>. The verification can be done from the link https://tools.wmflabs.org/fireflytools/linter/enwiki

Thanks for working through the problems @Adithyak1997!

Thanks for the help provided to resolved the issue @bd808 .