Page MenuHomePhabricator

task not run via crontab
Closed, ResolvedPublic

Description

The cronjob tasks on etwikibots projects have stopped running some months ago.

Error log is emty.

Have left only one line in crontab. But its still not running.

00 10 * * *   jsub /data/project/etwikibots/bin/kirjutaks.sh > /dev/null

Event Timeline

tools-bastion-03:/data/project/etwikibots/bots/twitter# python kirjutaks.py
Traceback (most recent call last):
  File "kirjutaks.py", line 5, in <module>
    from bs4 import BeautifulSoup
ImportError: No module named bs4
tools-bastion-03:/data/project/etwikibots/bots/twitter# echo $?
1

I don't see the module available on bastion-03 at least, and I'm not sure when this last ran successfully. @bd808 could this in some way be related to the rewrite in python?

tools-bastion-03:/data/project/etwikibots/bots/twitter# python kirjutaks.py
Traceback (most recent call last):
  File "kirjutaks.py", line 5, in <module>
    from bs4 import BeautifulSoup
ImportError: No module named bs4
tools-bastion-03:/data/project/etwikibots/bots/twitter# echo $?
1

I don't see the module available on bastion-03 at least, and I'm not sure when this last ran successfully.

The kirjutaks.sh script is running kirjutaks.py using a virtualenv in /data/project/etwikibots/env. That venv has these packages installed:

argparse (1.2.1)
beautifulsoup4 (4.4.1)
lxml (3.5.0)
oauthlib (1.0.3)
pip (1.5.4)
requests (2.9.1)
requests-oauthlib (0.6.0)
setuptools (2.2)
six (1.10.0)
tweepy (3.5.0)
wsgiref (0.1.2)

Running via the venv does not have the ImportError:

$ /data/project/etwikibots/env/bin/python kirjutaks.py
WARNING: /data/project/etwikibots/env/local/lib/python2.7/site-packages/bs4/__init__.py:166: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

To get rid of this warning, change this:

 BeautifulSoup([your markup])

to this:

 BeautifulSoup([your markup], "lxml")

  markup_type=markup_type))

WARNING: /data/project/etwikibots/env/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning

WARNING: /data/project/etwikibots/env/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning

WARNING: /data/project/etwikibots/env/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning

WARNING: /data/project/etwikibots/env/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning

@bd808 could this in some way be related to the rewrite in python?

Its possible, but I'm not seeing any obvious reason for that.

Last write to the .err file was may 9th

tools.etwikibots@tools-bastion-03:~$ ls -l /data/project/etwikibots/log/kirjutaks.err
-rw-rw---- 1 tools.etwikibots tools.etwikibots 317340 May  9 10:00 /data/project/etwikibots/log/kirjutaks.err

We deployed the jsub rewrite on may 10th, so it's likely to be somehow related.

I have edited the crontab to redirect output to /data/project/etwikibots/crontab.log rather than /dev/null. Hopefully that clarifies what's going on. Maybe the rewrite code prints errors to stdout instead of stderr?

This is caused by the jsub rewrite.

$ JOBUTILS_QSUB=echo jsub /data/project/etwikibots/bin/kirjutaks.sh
[Tue Jun 21 15:56:37 2016] Error options embedded in /data/project/etwikibots/bin/kirjutaks.sh: argument -l: invalid argparse_resource value: 'h_vmem=250M # max memory use, including shared libs'

The parser for embedded SGE config comments is choking on the trailing comments:

#!/bin/bash
#$ -l h_rt=0:30:00  # max runtime
#$ -l h_vmem=250M # max memory use, including shared libs
#$ -l release=trusty
#$ -N kirjutaks # name of the task
#$ -M an_email@some.domain
#$ -m a # mail only on abort
#$ -b y # mount net drives
#$ -o /dev/null # output to /dev/null
#$ -e /data/project/etwikibots/log/kirjutaks.err

Change 295361 had a related patch set uploaded (by BryanDavis):
jsub: Strip trailing comments from jsubrc and embedded options

https://gerrit.wikimedia.org/r/295361

Change 295361 merged by jenkins-bot:
jsub: Update jsubrc and embedded options parsing

https://gerrit.wikimedia.org/r/295361

Mentioned in SAL [2016-06-21T16:49:34Z] <bd808> Updated jobutils to v1.14 for T138178

bd808 edited projects, added Toolforge; removed Patch-For-Review.

@WikedKentaur I'd appreciate it if you could verify that the update to jsub fixes your job execution problem.

Task runs now via cron. Thanks.