Page MenuHomePhabricator

webservice2 failing to start Python web services with xml.etree.ElementTree.ParseError
Closed, ResolvedPublic

Description

My Python web API (https://tools.wmflabs.org/phabricator-bug-status/queryTasks) stopped working for unclear reasons.

When I try:

tools.phabricator-bug-status@tools-trusty:~$ webservice2 uwsgi-python restart

I get:

Traceback (most recent call last):
  File "/usr/local/bin/webservice2", line 150, in <module>
    job_id = get_job_id(queue_name, job_name)
  File "/usr/local/bin/webservice2", line 103, in get_job_id
    return xpath_string(output, './/JB_job_number')
  File "/usr/local/bin/webservice2", line 85, in xpath_string
    xml = ET.fromstring(string)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1300, in XML
    parser.feed(text)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1642, in feed
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 97, column 16

It looks like ET.fromstring is failing in /usr/local/bin/webservice2, probably because qstat_xml is not returning valid XML.

Event Timeline

Mattflaschen-WMF raised the priority of this task from to Needs Triage.
Mattflaschen-WMF updated the task description. (Show Details)

You're right, qstat_xml() does not return valid XML :-). In labs/toollabs's www/content/status.php, we worked around that with sed. I just tested doing that in Python and'll upload the patch soon. As part of testing, I restarted the webservice:

tools.phabricator-bug-status@tools-login:~$ /tmp/webservice2 uwsgi-python start
Your webservice is already running
tools.phabricator-bug-status@tools-login:~$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
8557644 0.31114 uwsgi-pyth tools.phabri r     03/02/2015 07:52:28 webgrid-generic@tools-webgrid-     1        
tools.phabricator-bug-status@tools-login:~$ /tmp/webservice2 uwsgi-python restart
Restarting
tools.phabricator-bug-status@tools-login:~$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
8816144 0.30000 uwsgi-pyth tools.phabri r     03/10/2015 13:49:07 webgrid-generic@tools-webgrid-     1        
tools.phabricator-bug-status@tools-login:~$

So if something's broken in the interim, please scream loudly :-).

Change 195556 had a related patch set uploaded (by Tim Landscheidt):
Tools: Fix XML output of qstat for webservice2

https://gerrit.wikimedia.org/r/195556

Change 195556 merged by Yuvipanda:
Tools: Fix XML output of qstat for webservice2

https://gerrit.wikimedia.org/r/195556

Seems to work - I tested on a couple of services :)

!close

yuvipanda claimed this task.
yuvipanda reassigned this task from yuvipanda to scfc.
yuvipanda set Security to None.