On Thu, 20 Jul 2017, at 10:02 PM, John Beard wrote:
> Hi!
>
> The BUB tool looks great for gettings works from Google or Hathi onto
> Commons via IA, which is a very common and time consuming task.
> However, it doesn't seem to have uploaded anything to IA since Sept
> 2016, and the queue is around 500 books long.
>
> Is there anything I can do to help get this useful tool back on the road?
>
> Cheers,
>
> John
The full queue can be viewed at http://tools.wmflabs.org/bub/queue/1000/
Formerly reported at: https://github.com/rohit-dua/BUB/issues/53
The main backend process is worker.py. Launching it with qstat has no effect. Running worker.py directly outputs:
```
~/public_html/BUB/bot$ python worker.py
Traceback (most recent call last):
File "worker.py", line 4522, in <module>
from retry import retry, ia_online, requestsMySQLdb
File "/data/project/bub/public_html/BUB/utils/retrybuild/bdist.linux-x86_64/egg/MySQLdb/__init__.py", line 19, in <module>
File "build/bdist.linux-x86_64/egg/_mysql.py", line 7, in <module>
File "build/bdist.linux-x86_64/egg/_mysql.py", line 3, in __bootstrap__
File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 36, in <module>
import email.parser
File "/usr/lib/python2.7/email/parser.py", line 12, in <module>
from email.feedparser import FeedParser
File "/usr/lib/python2.7/email/feedparser.py", line 27, in <module>
from email import message
File "/usr/lib/python2.7/email/message.py", line 16, in <module>
import email.charset
File "/usr/lib/python2.7/email/charset.py", line 13, in <module>
import email.base64mime
File "/usr/lib/python2.7/email/base64mime.py", line 40, in <module>
from email.utils import fix_eols
File "/usr/lib/python2.7/email/utils.py", line 267, in <module>
requests.packages.urllib3.disable_warnings()import random
File "/usr/lib/python2.7/random.py", line 49, in <module>
import hashlib as _hashlib
File "build/bdist.linux-x86_64/egg/hashlib.py", line 115, in <module>
"""
TypeError: 'frozenset' object is not callable
/data/project/bub/public_html/BUB/flask/bin/python --version
AttributeError: 'module' object has no attribute 'disable_warnings'Python 2.7.6
```
The mass_worker.log contains (from the last time a restart of the queue processing was attempted, probably):
```
2017-02-23 21:26:10.651983 func:manager args:(<__main__.QueueHandler object at 0x2b207522b510>,){}
:<class '_mysql_exceptions.OperationalError'>, Retry in 1 seconds...
:<class '_mysql_exceptions.OperationalError'>, Retry in 2 seconds... object at 0x2b2b04395510>,){}
:<class '_mysql_exceptions.OperationalError'>, Retry in 1 seconds...
:<class '_mysql_exceptions.OperationalError'>, Retry in 2 seconds... :<class '_mysql_exceptions.OperationalError'>, Retry in 4 seconds...--------------------
ERROR: <class '_mysql_exceptions.OperationalError'> func:manager args:(<__main__.QueueHandler object at 0x2ab669bed490>,){}
nc:manager args:(<__main__.QueueHandler object at 0x2b0e2bfa0510>,){}
:<class '_mysql_exceptions.OperationalError'>, Retry in 4 seconds...--------------------
ERROR: <class '_mysql_exceptions.OperationalError'> func:manager args:(<__main__.QueueHandler object at 0x2b21d8752510>,){}
2017-03-18 18:01:47.189082 Started Mass Worker #1
2017-03-18 18:01:47.199334 Started wait_and_add_to_queue
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/bot/mass_worker.py", line 392, in wait_and_add_to_queue
info = json.loads(q_bulk_order.pop_and_remove())
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Unterminated string starting at: line 1 column 2 (char 1)
2017-03-18 18:03:42.202126 Started Mass Worker #3
orker3-2' already active
2017-03-18 18:05:03.356338 Started Mass Worker #3
2017-03-18 18:05:45.160743 Started Mass Worker #3
2017-03-24 14:02:26.740201 func:manager args:(<__main__.QueueHandler object at 0x2ae930bbc490>,){}
:<class '_mysql_exceptions.OperationalError'>, Retry in 1 seconds...
:<class '_mysql_exceptions.OperationalError'>, Retry in 2 seconds... :<class '_mysql_exceptions.OperationalError'>, Retry in 4 seconds...--------------------
ERROR: <class '_mysql_exceptions.OperationalError'> func:manager args:(<__main__.QueueHandler object at 0x2ae930bbc490>,){}
2018-02-24 15:12:06.920545 Started Mass Worker #3
2018-03-08 04:10:57.095180 func:manager args:(<__main__.QueueHandler object at 0x2addb1b4d490>,){}
:<class '_mysql_exceptions.OperationalError'>, Retry in 1 seconds...
2018-04-05 14:35:38.878704 func:manager args:(<__main__.QueueHandler object at 0x2b4653ec1490>,){}
:<class '_mysql_exceptions.OperationalError'>, Retry in 1 seconds...
```
And from upload_checker (the queue which handles the upload part after the images have been downloaded):
```
2017-10-26 14:24:15.042495 upload-checker.py started
Traceback (most recent call last):
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/bot/upload_checker.py", line 192, in <module>
main()
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/bot/upload_checker.py", line 188, in main
check_if_upload_ready()
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/bot/upload_checker.py", line 157, in check_if_upload_ready
r = get_ia_metadata(ia_identifier)
File "../utils/retry.py", line 127, in f_retry
return f(*args, **kwargs)
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/bot/upload_checker.py", line 122, in get_ia_metadata
r = requests.get('http://archive.org/metadata/%s' %(ia_identifier) ).json()
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/flask/local/lib/python2.7/site-packages/requests/api.py", line 67, in get
return request('get', url, params=params, **kwargs)
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/flask/local/lib/python2.7/site-packages/requests/api.py", line 53, in request
return session.request(method=method, url=url, **kwargs)
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/flask/local/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/flask/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/flask/local/lib/python2.7/site-packages/requests/adapters.py", line 376, in send
timeout=timeout
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/flask/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/flask/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python2.7/httplib.py", line 1017, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1051, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 1013, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 864, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 826, in send
self.connect()
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/flask/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 162, in connect
conn = self._new_conn()
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/flask/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 137, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/mnt/nfs/labstore-secondary-tools-project/bub/public_html/BUB/flask/local/lib/python2.7/site-packages/requests/packages/urllib3/util/connection.py", line 81, in create_connection
sock.connect(sa)
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
```