adds/changes dumps did not complete for July 10 run
Closed, ResolvedPublic0 Estimated Story Points
Actions

Assigned To

Authored By

	ArielGlenn
	Jul 11 2019, 4:40 AM

Description

email from cron job:

Traceback (most recent call last):
  File "/srv/deployment/dumps/dumps/xmldumps-backup/generatemiscdumps.py", line 547, in <module>
    main()
  File "/srv/deployment/dumps/dumps/xmldumps-backup/generatemiscdumps.py", line 543, in main
    dump_all.do_all_wikis_til_done(3)
  File "/srv/deployment/dumps/dumps/xmldumps-backup/generatemiscdumps.py", line 323, in do_all_wikis_til_done
    (failures, todos) = self.do_run_on_all_wikis()
  File "/srv/deployment/dumps/dumps/xmldumps-backup/generatemiscdumps.py", line 299, in do_run_on_all_wikis
    dump = MiscDumpOne(self.args, wikiname, self.flags, self.log)
  File "/srv/deployment/dumps/dumps/xmldumps-backup/generatemiscdumps.py", line 192, in __init__
    self.wiki = Wiki(self.args['config'], wikiname)
  File "/srv/deployment/dumps/dumps-cache/revs/fd6100a5e98e125b8256f2fd4b9d0ac2d259a735/xmldumps-backup/dumps/wikidump.py", line 457, in __init__
    self.db_user, self.db_password = self.get_db_user_and_password()
  File "/srv/deployment/dumps/dumps-cache/revs/fd6100a5e98e125b8256f2fd4b9d0ac2d259a735/xmldumps-backup/dumps/wikidump.py", line 473, in get_db_user_and_password
    settings = json.loads(results.decode('utf-8'))
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Details

	Subject	Repo	Branch	Lines +/-
	handle exception when setting up Wiki object for misc dumps	operations/dumps	master	+9 -4

Customize query in gerrit

Event Timeline

ArielGlenn triaged this task as Medium priority.Jul 11 2019, 4:40 AM

ArielGlenn created this task.

I listed the directories to see which had been updated and which not:

ariel@dumpsdata1001:/data/otherdumps/incr$ ls -lt
total 3732
drwxrwxr-x 11 dumpsgen dumpsgen   4096 Jul 10 22:54 zeawiki
drwxrwxr-x 11 dumpsgen dumpsgen   4096 Jul 10 22:54 zawiki
drwxr-xr-x 11 dumpsgen dumpsgen   4096 Jul 10 22:54 yuewiktionary
drwxrwxr-x 11 dumpsgen dumpsgen   4096 Jul 10 22:54 yowiki
...
drwxrwxr-x 11 dumpsgen dumpsgen   4096 Jul 10 20:50 afwiki
drwxrwxr-x 11 dumpsgen dumpsgen   4096 Jul 10 20:50 adywiki
drwxrwxr-x 11 dumpsgen dumpsgen   4096 Jul 10 20:50 acewiki
drwxrwxr-x 11 dumpsgen dumpsgen   4096 Jul 10 20:50 abwiki
drwxrwxr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zh_classicalwiki
drwxrwxr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zhwikibooks
drwxrwxr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zh_min_nanwiki
drwxrwxr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zhwikivoyage
drwxrwxr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zhwikisource
drwxrwxr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zh_min_nanwiktionary
drwxrwxr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zuwiki
drwxrwxr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zhwiktionary
drwxrwxr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zhwikinews
drwxr-xr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zhwikiversity
drwxrwxr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zhwikiquote
drwxrwxr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zh_min_nanwikisource
drwxrwxr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zh_yuewiki
drwxrwxr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zhwiki
drwxrwxr-x 10 dumpsgen dumpsgen   4096 Jul 10 07:15 zuwiktionary
drwxrwxr-x  2 dumpsgen dumpsgen   4096 Jul  9 07:15 wikimania2018wiki
-rw-r--r--  1 dumpsgen dumpsgen 647258 May  3 13:17 index.html
drwxrwxr-x  2 dumpsgen dumpsgen   4096 Jan 11  2018 wikimania2017wiki

For the wikis dated Jul 10 07:xx I have run adds/changes and they are all now complete.

Change 522018 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/dumps@master] handle exception when setting up Wiki object for misc dumps

https://gerrit.wikimedia.org/r/522018

gerritbot added a project: Patch-For-Review.Jul 11 2019, 4:44 AM

I don't see any activity in SAL around the time of the failure, nor alerts from icinga in irc, so any glitch was so momentary it didn't set off any alarms. In any case, whatever it was, is obviously ok now since manual runs of the script completed without issue.

ArielGlenn moved this task from Backlog to Active on the Dumps-Generation board.Jul 11 2019, 4:52 AM

Shizhao added a project: Chinese-Sites.Jul 15 2019, 6:15 AM

Change 522018 merged by ArielGlenn:
[operations/dumps@master] handle exception when setting up Wiki object for misc dumps

https://gerrit.wikimedia.org/r/522018

Maintenance_bot removed a project: Patch-For-Review.Jul 19 2019, 3:10 PM

The above patch is deployed. If this occurs again we will at least get more information about it. Closing for now as there's not enough information to reproduce or track down the root cause.

ArielGlenn moved this task from Active to Blocked/Stalled/Waiting for event on the Dumps-Generation board.Jul 20 2019, 5:23 AM

ArielGlenn moved this task from Blocked/Stalled/Waiting for event to Done on the Dumps-Generation board.

Shizhao moved this task from Backlog to Closed on the Chinese-Sites board.Jul 22 2019, 3:16 AM

adds/changes dumps did not complete for July 10 runClosed, ResolvedPublic0 Estimated Story PointsActions

Description

Details

Event Timeline

adds/changes dumps did not complete for July 10 run
Closed, ResolvedPublic0 Estimated Story Points
Actions