Page MenuHomePhabricator

memento_client doesnt provide an informative error for network errors
Open, LowestPublic

Description

When there is a gateway error in the timegate server, Memento client response doesnt include what we expect, and doesnt raise an exception.

When we were using memento_info.get('mementos').get('closest').get('uri')[0], it failed with an AttributeError:

======================================================================
ERROR: test_newest (tests.weblinkchecker_tests.TestMementoWebCite)
Test WebCite for newest https://google.com.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/jayvdb/pywikibot-core/tests/weblinkchecker_tests.py", line 73, in test_newest
    archivedversion = self._get_archive_url('https://google.com')
  File "/home/travis/build/jayvdb/pywikibot-core/tests/weblinkchecker_tests.py", line 45, in _get_archive_url
    self.timegate_uri)
  File "/home/travis/build/jayvdb/pywikibot-core/scripts/weblinkchecker.py", line 181, in _get_closest_memento_url
    return memento_info.get('mementos').get('closest').get('uri')[0]
AttributeError: 'NoneType' object has no attribute 'get'

https://travis-ci.org/jayvdb/pywikibot-core/jobs/79818375#L3326

A patch to weblinkchecker.py has been merged to raise a more informative Exception. weblinkchecker.py doesn't have serious problem here, as it catches any error and falls back from Internet Archive -> WebCite -> user error.

Event Timeline

jayvdb created this task.Sep 11 2015, 11:19 AM
jayvdb raised the priority of this task from to Needs Triage.
jayvdb updated the task description. (Show Details)
jayvdb added a subscriber: jayvdb.
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptSep 11 2015, 11:19 AM

Another bunch of these in testInternetArchiveNewest and testInternetArchiveOlder
https://travis-ci.org/wikimedia/pywikibot-core/builds/80159182

jayvdb triaged this task as Unbreak Now! priority.Sep 16 2015, 10:42 PM

This is now causes many builds to break.

jayvdb lowered the priority of this task from Unbreak Now! to Medium.Sep 16 2015, 11:00 PM

Change 238999 had a related patch set uploaded (by John Vandenberg):
Safer use of mementos returned data structure

https://gerrit.wikimedia.org/r/238999

Xqt assigned this task to jayvdb.Sep 17 2015, 6:59 AM
Xqt added a subscriber: Xqt.

Change 238999 merged by jenkins-bot:
Safer use of mementos returned data structure

https://gerrit.wikimedia.org/r/238999

jayvdb closed this task as Resolved.Sep 17 2015, 8:17 AM
XZise added a subscriber: XZise.Sep 17 2015, 4:16 PM

Is this really fixed? Your patch “just” makes it easier for us, which get() failed.

jayvdb renamed this task from weblinkchecker _get_closest_memento_url to memento_client doesnt provide an informative error for network errors.Sep 17 2015, 7:52 PM
jayvdb reopened this task as Open.
jayvdb removed jayvdb as the assignee of this task.
jayvdb lowered the priority of this task from Medium to Lowest.
jayvdb updated the task description. (Show Details)
jayvdb edited projects, added Upstream; removed Patch-For-Review.
jayvdb set Security to None.

We are going to fix this upstream. Thanks for reporting it.

Restricted Application added a project: Internet-Archive. · View Herald TranscriptSep 1 2016, 3:20 PM
Restricted Application added a subscriber: TerraCodes. · View Herald Transcript

@Shawnmjones What is the status of this task in Memento Client?

Dvorapa moved this task from Backlog to Upstream issues on the Pywikibot board.
Dvorapa moved this task from Backlog to Upstream issues on the Pywikibot-tests board.