Page MenuHomePhabricator

badcontinue api response if timestamp is copied from continue
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • Use API via pywikibot in a way that requires continuation (via the continue parameter)
  • Example script:
import pywikibot as pwb

site = pwb.Site('wikidata', 'wikidata')
cat = pwb.Category(site, 'Wikidata status updates')
payload = list(cat.articles(namespaces=4, sortby='timestamp', reverse=True))

What happens?:

pywikibot.exceptions.APIError is raised

WARNING: API error badcontinue: Invalid continue param. You should pass the original value returned by the previous query.
Traceback (most recent call last):
  File "/home/msyn/pywikibot_scripts/../pywikibot/pwb.py", line 39, in <module>
    sys.exit(main())
  File "/home/msyn/pywikibot_scripts/../pywikibot/pwb.py", line 35, in main
    runpy.run_path(str(path), run_name='__main__')
  File "/usr/lib/python3.9/runpy.py", line 268, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/msyn/pywikibot_scripts/../pywikibot/pywikibot/scripts/wrapper.py", line 516, in <module>
    main()
  File "/home/msyn/pywikibot_scripts/../pywikibot/pywikibot/scripts/wrapper.py", line 500, in main
    if not execute():
  File "/home/msyn/pywikibot_scripts/../pywikibot/pywikibot/scripts/wrapper.py", line 487, in execute
    run_python_file(filename, script_args, module)
  File "/home/msyn/pywikibot_scripts/../pywikibot/pywikibot/scripts/wrapper.py", line 147, in run_python_file
    exec(compile(source, filename, 'exec', dont_inherit=True),
  File "test.py", line 15, in <module>
    main()
  File "test.py", line 9, in main
    ps = list(cat.articles(namespaces=4, sortby='timestamp', reverse=True))
  File "/home/msyn/pywikibot/pywikibot/page/_category.py", line 183, in members
    for member in self.site.categorymembers(self, total=total, **kwargs):
  File "/usr/lib/python3.9/_collections_abc.py", line 330, in __next__
    return self.send(None)
  File "/home/msyn/pywikibot/pywikibot/tools/collections.py", line 275, in send
    return next(self._started_gen)
  File "/home/msyn/pywikibot/pywikibot/data/api/_generators.py", line 610, in generator
    self.data = self.request.submit()
  File "/home/msyn/pywikibot/pywikibot/data/api/_requests.py", line 1088, in submit
    raise pywikibot.exceptions.APIError(**error)
pywikibot.exceptions.APIError: badcontinue: Invalid continue param. You should pass the original value returned by the previous query.
[servedby: mw1421;
 help: See https://www.wikidata.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/&gt; for notice of API deprecations and breaking changes.]
CRITICAL: Exiting due to uncaught exception <class 'pywikibot.exceptions.APIError'>

What should have happened instead?:

  • API interaction should not result in an error

Software version (skip for WMF-hosted wikis like Wikipedia):

  • Using pywikibot revision af8d9fe98 from yesterday evening

Other information (browser name/version, screenshots, etc.):

Event Timeline

Xqt triaged this task as High priority.EditedNov 28 2022, 3:28 PM
Xqt subscribed.

This exception is only raised it sortby parameter is 'timestamp'.

I was able to reproduce it via request url as follows:

https://www.wikidata.org/w/api.php?gcmtitle=Category:Wikidata%20status%20updates&gcmprop=ids|title|sortkey&gcmsort=timestamp&gcmdir=desc&prop=info|imageinfo|categoryinfo&inprop=protection&iiprop=timestamp|user|comment|url|size|sha1|metadata&iilimit=max&generator=categorymembers&action=query&indexpageids=&continue=&gcmnamespace=4&gcmlimit=5&meta=userinfo&uiprop=blockinfo|hasmsg&maxlag=5&format=jsonfm

gives

{
    "batchcomplete": "",
    "continue": {
        "gcmcontinue": "2022-10-24 15:31:32|109463830",
        "continue": "gcmcontinue||userinfo"
    },
    "query": {
        "pageids": [
            "110050339",
            "109918126",
            "109829041",
            "109758836",
            "109622558"
        ],
        "pages": {
            "110050339": {
                "pageid": 110050339,
                "ns": 4,
                "title": "Wikidata:Status updates/2022 11 28",
                "contentmodel": "wikitext",
                "pagelanguage": "en",
                "pagelanguagehtmlcode": "en",
                "pagelanguagedir": "ltr",
                "touched": "2022-11-28T15:24:13Z",
                "lastrevid": 1780434722,
                "length": 14063,
                "protection": [],
                "restrictiontypes": [
                    "edit",
                    "move"
                ]
            },
            "109918126": {
                "pageid": 109918126,
                "ns": 4,
                "title": "Wikidata:Status updates/2022 11 21",
                "contentmodel": "wikitext",
                "pagelanguage": "en",
                "pagelanguagehtmlcode": "en",
                "pagelanguagedir": "ltr",
                "touched": "2022-11-22T14:37:40Z",
                "lastrevid": 1777147454,
                "length": 7331,
                "protection": [],
                "restrictiontypes": [
                    "edit",
                    "move"
                ]
            },
            "109829041": {
                "pageid": 109829041,
                "ns": 4,
                "title": "Wikidata:Status updates/2022 11 14",
                "contentmodel": "wikitext",
                "pagelanguage": "en",
                "pagelanguagehtmlcode": "en",
                "pagelanguagedir": "ltr",
                "touched": "2022-11-17T11:11:27Z",
                "lastrevid": 1772883130,
                "length": 11916,
                "protection": [],
                "restrictiontypes": [
                    "edit",
                    "move"
                ]
            },
            "109758836": {
                "pageid": 109758836,
                "ns": 4,
                "title": "Wikidata:Status updates/2022 11 07",
                "contentmodel": "wikitext",
                "pagelanguage": "en",
                "pagelanguagehtmlcode": "en",
                "pagelanguagedir": "ltr",
                "touched": "2022-11-07T16:48:41Z",
                "lastrevid": 1766770860,
                "length": 9534,
                "new": "",
                "protection": [],
                "restrictiontypes": [
                    "edit",
                    "move"
                ]
            },
            "109622558": {
                "pageid": 109622558,
                "ns": 4,
                "title": "Wikidata:Status updates/2022 10 31",
                "contentmodel": "wikitext",
                "pagelanguage": "en",
                "pagelanguagehtmlcode": "en",
                "pagelanguagedir": "ltr",
                "touched": "2022-11-07T20:41:49Z",
                "lastrevid": 1761854336,
                "length": 10707,
                "protection": [],
                "restrictiontypes": [
                    "edit",
                    "move"
                ]
            }
        },
        "userinfo": {
            "id": 1726,
            "name": "Xqt"
        }
    },
    "limits": {
        "imageinfo": 500
    }
}

But the follow-up request

https://www.wikidata.org/w/api.php?gcmtitle=Category:Wikidata%20status%20updates&gcmprop=ids|title|sortkey&gcmsort=timestamp&gcmdir=desc&prop=info|imageinfo|categoryinfo&inprop=protection&iiprop=timestamp|user|comment|url|size|sha1|metadata&iilimit=max&generator=categorymembers&action=query&indexpageids=&continue=gcmcontinue||userinfo&gcmnamespace=4&gcmlimit=500&meta=userinfo&uiprop=blockinfo|hasmsg&maxlag=5&format=jsonfm&gcmcontinue=2022-10-24%2015:31:32|109463830

fails with

{
    "error": {
        "code": "badcontinue",
        "info": "Invalid continue param. You should pass the original value returned by the previous query.",
        "*": "See https://www.wikidata.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/&gt; for notice of API deprecations and breaking changes."
    },
    "servedby": "mw1412"
}

Don't see that this is a Pywikibot issue

Xqt raised the priority of this task from High to Needs Triage.Nov 28 2022, 5:31 PM
Xqt renamed this task from pywikibot badcontinue api response to badcontinue api response if timestamp is copied from continue.Nov 28 2022, 5:53 PM
Xqt added a subscriber: Umherirrender.
Xqt moved this task from Backlog to Reported Upstream on the Upstream board.

Change 868715 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] api: Convert cl_timestamp format for continue on categorymembers

https://gerrit.wikimedia.org/r/868715

Change 868715 merged by jenkins-bot:

[mediawiki/core@master] api: Convert cl_timestamp format for continue on categorymembers

https://gerrit.wikimedia.org/r/868715

Should be fixed on wikidata on January 3, see Roadmap - https://www.mediawiki.org/wiki/MediaWiki_1.40/Roadmap
Let's wait until there before closing

This is solved upstream. I keep it open that Pywikibot can fix it for older MW versions by using isoformat. Are there any hints when this problem was introduced?

The error was in https://gerrit.wikimedia.org/r/c/mediawiki/core/+/851144/ (wmf/1.40.0-wmf.10), fix is in wmf/1.40.0-wmf.17 - no older version to take care of