Page MenuHomePhabricator

flow.Board.topics continuation breaks when sort_by='updated'
Open, HighPublic

Description

When using Board.topics(sort_by='updated'), after the first batch has been iterated (default is 100 topics), the generator fails with TypeError.

from pywikibot.flow import Board
site = pywikibot.Site('mediawiki', 'mediawiki')
board = Board(site, 'Project talk:Sandbox/Flow test')
topics_list = board.topics(sort_by='updated')
for topic in topics_list:
    print(topic)

returns

Found 1 mediawiki:mediawiki processes running, including this one.
[[mediawiki:Topic:T66k4pyiyz5yxuyo]]
[[mediawiki:Topic:T56k61wm97m97zly]]
[[mediawiki:Topic:T3861hfbwvg994dt]]
[[mediawiki:Topic:T47jqwe15hsw3ai0]]
[[mediawiki:Topic:Sy8491obqj9ak5as]]
[[mediawiki:Topic:T1eljfcxcfozigfo]]
[[mediawiki:Topic:T3s5g4btggew2tw3]]
[[mediawiki:Topic:T3m3vd24vrsb8j1i]]
....
Traceback (most recent call last):
  File "<console>", line 1, in <module>
    for topic in topics_list:
  File "pywikibot/flow.py", line 140, in topics
    data = self.site.load_topiclist(self, **cont_args)
  File "pywikibot/site.py", line 1357, in callee
    return fn(self, *args, **kwargs)
  File "pywikibot/site.py", line 6455, in load_topiclist
    offset = pywikibot.Timestamp.fromtimestampformat(offset)
  File "pywikibot/__init__.py", line 178, in fromtimestampformat
    return cls.strptime(ts, cls.mediawikiTSFormat)
TypeError: must be string, not list

Event Timeline

Restricted Application added subscribers: pywikibot-bugs-list, Zppix, Aklapper. · View Herald TranscriptJun 20 2016, 12:47 PM
darthbhyrava renamed this task from flow.topics doesn't support parameter sort_by='updated' to flow.topics doesn't support parameter sort_by='updated' with limit values set..Jun 20 2016, 12:54 PM
jayvdb updated the task description. (Show Details)Jun 21 2016, 11:05 AM

In your first example, limit=2 should only return two topics. I've confirmed that many topics are returned, seemingly unending, except of course that it eventually fails with

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "pywikibot/flow.py", line 137, in topics
    topic = Topic.from_topiclist_data(self, root, data)
  File "pywikibot/flow.py", line 210, in from_topiclist_data
    topic._root = Post.fromJSON(topic, root_uuid, topiclist_data)
  File "pywikibot/flow.py", line 355, in fromJSON
    post = cls(page, post_uuid)
  File "pywikibot/flow.py", line 330, in __init__
    raise NoPage(page, 'Topic must exist: %s')
NoPage: Topic must exist: [[test:Topic:Sel4ipu6maqlbm0d]]

Werent you going to raise a bug about that?

jayvdb triaged this task as High priority.Jun 21 2016, 11:28 AM

Ok, there is definitely a bug in flow.py 's Board.topics, that it effectively ignores the limit argument, because it grabs the pagination data and loops again.

jayvdb renamed this task from flow.topics doesn't support parameter sort_by='updated' with limit values set. to flow.topics continuation breaks when sort_by='updated'.Jun 21 2016, 12:32 PM

In your first example, limit=2 should only return two topics. I've confirmed that many topics are returned, seemingly unending, except of course that it eventually fails with

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "pywikibot/flow.py", line 137, in topics
    topic = Topic.from_topiclist_data(self, root, data)
  File "pywikibot/flow.py", line 210, in from_topiclist_data
    topic._root = Post.fromJSON(topic, root_uuid, topiclist_data)
  File "pywikibot/flow.py", line 355, in fromJSON
    post = cls(page, post_uuid)
  File "pywikibot/flow.py", line 330, in __init__
    raise NoPage(page, 'Topic must exist: %s')
NoPage: Topic must exist: [[test:Topic:Sel4ipu6maqlbm0d]]

Werent you going to raise a bug about that?

As today is the only day this week that I have to look at these issues, I've had to investigate and raise it myself: T138306: Broken topics on mediawiki.org and test wiki that the API doesnt believe exists

jayvdb renamed this task from flow.topics continuation breaks when sort_by='updated' to flow.Board.topics continuation breaks when sort_by='updated'.Jun 21 2016, 2:26 PM
jayvdb updated the task description. (Show Details)Jun 21 2016, 3:45 PM
jayvdb updated the task description. (Show Details)Jun 21 2016, 4:40 PM

In your first example, limit=2 should only return two topics. I've confirmed that many topics are returned, seemingly unending, except of course that it eventually fails with

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "pywikibot/flow.py", line 137, in topics
    topic = Topic.from_topiclist_data(self, root, data)
  File "pywikibot/flow.py", line 210, in from_topiclist_data
    topic._root = Post.fromJSON(topic, root_uuid, topiclist_data)
  File "pywikibot/flow.py", line 355, in fromJSON
    post = cls(page, post_uuid)
  File "pywikibot/flow.py", line 330, in __init__
    raise NoPage(page, 'Topic must exist: %s')
NoPage: Topic must exist: [[test:Topic:Sel4ipu6maqlbm0d]]

Werent you going to raise a bug about that?

I was, along with the bug for Topic:Sel4ipu6maqlbm0d] (which you've already raised in T138306). I was of the opinion that the limit parameter failed only with sort_by='updated', though. Clearly, that wasn't just the case.

Something came up, and I couldn't raise either bug, thank you for taking it up instead.

@jayvdb - I think this needs to be addressed before we go ahead with T135411. Has there been any progress on this front?

Blame parse_qs for this one. It returns a list, of 1 value in this case, for each query string argument.