flow.Board.topics continuation breaks when sort_by='updated'
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	darthbhyrava
	Jun 20 2016, 12:47 PM

Description

When using Board.topics(sort_by='updated'), after the first batch has been iterated (default is 100 topics), the generator fails with TypeError.

from pywikibot.flow import Board
site = pywikibot.Site('mediawiki', 'mediawiki')
board = Board(site, 'Project talk:Sandbox/Flow test')
topics_list = board.topics(sort_by='updated')
for topic in topics_list:
    print(topic)

returns

Found 1 mediawiki:mediawiki processes running, including this one.
[[mediawiki:Topic:T66k4pyiyz5yxuyo]]
[[mediawiki:Topic:T56k61wm97m97zly]]
[[mediawiki:Topic:T3861hfbwvg994dt]]
[[mediawiki:Topic:T47jqwe15hsw3ai0]]
[[mediawiki:Topic:Sy8491obqj9ak5as]]
[[mediawiki:Topic:T1eljfcxcfozigfo]]
[[mediawiki:Topic:T3s5g4btggew2tw3]]
[[mediawiki:Topic:T3m3vd24vrsb8j1i]]
....
Traceback (most recent call last):
  File "<console>", line 1, in <module>
    for topic in topics_list:
  File "pywikibot/flow.py", line 140, in topics
    data = self.site.load_topiclist(self, **cont_args)
  File "pywikibot/site.py", line 1357, in callee
    return fn(self, *args, **kwargs)
  File "pywikibot/site.py", line 6455, in load_topiclist
    offset = pywikibot.Timestamp.fromtimestampformat(offset)
  File "pywikibot/__init__.py", line 178, in fromtimestampformat
    return cls.strptime(ts, cls.mediawikiTSFormat)
TypeError: must be string, not list

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	[bugfix] Respect limit argument with Board.topics()	pywikibot/core	master	+59 -39
	[bugfix] Respect limit argument with Board.topics()	pywikibot/core	master	+57 -39

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Xqt	T138215 flow.Board.topics continuation breaks when sort_by='updated'
		Resolved		Xqt	T138307 Board.topics does not respect limit argument

Event Timeline

darthbhyrava created this task.Jun 20 2016, 12:47 PM

Restricted Application added subscribers: pywikibot-bugs-list, Zppix, Aklapper. · View Herald TranscriptJun 20 2016, 12:47 PM

darthbhyrava mentioned this in T135411: Implement pywikibot support for adding thanks to comments on flow boards.Jun 20 2016, 12:49 PM

darthbhyrava renamed this task from flow.topics doesn't support parameter sort_by='updated' to flow.topics doesn't support parameter sort_by='updated' with limit values set..Jun 20 2016, 12:54 PM

jayvdb updated the task description. (Show Details)Jun 21 2016, 11:05 AM

In your first example, limit=2 should only return two topics. I've confirmed that many topics are returned, seemingly unending, except of course that it eventually fails with

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "pywikibot/flow.py", line 137, in topics
    topic = Topic.from_topiclist_data(self, root, data)
  File "pywikibot/flow.py", line 210, in from_topiclist_data
    topic._root = Post.fromJSON(topic, root_uuid, topiclist_data)
  File "pywikibot/flow.py", line 355, in fromJSON
    post = cls(page, post_uuid)
  File "pywikibot/flow.py", line 330, in __init__
    raise NoPage(page, 'Topic must exist: %s')
NoPage: Topic must exist: [[test:Topic:Sel4ipu6maqlbm0d]]

Werent you going to raise a bug about that?

Ok, there is definitely a bug in flow.py 's Board.topics, that it effectively ignores the limit argument, because it grabs the pagination data and loops again.

jayvdb renamed this task from flow.topics doesn't support parameter sort_by='updated' with limit values set. to flow.topics continuation breaks when sort_by='updated'.Jun 21 2016, 12:32 PM

In T138215#2396051, @jayvdb wrote:
In your first example, limit=2 should only return two topics. I've confirmed that many topics are returned, seemingly unending, except of course that it eventually fails with
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "pywikibot/flow.py", line 137, in topics
    topic = Topic.from_topiclist_data(self, root, data)
  File "pywikibot/flow.py", line 210, in from_topiclist_data
    topic._root = Post.fromJSON(topic, root_uuid, topiclist_data)
  File "pywikibot/flow.py", line 355, in fromJSON
    post = cls(page, post_uuid)
  File "pywikibot/flow.py", line 330, in __init__
    raise NoPage(page, 'Topic must exist: %s')
NoPage: Topic must exist: [[test:Topic:Sel4ipu6maqlbm0d]]
Werent you going to raise a bug about that?

As today is the only day this week that I have to look at these issues, I've had to investigate and raise it myself: T138306: Broken topics on mediawiki.org and test wiki that the API doesnt believe exists

jayvdb renamed this task from flow.topics continuation breaks when sort_by='updated' to flow.Board.topics continuation breaks when sort_by='updated'.Jun 21 2016, 2:26 PM

jayvdb updated the task description. (Show Details)Jun 21 2016, 3:45 PM

jayvdb updated the task description. (Show Details)Jun 21 2016, 4:40 PM

In T138215#2396051, @jayvdb wrote:
In your first example, limit=2 should only return two topics. I've confirmed that many topics are returned, seemingly unending, except of course that it eventually fails with
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "pywikibot/flow.py", line 137, in topics
    topic = Topic.from_topiclist_data(self, root, data)
  File "pywikibot/flow.py", line 210, in from_topiclist_data
    topic._root = Post.fromJSON(topic, root_uuid, topiclist_data)
  File "pywikibot/flow.py", line 355, in fromJSON
    post = cls(page, post_uuid)
  File "pywikibot/flow.py", line 330, in __init__
    raise NoPage(page, 'Topic must exist: %s')
NoPage: Topic must exist: [[test:Topic:Sel4ipu6maqlbm0d]]
Werent you going to raise a bug about that?

I was, along with the bug for Topic:Sel4ipu6maqlbm0d] (which you've already raised in T138306). I was of the opinion that the limit parameter failed only with sort_by='updated', though. Clearly, that wasn't just the case.

Something came up, and I couldn't raise either bug, thank you for taking it up instead.

darthbhyrava mentioned this in T138531: GSoC: Pywikibot-Thanks | Meeting 9 | 24th June.Jun 24 2016, 3:32 PM

@jayvdb - I think this needs to be addressed before we go ahead with T135411. Has there been any progress on this front?