I think it should be implemented as an infinite generator. Thus, the developer calling Pywikibot should simply be able to iterate as if it were an infinite list. Something like (exact parameters to topics can be hashed out later; you may also want to let them choose the number loaded per page, as you mentioned, but it should have a sane default):

(Note also there are two orderings, both of which should be supported)

for topic in board.topics( 'newest' ):

Internally, it would call view-topiclist as many times as necessary, using the continuation options (offset/offset-id) to get new pages as needed.

There's the question of what happens if they call board.topics() more than once. I would suggest it just restarts uncached; otherwise, you have to cache every topic you've already returned from the generator.

There is a common use case where a bot author may need e.g. the first 100 topics. If they just loop through the first 100 once, they can use the generator and simply break. If they need to do so more then once (on the same first 100), they could use https://docs.python.org/2/library/itertools.html#itertools.islice . We could provide dedicated support for this use case (looping over same first n topics multiple times), but I don't think it's necessary.

See Page.revisions, APISite.loadrevisions, and APISite._generator for how an infinite generator is implemented for revisions. The same approach should be usable for loading the topics, and loading posts.

(And this is uncached; getting the API 'correct' is more important than performance; caching can be implemented under the covers later.)

The same approach should be usable for loading the topics, and loading posts.

Note, currently view-topiclist is the only thing that's paged. For any other view operation, you get everything (e.g. all posts in a topic for view-topic).

In T101261#1340358, @Mattflaschen wrote:

The same approach should be usable for loading the topics, and loading posts.

Note, currently view-topiclist is the only thing that's paged. For any other view operation, you get everything (e.g. all posts in a topic for view-topic).

Getting everything isnt a problem for APISite._generator , but a few extra cycles will be wasted when it sets up pagination that wont be used.
Will view-topic posts be paginated eventually?

In T101261#1340362, @jayvdb wrote:

Will view-topic posts be paginated eventually?

There are no plans to do so, but it's possible.

Change 217717 had a related patch set uploaded (by Happy5214):
[WIP] Load Flow object content in Pywikibot

https://gerrit.wikimedia.org/r/217717

Change 217717 merged by jenkins-bot:
Load Flow object content in Pywikibot

https://gerrit.wikimedia.org/r/217717