Huge memory usage when using archivebot with -async option
Open, MediumPublicBUG REPORT
Actions

Assigned To

None

Authored By

	Xqt
	Apr 1 2023, 8:43 AM

Description

Steps to replicate the issue:

pwb -simulate archivebot -lang:en -async

What happens?:
The memory usage increases a lot. Probably a MemoryError could occurr when running for the whole wiki.

The reason is the internal queue of ThreadPoolExecutor which is an unlimited SimpleQueue (or unlimited Queue for < Python 3.7). This queue holds all pending tasks references to the archive template retrieved from generator.

What should have happened instead?:
The memory usage should not be exhausing.

Software version:
Pywikibot 8.1.0dev3

Event Timeline

Xqt created this task.Apr 1 2023, 8:43 AM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptApr 1 2023, 8:43 AM

Xqt triaged this task as Medium priority.Apr 1 2023, 8:44 AM

Xqt renamed this task from Huge memory usage when using archivebot with -async optin to Huge memory usage when using archivebot with -async option.Apr 2 2023, 7:57 AM

Huge memory usage when using archivebot with -async optionOpen, MediumPublicBUG REPORTActions

Description

Event Timeline

Huge memory usage when using archivebot with -async option
Open, MediumPublicBUG REPORT
Actions