Huge memory usage when using archivebot with -async option
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Xqt
	Apr 1 2023, 8:43 AM

Description

Steps to replicate the issue:

pwb -simulate archivebot -lang:en -async

What happens?:
The memory usage increases a lot. Probably a MemoryError could occurr when running for the whole wiki.

The reason is the internal queue of ThreadPoolExecutor which is an unlimited SimpleQueue (or unlimited Queue for < Python 3.7). This queue holds all pending tasks references to the archive template retrieved from generator.

What should have happened instead?:
The memory usage should not be exhausing.

Software version:
Pywikibot 8.1.0dev3

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	IMPR: Add BoundedPoolExecutor	pywikibot/core	master	+232 -15

Customize query in gerrit

Event Timeline

Xqt created this task.Apr 1 2023, 8:43 AM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptApr 1 2023, 8:43 AM

Xqt triaged this task as Medium priority.Apr 1 2023, 8:44 AM

Xqt renamed this task from Huge memory usage when using archivebot with -async optin to Huge memory usage when using archivebot with -async option.Apr 2 2023, 7:57 AM

Change #1105000 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] IMPR: Add BoundedThreadPoolExecutor

https://gerrit.wikimedia.org/r/1105000

gerritbot added a project: Patch-For-Review.Dec 17 2024, 1:53 PM

Xqt claimed this task.Dec 23 2024, 4:05 PM

Change #1105000 merged by jenkins-bot:

[pywikibot/core@master] IMPR: Add BoundedPoolExecutor

https://gerrit.wikimedia.org/r/1105000

Maintenance_bot removed a project: Patch-For-Review.Dec 29 2024, 10:30 PM

Xqt closed this task as Resolved.Dec 30 2024, 9:19 AM