There are some problems with textlib.replaceExcept() using 'templates' as exceptions e.g. menioned in T63024 or T105620. CPU is time consuming, I found 12% on 8 core processor. I guess there are problems with nested templates.
You may check it with
pwb.py cosmetic_changes -page:Kalle_Svensson -simulate -lang:cs
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Xqt | T63024 cosmetic changes must not remove useless spaces inside templates | |||
| Resolved | jayvdb | T105620 cosmetic_changes hangs or forms an infinite loop | |||
| Invalid | None | T72936 Important tasks to be solved (tracking) | |||
| Resolved | jayvdb | T105621 textlib.replaceExcept() doesn't work properly with template exceptions |
Change 224301 had a related patch set uploaded (by Xqt):
Deactivate removeUselessSpaces due to several bugs
I was investigationg in this problem: The main problem is the new NESTED_TEMPLATE_REGEX which consumes a lot of time resolving nested template recursivly. e.g. this [1] version needs 6 seconds to process, whereas [2] needs high increasing 149 secondes, both running cosmetic_changes.removeUselessSpaces() only. There was only that [3] small difference between the versions. In result the bug can be solved by going back from umpteenth attempt using regex for nested template to the previous iterative solution by inline markers which is very faster.
[1] https://de.wikipedia.org/w/index.php?title=Benutzer:Xqt/Labor&oldid=143951267
[2] https://de.wikipedia.org/w/index.php?title=Benutzer:Xqt/Labor&oldid=143951307
[3] https://de.wikipedia.org/w/index.php?title=Benutzer%3AXqt%2FLabor&type=revision&diff=143951307&oldid=143951267
I don't understand why adding that template at the bottom is making it that slow. And doing modifications on a string should be more complex than just iterating over the text.
Anyway it needs 150 seconds for the 2nd test run. I bet that page on cs-wiki will also terminate but uses few minutes.
I breaked cc on cs:Kalle_Svensson after 61937 seconds. Seems that regex needs exponentially increasing time.
Yes you are right. When I run the regex it won't even try the text from your test page regex101 and mentions catastrophic backtracking.
Breaks various scripts e.g. while removing templates. Currently it fails for different pages for
pwb.y template.py -remove "Navigationsleiste Oscar Bester fremdsprachiger Film" -summary:"Bot: Entferne gelöschte Navigation Oscar Bester fremdsprachiger Film"
and cc is mainly unusable.
Change 226531 had a related patch set uploaded (by John Vandenberg):
Reduce complexity of NESTED_TEMPLATE_REGEX
This task has been "Unbreak Now!" priority for nearly two weeks. Please set an assignee.
Change 299137 had a related patch set uploaded (by Xqt):
Deactivate removeUselessSpaces due to several bugs
Change 299137 merged by jenkins-bot:
Deactivate removeUselessSpaces due to several bugs
Sample of wrong action:
https://de.wikipedia.org/w/index.php?oldid=155655029&diff=156228716
This problem, related to extremely high CPU usage and pywikibot hanging, is fixed.
I have created T140608 to investigate your problem.