Page MenuHomePhabricator
Create Task
Maniphest T85389

'-titleregex' does not handle namespaces other than 0
Closed, ResolvedPublic
Actions

Description

When -titleregex is called with a namespace other than 0 (with GeneratorFactory), no pages are returned. Only pages with namespace 0 are ever returned.

In GeneratorFactory, -titleregex calls RegexFilterPageGenerator (wich is really RegexFilter.titlefilter) with an argument of site.allpages(). allpages defaults to namespace 0. When getCombinedGenerator() is called, the generator (already filled with namespaces of 0) is not considered a pywikibot.data.api.QueryGenerator and is filtered out with NamespaceFilterPageGenerator for the appropriate namespaces, either resulting empty results or only with namespaces 0.

Related Objects

Event Timeline

Daviskr created this task.Dec 28 2014, 2:36 AM
Daviskr raised the priority of this task from to Medium.
Daviskr updated the task description. (Show Details)
Daviskr added a project: Pywikibot-pagegenerators.py.
Daviskr subscribed.
Daviskr added a project: Pywikibot.Dec 28 2014, 2:43 AM
Daviskr set Security to None.
Daviskr added a comment.EditedDec 28 2014, 4:39 AM

Another side effect of the current setup is that it fetches all pages (that match the regex) before it applies limit*. This causes extreme slowdown as seen in this build.

*: As Mpaa said below, this only applies to namespaces other than zero.

jayvdb awarded a token.Dec 28 2014, 5:12 AM
Ricordisamoa subscribed.Dec 28 2014, 5:25 AM
Mpaa subscribed.Dec 28 2014, 5:46 PM
In T85389#945688, @Daviskr wrote:

Another side effect of the current setup is that it fetches all pages (that match the regex) before it applies limit. This causes extreme slowdown as seen in this build.

One clarification: only if a namespace different from 0 is specified, as it is done in the test, for the reason explained above: no pages will be yielded at all, so the test will end only when all the pages in the specified ns have been fetched by allpages().

Mpaa added a comment.Dec 28 2014, 6:23 PM
This comment was removed by Mpaa.
gerritbot added a project: Patch-For-Review.Dec 29 2014, 10:55 AM

Change 181993 had a related patch set uploaded (by Mpaa):
Pagegenerators.py: ns handling for titleregex option

https://gerrit.wikimedia.org/r/181993

Patch-For-Review

Ricordisamoa merged a task: T57226: -titleregex only searches mainspace.Dec 30 2014, 2:51 AM
Ricordisamoa added subscribers: Ladsgroup, Legoktm.
jayvdb subscribed.Dec 30 2014, 12:03 PM

After T57226 is merged, we can re-purpose this task to track the underlying problem; I suspect we want to wait until argparse has landed before fixing the real problem, as it should be much simpler then.

Daviskr mentioned this in rPWBC976705c28f34: Add Generator Factory Tests.Mar 4 2015, 8:46 AM
Daviskr mentioned this in rPWBC27b81634f2ab: Skip TestFactoryGenerator.test_regexfilter_ns.
jayvdb mentioned this in rPWBCcf366fc3ffa5: Pagegenerators: ns handling for titleregex option.Mar 4 2015, 8:46 AM
Mpaa added a comment.Sep 27 2015, 2:01 PM

https://gerrit.wikimedia.org/r/181993 is merged.

Mpaa closed this task as Resolved.Sep 27 2015, 2:01 PM
Mpaa added a comment.Oct 10 2015, 9:15 AM

See also T114015, where it is suggested to change how titleregex works.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL