Page MenuHomePhabricator
Create Task
Maniphest T106765

migrate textlib parser functionality to mwparserfromhell
Closed, DeclinedPublic
Actions

Description

textlib contains a lot of parsing functionality, built using regex. This usually works well, however it becomes complicated / error prone especially with nested templates.

However, as mwparserfromhell tokenizer doesn't expose start & end of each token, the textlib algorithms will need to be heavily revised, or mwparserfromhell needs improvements.

Related Objects
Search...

Event Timeline

jayvdb created this task.Jul 23 2015, 11:57 PM
jayvdb raised the priority of this task from to Medium.
jayvdb updated the task description. (Show Details)
jayvdb added projects: Pywikibot, Pywikibot-textlib.
jayvdb added a subtask: T106763: Mandatory dependency on mwparserfromhell.
jayvdb added subscribers: Xqt, Earwig, Ricordisamoa and 3 others.
Ricordisamoa awarded a token.Jul 24 2015, 12:03 AM
Ricordisamoa updated the task description. (Show Details)
Ricordisamoa set Security to None.

Also reflinks.py

Dalba added a parent task: T110529: template.py does not recognize nested templates.Apr 11 2016, 11:51 PM
Dalba subscribed.
Dvorapa added a subtask: T71384: extract_templates_and_params parser bugs loading w:en:Main_Page with mwparserfromhell.May 14 2019, 2:44 PM
Xqt added a parent task: T229723: Insufficient wikitext regex parser functions in textlib (tracking).Aug 3 2019, 9:57 AM
Xqt closed subtask T71384: extract_templates_and_params parser bugs loading w:en:Main_Page with mwparserfromhell as Resolved.Mar 30 2021, 8:39 AM
Xqt added a subtask: T279005: Use textlib.extract_templates_and_param or Page.templatesWithParams in commons_information.py.Apr 1 2021, 6:25 AM
JJMC89 closed subtask T106763: Mandatory dependency on mwparserfromhell as Resolved.Apr 13 2021, 4:05 PM
Izno removed a parent task: T229723: Insufficient wikitext regex parser functions in textlib (tracking).Oct 30 2021, 6:48 PM
Xqt closed this task as Declined.Dec 12 2025, 8:38 AM

I’ve declined this task after ten years because:

  • Pywikibot currently uses both mwparserfromhell and wikitextparser packages. There is no preference for one over the other.
  • The task does not describe which concrete textlib functions or behaviours should be migrated. Without a clear specification, the scope is undefined and cannot be meaningfully evaluated.
  • Any migration would therefore require a feature request or upstream change to mwparserfromhell or wikitextparser as well.
  • A migration of non-trivial textlib parts would be a major redesign of stable code. Such a change requires a detailed proposal, justification, and upstream support before it can be considered.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits