A lot of search entries are missing in core documentation
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Xqt
	Apr 3 2023, 11:19 AM

Description

Steps to replicate the issue:

Search for backports in the documentation
https://doc.wikimedia.org/pywikibot/master/search.html?q=backports&check_keywords=yes&area=default#

What happens?:
The result is empty but the module is available through the index https://doc.wikimedia.org/pywikibot/master/api_ref/x_backports.html#module-backports

What should have happened instead?:
It works with stable release:
https://doc.wikimedia.org/pywikibot/stable/api_ref/x_backports.html#module-backports

Software version:
Pywikibot 8.1.0dev3

Maybe this is due to Sphinx 6.1 (stable uses 5.3) but it works locally

Details

Subject	Repo	Branch	Lines +/-
[doc] Use Sphinx >= 6.2.1	pywikibot/core	master	+9 -9
[doc] Workaround for search index generation	pywikibot/core	master	+1 -1
Revert "[doc] Run Sphinx 6 for creating documentation"	pywikibot/core	master	+2 -2

Customize query in gerrit

Event Timeline

Xqt created this task.Apr 3 2023, 11:19 AM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptApr 3 2023, 11:19 AM

Xqt triaged this task as High priority.Apr 3 2023, 11:19 AM

Change 904945 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] Revert "[doc] Run Sphinx 6 for creating documentation"

https://gerrit.wikimedia.org/r/904945

gerritbot added a project: Patch-For-Review.Apr 3 2023, 3:54 PM

Change 904945 merged by Xqt:

[pywikibot/core@master] Revert "[doc] Run Sphinx 6 for creating documentation"

https://gerrit.wikimedia.org/r/904945

Maintenance_bot removed a project: Patch-For-Review.Apr 3 2023, 4:11 PM

Looking into this as time permits. Testing locally, I can see that search works correctly with Sphinx 6.0.1 and breaks with 6.1.0. I'll look into this further and claim this task if I come up with a solution.

This is fixed in Sphinx 6.2.0 (yet to be released), specifically this PR. Essentially, 6.1.0 introduced parallelization of multiple tasks, including search index building. That turned out to be buggy and is reverted by the pull request I linked.

Sphinx (>6.1 and <6.2) only attempts to parallelize search index building when we use the -j parameter - which tells Sphinx to build using multiple processes. We started using the -j parameter when we switched to Furo because it shortened build time significantly. As a workaround, we could disable the -j parameter in CI. This would result in a longer build time (and potential timeouts), but the search index would be created correctly even in 6.1.3.

My preference would be to wait for 6.2.0 to be released (or 6.1.x if it includes the above fix) and only switch to Python 3.9/Sphinx>6.1 then.

Change 905572 had a related patch set uploaded (by KBach; author: KBach):

[pywikibot/core@master] [doc] Workaround for search index generation

https://gerrit.wikimedia.org/r/905572

gerritbot added a project: Patch-For-Review.Apr 4 2023, 9:15 AM

Change 905572 abandoned by KBach:

[pywikibot/core@master] [doc] Workaround for search index generation

Reason:

Doesn't build or fix the problem

https://gerrit.wikimedia.org/r/905572

In T333816#8753499, @KBach wrote:

My preference would be to wait for 6.2.0 to be released (or 6.1.x if it includes the above fix) and only switch to Python 3.9/Sphinx>6.1 then.

Ok. Thanks for investigation.

Change 917909 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [doc] Use Sphinx >= 6.2.1

https://gerrit.wikimedia.org/r/917909

gerritbot added a project: Patch-For-Review.May 9 2023, 3:29 PM

Change 917909 merged by jenkins-bot:

[pywikibot/core@master] [doc] Use Sphinx >= 6.2.1

https://gerrit.wikimedia.org/r/917909

Xqt closed this task as Resolved.May 9 2023, 4:54 PM

Xqt claimed this task.

KBach awarded a token.May 10 2023, 7:04 AM