This is a very large page? Is there something unexpected here?

@aaron - It's a very large page - but I think it's a lack of performance if it cannot be opened because of it. So I expect an issue for the performance team. Why did you remove the project?

because the Performance Team does not plan to work on this

@Aklapper - So how can I get the content of the page now?

@Aklapper - We have maximal page sizes that such an error should not happen. The page size is 1.079.263 Bytes. But I want to get the page content anyhow. Please subscribe somebody to this ticket who can help with this problem.

In T316858#8216559, @doctaxon wrote:

@Aklapper - So how can I get the content of the page now?

You go to https://de.wikipedia.org/w/index.php?title=Benutzer:Doc_Taxon/Test4&action=edit

In T316858#8217708, @doctaxon wrote:

Please subscribe somebody to this ticket who can help with this problem.

That's not how it works, sorry. See also https://www.mediawiki.org/wiki/Bug_management/Development_prioritization - thanks.

You can use https://de.wikipedia.org/wiki/Benutzer:Doc_Taxon/Test4?action=edit or https://de.wikipedia.org/wiki/Benutzer:Doc_Taxon/Test4?action=raw to get access to the wikitext. The revisions query API is another option to do the same.

In it's current state the page contains 5,020 <syntaxhighlight> tags in a little more than 1 megabyte of wikitext. Syntax highlighting is generally not a cheap operation. In this case it's done server-side. A timeout is not really a surprise. I agree that a more helpful failure state would be nice. But as of now the behavior we see is the expected one.

https://www.mediawiki.org/wiki/Extension:SyntaxHighlight might contain more information.

You have more than 5000 <syntaxhighlight> snippets in one page. What's the point? That's just too many to highlight in 60 seconds.

Thank you for your help!

In my opinion it's not a very good behavior of the Phabricator when replies come only after posting the problems onto wikitech-l mailing list. It looks like there is nobody who pushes the tickets to users who can work on issues or bugs. I don't know all the projects, components and users behind the WMF technics to forward bugs and issues to them properly. The same here with this sleeping bug for more than one month: https://phabricator.wikimedia.org/T313688

KInd regards

@doctaxon: Again, please see https://www.mediawiki.org/wiki/Bug_management/Development_prioritization

In T316858#8217871, @doctaxon wrote:

In my opinion it's not a very good behavior of the Phabricator when replies come only after posting the problems onto wikitech-l mailing list. It looks like there is nobody who pushes the tickets to users who can work on issues or bugs.

In this case had your bug been filed against SyntaxHighlight I probably would've seen it much earlier.

@Legoktm - yes I know and I'm knowing "Bug_management/Development_prioritization", but I'm talking about bug or ticket management. There should be someone who knows the structures here and puts the tasks on the road to the proper direction. I am not a software or WMF professional who knows that this problem is to be working on by the SyntaxHighlight project. I didn't know the SyntaxHighlight project until now.

But thank you for redirecting this error now to the proper project team.

There are hundreds of people here that do this all day. There was just not enough information in your report, and still isn't. What do you want us to do?

Out of curiosity I fired up xdebug and profiled the parsing process for a small subset of your example page. Turns out about 99.9% of the time is not spend in MediaWiki. Not even in PHP. It's spend in pygments, a Python library. MediaWiki shells out to that, i.e. calls it as a separate executable. The performance of pygments is discussed and taken care of, it seems. Problem is: When we do this 5,000 times and expect it to be done in 60 seconds or less we have about 10 milliseconds for each. I don't think this can ever work, no matter how fast pygments is.

Combining the HTML snippets into a single <syntaxhighlight> might help.

Change 830683 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/extensions/SyntaxHighlight_GeSHi@master] Track syntaxhighlighting as expensive parser tag hook

https://gerrit.wikimedia.org/r/830683

Change 830683 merged by jenkins-bot:

[mediawiki/extensions/SyntaxHighlight_GeSHi@master] Track syntaxhighlighting as expensive parser tag hook

https://gerrit.wikimedia.org/r/830683

+User-notice

Proposed: [[mw:Special:MyLanguage/Extension:SyntaxHighlight|Syntax highlighting]] is now tracked as an [[mw:Special:MyLanguage/Manual:$wgExpensiveParserFunctionLimit|expensive parser function]], only 500 expensive function calls can be used on a single page. Pages that exceed the limit are added to a [[Category:{{int:expensive-parserfunction-category}}|tracking category]].

Shouldn’t inline syntax highlighting be exempted from this (or have higher limit)? I’m not sure whether there are many pages that use inline <syntaxhighlight> that frequently, but I also don’t think that there’s as much threat with the inline tags as much as with, tbh, atrocious page that was linked to above.

In T316858#8221346, @stjn wrote:

Shouldn’t inline syntax highlighting be exempted from this (or have higher limit)? I’m not sure whether there are many pages that use inline <syntaxhighlight> that frequently, but I also don’t think that there’s as much threat with the inline tags as much as with, tbh, atrocious page that was linked to above.

The cost for each syntax highlight is mostly the same, regardless of whether it's full or inline (HTTP request to Shellbox, plus invoking the Python interpreter). Ideally we would batch requests to cut down on the overall overhead, see T271751#7386683.

Limit is too low for wikibooks projects: this limit is reached for every programming books because they include a printable version where are chapters are transcluded. This unique page enables PDF generation as collections PDF generation is disabled. Now we cannot generate PDF for books.

Evolution should bring solutions, not problems.

Now we cannot generate PDF for books.

I''m curious. What happens? Do you have an example?

For example:
https://fr.wikibooks.org/wiki/Programmation_Java/Version_imprimable

PDF file is generated from the page, either by using PDF printer, or with
https://fr.wikibooks.org/w/index.php?title=Sp%C3%A9cial:DownloadAsPdf&page=Programmation_Java%2FVersion_imprimable&action=show-download-screen

Collection is disabled, for example with book Programmation_en_Go:
https://fr.wikibooks.org/wiki/Sp%C3%A9cial:Livre?colltitle=Wikilivres%3ACompilations%2FProgrammation_en_Go&bookcmd=render_collection

Hm. It looks like the problem is in the Lua code used to render the book on a single page. When I manually do the same what the Lua module is supposed to do and combine the pages manually like follows, it all works just fine. Including the syntax highlighting. I'm not super familiar with every detail of our Lua environment. But I suspect it should be possible to make the Lua module output the same wikitext and parse that in a single step instead of calling expandTemplate in a loop.

{{:Programmation Java/Introduction}}
{{:Programmation Java/Machine virtuelle}}
{{:Programmation Java/Installation}}
{{:Programmation Java/NetBeans}}
{{:Programmation Java/Eclipse}}
{{:Programmation Java/Visual Studio Code}}
{{:Programmation Java/Bases du langage}}
{{:Programmation Java/Premier programme}}
{{:Programmation Java/Commentaires}}
{{:Programmation Java/Types de base}}
{{:Programmation Java/Opérateurs}}
{{:Programmation Java/Conditions}}
{{:Programmation Java/Bloc de portée}}
{{:Programmation Java/Itérations}}
{{:Programmation Java/Tableaux}}
{{:Programmation Java/Les classes en Java}}
{{:Programmation Java/Membres}}
{{:Programmation Java/Modificateur}}
{{:Programmation Java/Héritage}}
{{:Programmation Java/Encapsulation}}
{{:Programmation Java/Polymorphisme}}
{{:Programmation Java/Classes abstraites}}
{{:Programmation Java/Interfaces}}
{{:Programmation Java/Classes internes}}
{{:Programmation Java/Transtypage}}
{{:Programmation Java/Types génériques}}
{{:Programmation Java/Instanciation et cycle de vie}}
{{:Programmation Java/this}}
{{:Programmation Java/Objets comparables et clés}}
{{:Programmation Java/Énumérations}}
{{:Programmation Java/Exceptions}}
{{:Programmation Java/Paquetages}}
{{:Programmation Java/Classes de base}}
{{:Programmation Java/Collections}}
{{:Programmation Java/Nombre de taille arbitraire}}
{{:Programmation Java/Tags Javadoc}}
{{:Programmation Java/Annotations}}
{{:Programmation Java/Entrées Sorties}}
{{:Programmation Java/Processus légers et synchronisation}}
{{:Programmation Java/Méthodes natives}}
{{:Programmation Java/Réflexion}}
{{:Programmation Java/Regex}}
{{:Programmation Java/JUnit}}
{{:Programmation Java/Conclusion}}
{{:Programmation Java/Générer un carré magique}}
{{:Programmation Java/Générer un triangle de Sierpiński}}
{{:Programmation Java/XML}}
{{:Programmation Java/Swing}}
{{:Programmation Java/Swing/Hello}}
{{:Programmation Java/Swing/Boutons sur image}}
{{:Programmation Java/Swing/Somme}}
{{:Programmation Java/Swing/Distance}}
{{:Programmation Java/JDK}}
{{:Programmation Java/Paramétrer la JVM}}
{{:Programmation Java/Liste des mots réservés}}
{{:Programmation Java/Débogage}}
{{:Programmation Java/Autres langages}}

Actually by experience the hard-coded page solution in the print version has the exact withdraw of the redundancy code anti-design pattern: it's never up-to-date.

Worse, it doesn't work as expected, letting the page end with the subpages titles instead of their contents. Ex:

• https://en.wikibooks.org/w/index.php?title=Haskell/Print_version
• https://en.wikibooks.org/w/index.php?title=LaTeX/Print_version

Please ping me if anyone sees how to implement this or bypass the new limitation, because we have at least three huge French Wikibooks and a few other in the English one, using this Lua module or not, which represents years of good work, now sabotaged by the MediaWiki update.

PS: now I just hope that the brain drain due to the JavaScript regressions won't expand to the Lua contributors.

Please have a break and consider reading https://mediawiki.org/wiki/Phabricator/Project_management#Setting_task_priorities as well as https://mediawiki.org/wiki/Code_of_Conduct again. The tone here is not acceptable.

One option I can think of is to raise the $wgExpensiveParserFunctionLimit for Wikibooks, but leave it as is for all other projects. MediaWiki's default is 100, and we already raised it to 500 for all Wikimedia wikis. Maybe it's possible to raise it further for low-traffic wikis when there is a good reason to do so, like here. Just an idea.

In T316858#8247615, @DavidL wrote:

For example:
https://fr.wikibooks.org/wiki/Programmation_Java/Version_imprimable

PDF file is generated from the page, either by using PDF printer, or with
https://fr.wikibooks.org/w/index.php?title=Sp%C3%A9cial:DownloadAsPdf&page=Programmation_Java%2FVersion_imprimable&action=show-download-screen

Collection is disabled, for example with book Programmation_en_Go:
https://fr.wikibooks.org/wiki/Sp%C3%A9cial:Livre?colltitle=Wikilivres%3ACompilations%2FProgrammation_en_Go&bookcmd=render_collection

It seems there are 414 snippet to highlight on that page.

<syntaxhighlight inline> are also counted even if it contains one word only.

Edit Preview says :

Attention : cette page contient de trop nombreux appels à des fonctions coûteuses de l’analyseur syntaxique.

Il devrait y avoir moins de 500 appels, alors qu’il y en a maintenant 505.

In T316858#8255061, @DavidL wrote:

<syntaxhighlight inline> are also counted even if it contains one word only.

A use of syntax highlight for 1 word is relatively as expensive as one for 100 words. It doesn’t matter if an airplane flight flies with 1 person or with 200 persons, it still requires a plane, an airport, a gate, take off and a landing.

This has come up at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Template:Code_and_Too_many_expensive_function_calls as well

Currently we have a book with 1310 <syntaxhighlight> : https://fr.wikibooks.org/wiki/Programmation_PHP/Version_imprimable
(number is visible in error message shown in edit preview)

The limit should be raised to about 2000 for wikibooks projects.

Other solutions :

• Independent server for each wikibooks project to not depend on other wiki project server resource shortage,
• Move all pages to another wiki where PHP function incrementExpensiveFunctionCount has been removed,
• Not resolve this issue and let user experience be very bad with wiki projects, blaming MediaWiki developpers.

Third solution seems to be the chosen one for the moment.

@DavidL, I said it before and say it again: This tone is not acceptable. Please take a step back.

Sorry to ask but if the problem won't be solved here in a few days, I need to know in order to replace all smallest <syntaxhighlight> by <pre> tags by bot.
Because print versions are worth a little less readability.

And we have no project to launch static code formatter tools like phpcbf in the Wikibooks <syntaxhighlight> tags.

Change 840667 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/extensions/SyntaxHighlight_GeSHi@master] Count only real highlighting as expensive parser tag hooks

https://gerrit.wikimedia.org/r/840667

Change 840667 merged by jenkins-bot:

[mediawiki/extensions/SyntaxHighlight_GeSHi@master] Count only real highlighting as expensive parser tag hooks

https://gerrit.wikimedia.org/r/840667

All patches are merged; should this task remain open?

DavidL has taken the time to split several books, but on https://en.wikibooks.org/wiki/Category:Pages_with_too_many_expensive_parser_function_calls we can see that 14 are still broken for more than seven months.
And some readers are complaining: https://en.wikibooks.org/wiki/Talk:F_Sharp_Programming/Print_version