The folks at Khan Academy have open sourced a small flask application and a shell script client that can be used to provide a drop-in replacement for shelling out to the pygmentize binary. Their use case was Phabricator, but on casual review it looks like it should work with SyntaxHighlight as well. This might be an interesting optimization to explore for large wikis/wikifarms using the extension (like Wikimedia's production wikis). Pygmentize-oid!
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T122510 Many calls to syntaxhighlight is very slow, especially noticeable when previewing | |||
| Open | None | T271751 SyntaxHighlight extension is slow (needs more active stewardship) | |||
| Open | None | T257988 Optional use of https://github.com/Khan/pygments-server |
Event Timeline
Comment Actions
I suspect for the SyntaxHighlight implementation it's probably fairly minimal, mostly just some documentation etc
Comment Actions
Some solution like this could help folks having the problem from T250763: Cannot use SyntaxHighlight: "Unable to run external programs, proc_open() is disabled". Assuming of course that they can also run additional daemons.
Comment Actions
FWIW, I tried using Khan/pygments-server for the Not Wikilambda test wiki on Toolforge, and it didn’t quite work, because Khan/pygments-server expects a fairly specific set of command line arguments and MediaWiki uses different ones. So I built lucaswerkmeister/pygments-server instead, which supports the full pygments CLI (but still has other limitations, just like the Khan version). This worked reasonably well for me until the port to BoxedCommand – with that change, SyntaxHighlight provides the input to pygments as a file, rather than as stdin, which isn’t supported by either pygments-server implementation.
@Legoktm pointed out on IRC that it probably wouldn’t be too hard to cut out the pygmentize script entirely, and instead write a version of the Pygmentize PHP class that directly talks to some kind of pygments-server via HTTP. It would probably need three endpoints: /css, /lexers and /highlight, or something like that. (For my Toolforge use case, this would be attractive because the server could run in a container with only Python installed – with the Shellbox approach, I need a container image with both Python and PHP, and no such image is available on Toolforge.)
Comment Actions
Update: my pygments-server now supports input files, so it works with latest SyntaxHighlight again.
Comment Actions
Hi, any updates? This sounds like the most promising way to solve the performance issue.
Background: this blocks Arch Wiki adoption of the syntax highlighting module https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/667