Page MenuHomePhabricator

Optional use of https://github.com/Khan/pygments-server
Open, Needs TriagePublic

Description

The folks at Khan Academy have open sourced a small flask application and a shell script client that can be used to provide a drop-in replacement for shelling out to the pygmentize binary. Their use case was Phabricator, but on casual review it looks like it should work with SyntaxHighlight as well. This might be an interesting optimization to explore for large wikis/wikifarms using the extension (like Wikimedia's production wikis). Pygmentize-oid!

Event Timeline

Reedy added a subscriber: Reedy.

I suspect for the SyntaxHighlight implementation it's probably fairly minimal, mostly just some documentation etc

Some solution like this could help folks having the problem from T250763: Cannot use SyntaxHighlight: "Unable to run external programs, proc_open() is disabled". Assuming of course that they can also run additional daemons.

FWIW, I tried using Khan/pygments-server for the Not Wikilambda test wiki on Toolforge, and it didn’t quite work, because Khan/pygments-server expects a fairly specific set of command line arguments and MediaWiki uses different ones. So I built lucaswerkmeister/pygments-server instead, which supports the full pygments CLI (but still has other limitations, just like the Khan version). This worked reasonably well for me until the port to BoxedCommand – with that change, SyntaxHighlight provides the input to pygments as a file, rather than as stdin, which isn’t supported by either pygments-server implementation.

@Legoktm pointed out on IRC that it probably wouldn’t be too hard to cut out the pygmentize script entirely, and instead write a version of the Pygmentize PHP class that directly talks to some kind of pygments-server via HTTP. It would probably need three endpoints: /css, /lexers and /highlight, or something like that. (For my Toolforge use case, this would be attractive because the server could run in a container with only Python installed – with the Shellbox approach, I need a container image with both Python and PHP, and no such image is available on Toolforge.)

Update: my pygments-server now supports input files, so it works with latest SyntaxHighlight again.