I'm in favour of removing the uploading of graph images as wiki files (e.g. this patch), which would solve this issue too I think.

There's some discussion about that on T215391.

We use the GraphViz extension on the slicer.org wiki which we're upgrading from 1.27 to 1.33. Is there any hope for a fix? I've tried patching the database as described here, but get Error: Image is invalid or non-existent.

One solution here might be to create a small PHP web service, external to MediaWiki, that renders the graphs so they don't have to be added to the wiki. It seems like the idea of rendering the graphs into the images/ directory (the patch I linked above) and serving them directly from there is no good for people using the AWS extension (and other file repositories).

Does this sound like it'd be worth it? Does anyone have any other ideas?

As the committer of the PlantUML extension i see similar issues since PlantUML is based on graphviz. A common approach to image handling might be helpful here.

An external rendering tool would be best for PlantUML as well. Maybe we could have one that handles graphviz, mscgen, and plantuml.jar (each being optional).

I might start working on such a thing.

I don't mind too much storing the graph images on the wiki side...it's helpful for abstracting image auth, hosting and caching. Also it provides users with an easy way to track graph changes over time visually. We've had some success doing this and storing the md5 hash of the image source in the file metadata (in the image table). You can do the actual rendering locally or remotely depending on what you need.

In terms of getting Graphviz working on 1.33 as-is, we had good results by editing includes/UploadLocalFile.php function upload2 to use the core code recordUpload2 instead of recordUpload3

$pageText = '';
if ( !$this->recordUpload2( $oldver, $comment, $pageText, $props, false, $user ) ) {
	$status->fatal( 'filenotfound', $srcPath );
}

The behavior change is that log items are created on each upload. This was fine for us.

We could certainly improve the current handling of graph image uploads, and make it work again. I started working on a patch to do that a little while ago: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GraphViz/+/519962

But I think the main issues with this approach is that a) it means that every version of every previewed graph is also added to the database; and b) every time someone views an old revision of page the graphs will be re-rendered. I guess that's been the way for years and it's not bothered anyone though. :)

I know some people have suggested that it's a bad idea to have the web server executing external commands as well, but that seems less of a worry given that thumbnails are generally done that way on most wikis.

For the preview scenario, we figured out a way to base64-encode the rendered PNGs and display them with data URIs (data:image/png;base64,...) instead of saved wiki images. This way no previews end up in the FileRepo or Database. The old revisions is a good point! I think it would be okay to do the same in that case.

I guess you can never be too safe on that last point. It also may scale better if you decouple the wiki servers from the graph rendering services, but it would only really matter at large scales.

Edit: I just saw your comment on T215391. Basically, we trick ImageMap and the Parser into believing that the image is uploaded by adding it to the LocalRepo cache. Then, we do a preg_replace on the images src, swapping it with the base64 data.

If there is an external command to render as with graphviz/plantuml and other extensions i think it's not much more of an effort to create a local git repository to keep track of the rendered/cached results - with tagging it would be quite easy to access the information and supply it for rendering. This approach would mean to abandon the approach of making the rendered results directly available as images via the mediawiki mechanisms in the traditional way images/e/f ... . Instead some kind of other access mechanism would be needed - i don't know whether this can be nicely intergrated with newer Mediawik APIs. The extra space needed for storage might be o.k. in quite a few use case scenarios. If a few extensions would share the base code for this the effort of creating such a solution might be worthwhile. Currently I'd also need such an approach for the plantuml and openjscad extensions.

In T226616#5727002, @Seppl2013 wrote:

If there is an external command to render as with graphviz/plantuml and other extensions i think it's not much more of an effort to create a local git repository to keep track of the rendered/cached results - with tagging it would be quite easy to access the information and supply it for rendering.

I think this makes sense in a single-server scenario, but it can become way more complex in a cluster. It would be interesting to investigate using the newly developed slots and multi-content revisions. I'm not sure what git could provide that MediaWiki's revision system couldn't.

I've been experimenting with splitting out the graph-rendering to a service:

• Service: https://github.com/samwilson/diagrams-service
• Extension: https://github.com/samwilson/diagrams-extension

(Still pretty rough, but sort of works. I didn't try incorporating anything into the Graphviz extension for now.)

@Seppl2013 @osorio-juan-microsoft: You both seem to be saying that it'd be good to have old copies of rendered graphs. Why is this? Isn't it enough to be able to re-render them at any time from previous revisions in the wiki content? Or is this about future-proofing against changes in graph syntax? Sorry if I'm missing the obvious.

@Samwilson: The assumption is that rendering with external programs like dot or openjscad is too costly a process to do it without caching. So there should be a copy of the rendered image in png, svg or whatever format readily available. If two users now render different versions of a page obviously there is a need for multiple version of such png and svg files. Cluterring the filesystem with these in a straight-forward manner seems awkward to me. I'd prefer to be able to put things under proper version control from an external tool. I don't like quite a few of the "do-it-yourself" approaches of mediawiki. If you use these solutions you are stuck with the mediawiki way of doing things instead of moving with the readily available APIs that are evolving "out there". I think the mediawiki architecture should be much more open and flexibile than simply having a two layer folder hierarchy that will make your https://commons.wikimedia.org/wiki/File:Sous_la_Tour_Eiffel_1.jpg url end up as https://upload.wikimedia.org/wikipedia/commons/thumb/b/bd/Sous_la_Tour_Eiffel_1.jpg/800px-Sous_la_Tour_Eiffel_1.jpg when looking at the thumbnail of it - the b/bd part is just ridiculous as of 2019 since it's got technical reasons that should be hidden.

https://github.com/pjkersten/PlantUML/blob/master/PlantUML.body.php

has the source code of the plant uml extension. There is a pull request

https://github.com/pjkersten/PlantUML/pull/29

which proposes to change the image handling of PlantUML see e.g. https://github.com/pjkersten/PlantUML/pull/29/commits/edf0f70cafa94fa27835ca4b02fc040f0b4e81b5

It'd be much simpler to split the wiki side of things from the external-command-running side of things.

That is true. I also would like to add a Special Page like the one from http://wiki.bitplan.com/index.php/Special:CMSFrontend see https://github.com/BITPlan/WikiCMS/blob/master/specials/SpecialCMSFrontend.php

There are other extensions that call external code to create graphics see e..g

https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/3D/+/master/src/ThreeDHandler.php

which uses xvfb to do so. It uses MediaWiki Shell

Part of our e-mail discussion might need to be copied to the Phabricator task - feel free to pick the relevant bits.

For me the weeks until christmas are the busy season. I might be able to work a bit more seriously on this from end of december. Until then it's only a few hours on the weekends.

Hope we'll make some progress by then. My main question is:

What is the work around for now? Would it be sufficient to readd the img_description column that graphviz is complaining about?

On 2020-01-14 at 03:57 Sam Wilson wrote:

I've done a little bit more work on this topic, and released a new experimental extension to capture my ideas:

https://www.mediawiki.org/wiki/Extension:Diagrams

It's still pretty limited, but works to some extent.

http://wiki.bitplan.com/index.php/GraphViz_MediaWiki_extension has my recent steps

Change 620525 had a related patch set uploaded (by Alex Mashin; owner: mashin):
[mediawiki/extensions/GraphViz@master] Make GraphViz compatible with Actor and Comment migration

https://gerrit.wikimedia.org/r/620525

Could anybody review the change https://gerrit.wikimedia.org/r/620525?

Is there still someone working on making GraphViz compatible with Mediawiki 1.33+?

So far there is no replacement for the functionality GraphViz adds to SemanticMediawiki when using the SemanticResult extension graph format. My company is willing to contribute financially to the development to making GraphViz compatible with the current Mediawiki LTS (1.35) and potentially future LTS versions.

@AID-PMBD just get in touch with me @Samwilson and I have been working on a solution which works in my wikis see http://wiki.bitplan.com/index.php/GraphViz. http://wiki.bitplan.com/index.php/Special:Version. There is a contact form at http://www.bitplan.com/index.php/Contact or use any of the contact options at http://training.bitplan.com/index.php/Wolfgang_Fahl

The extension that @WolfgangFahl mentions is Diagrams, and it's meant to be a drop-in replacement for GraphViz and PlantUML, but has some design differences (e.g. doesn't store graph images in the wiki, works when viewing old versions of a page, etc.). It can use an external service to render the graphs, or do so within MediaWiki. It's marked as experimental, but could probably graduate to beta by now. :-)

Change 620525 abandoned by Alex Mashin:

[mediawiki/extensions/GraphViz@master] Make GraphViz compatible with Actor and Comment migration

Reason:

https://gerrit.wikimedia.org/r/620525