If we're talking about something used by both MediaWiki and Phabricator, it sounds like what this calls for is a service with accompanying MediaWiki and Phabricator extensions to interface with the service.

Discourse integration would also be nice, so all our technical spaces can use the same diagram markup. Discourse has plugins for Mermaid and PlantUML although neither has much commit activity.

In T183689#3928927, @Harej wrote:

If we're talking about something used by both MediaWiki and Phabricator, it sounds like what this calls for is a service with accompanying MediaWiki and Phabricator extensions to interface with the service.

PlantUML works like that, there is a server component that you can host locally. Mermaid runs in the browser (which means users with JS disabled wouldn't see the graphs, although I'm sure it's not too hard to run it as a Node service). GraphViz has plenty web service wrappers and even a full-JS implementation.

There's a GraphViz Discourse plugin now, too.

The fact that this task says "on Phabricator and mediawiki.org" makes me want to close this as non-actionable. Two totally different and unrelated systems... :-/

How is that non-actionable? The action is trivially described by the title. Less manageable, maybe; OTOH it wouldn't be very useful to end up with non-compatible diagram notation syntax on the two primary platforms for discussing system architecture.

In T183689#6218959, @Aklapper wrote:

... makes me want to close this as non-actionable. Two totally different and unrelated systems... :-/

Maybe two subtasks for the two systems after a common notation is established.

It would be great to having a diagram generation tool on mediawiki.org. Of the three extensions listed, the Mermaid extension is the only one marked as stable, so I tried re-creating a few diagrams from mediawiki.org to see how Mermaid works.

Overall, I found Mermaid’s syntax to be fairly straightforward and easy to use. In my experiments, it worked best for decision trees, sequence diagrams, and simple-to-medium complexity flowcharts. Mermaid also has nice features that allow you to assign links to nodes, incorporate Font Awesome icons, and customize styling. If it’s possible to enable on mediawiki.org, I think the Mermaid extension would be a useful tool for improving documentation.

For example, this diagram of MediaWiki’s architectural modules could be replaced with a Mermaid version, replacing the complex imagemap and making the diagram easier to maintain. See this wikifarm page for a demo.

Source:

{{#mermaid:graph LR;
 index.php-->MediaWiki;
 MediaWiki-->Page;
 Page-->Action;
 Action-->ViewAction;
 ViewAction---id1{ };
 id1{ }-->Parser;
 id1{ }-->Cache;
 Cache-->OutputPage;
 Parser-->OutputPage;
 OutputPage-->Skin;

 classDef dottedOutline stroke-dasharray: 3, 3;
 classDef whiteFill fill:white;
 class ViewAction dottedOutline;
 class index.php,MediaWiki,ViewAction,OutputPage,id1,id2 whiteFill

 click index.php "https://www.mediawiki.org/wiki/Manual:Index.php"
 click MediaWiki "https://www.mediawiki.org/wiki/Manual:Architectural_modules/MediaWiki"
 click Page "https://www.mediawiki.org/wiki/Manual:Architectural_modules/Page"
 click Action "https://www.mediawiki.org/wiki/Manual:Architectural_modules/Action"
 click ViewAction "https://doc.wikimedia.org/mediawiki-core/master/php/classViewAction.html"
 click Parser "https://www.mediawiki.org/wiki/Manual:Architectural_modules/Parser"
 click Cache "https://www.mediawiki.org/wiki/Manual:Architectural_modules/Cache"
 click OutputPage "https://www.mediawiki.org/wiki/Manual:Architectural_modules/OutputPage"
 click Skin "https://www.mediawiki.org/wiki/Manual:Architectural_modules/Skin"
}}

Output:

To see more examples of diagrams from mediawiki.org re-created with Mermaid, see the slides from my presentation to the Friends of the Docs group on 2020-06-17.

Thanks a lot for looking into this @apaskulin!

Extension stability does not mean much IMO, I would rather choose a solution based on the strength of the various markup languages + the tool used for rendering the image on the server side (which is something MediaWiki will just shell out to). We can fix up any of these extensions (or write a new one) without too much effort, they are really just scaffolding between the wiki and the (non-PHP) rendering server or library used.

Is comparing the other options (not necessarily the extensions, there are online tools like http://www.plantuml.com/plantuml/ and https://dreampuf.github.io/GraphvizOnline/ ) something you'd be interested in doing?

there is another extension looking interesting: https://www.mediawiki.org/wiki/Extension:Diagrams , what you think about enabling that one @Tgr ?

In T183689#7661676, @ThurnerRupert wrote:

there is another extension looking interesting: https://www.mediawiki.org/wiki/Extension:Diagrams , what you think about enabling that one @Tgr ?

@Samwilson is probably the better person to ask about that.

The Diagrams extension is probably not ready for deployment on Wikimedia wikis. It's main issue is that it needs a secondary diagram-rendering web service to be hosted somewhere, and I'm not sure how that'd work in WMF production. The extension does have the option (actually the default) to render locally, but that requires shelling out to run the external programs, so isn't the best way.

So, there are two directions to take:

• use Mermaid, which does not require shelling out. Extension:Mermaid: is not on Gerrit + seems to require yarn to install the library? (meaning we'd have to fork it), Extension:Flex Diagrams is on Gerrit and does bundle Mermaid, but also bundles bpmn-js which we don't need and is 50KLoC.
• use GraphViz or PlantUML: requires shelling out (both full-JS GraphViz implementations, Dagre and viz.js, are unmaintained), so we'd have to turn one of them into a ShellBox-compatible web service. Both tools are packaged for Debian; PlantUML is Java-based while Graphviz is native so I assume the latter is preferred. Extension:GraphViz is on Gerrit; unmaintained and needs to be fixed to even work with current MediaWiki but it's small extension so that's probably not too much effort.

@Samwilson who would be able to judge if WMF hosting such a service would be an option, or, in general, if, calling a service and caching the service calls result to render something is desireable?

It would be a good part of Mediawiki if there is some software diagrams. Because it will help us easily to realise a topic.

as additional info:

• there seems to be no standard notation implemented by multiple tools. mermaid has a ticket to adapt plantuml syntax: https://github.com/mermaid-js/mermaid/issues/149 and a couple of followups to support plantuml diagram types.
• sometimes plantuml syntax was used though, e.g. for c#: https://github.com/plantuml/plantuml/issues/63 .
• the c# based .plantuml language service for vscode extracted as well some grammar: https://github.com/kiebor/PlantUml-Language-Service/blob/master/src/Grammars/plantuml.json
• plantuml uses graphviz to layout diagrams.
• graphviz , and its DOT notation, does not well support e.g. sequence diagrams. what is one line in mermaid or plantuml is quite large in others: https://stackoverflow.com/questions/1491322/graphviz-top-to-bottom-and-left-to-right.
• mermaid does not support some of the UML diagrams, like activity diagram: https://github.com/mermaid-js/mermaid/issues/1674 .

use GraphViz or PlantUML: requires shelling out (both full-JS GraphViz implementations, Dagre and viz.js, are unmaintained), so we'd have to turn one of them into a ShellBox-compatible web service. Both tools are packaged for Debian; PlantUML is Java-based while Graphviz is native so I assume the latter is preferred. Extension:GraphViz is on Gerrit; unmaintained and needs to be fixed to even work with current MediaWiki but it's small extension so that's probably not too much effort.

The Diagrams extension already has a web service; I'm not sure what it'd take to make it ShellBox-compatible (mw:Shellbox and other docs seem to be more about setting up the server and using command in MediaWiki, rather than writing new services for the server — but I'll dig into it a bit more, I'm sure I could figure it out). The web service is supposed to handle rendering and caching.

The troubles with the GraphViz extension, that I found (and which were why I ended up writing Diagrams), were that it didn't handle previewing or old revisions and it wasn't compatible with the AWS extension. I tried fixing it a few times, but gave up.

there seems to be no standard notation implemented by multiple tools.

That's true, and I'm not sure what can be done. I think GraphViz is probably the widest-supported syntax, because it's pretty old, but maybe Mermaid is more popular these days. Mermaid gets away from the troubles of calling 3rd party rendering programs, but means graphs can't be as used in some situations (e.g. exporting to some formats). My idea with Diagrams was to just add support for everything. :-)

In T183689#7666295, @Samwilson wrote:

The Diagrams extension already has a web service; I'm not sure what it'd take to make it ShellBox-compatible (mw:Shellbox and other docs seem to be more about setting up the server and using command in MediaWiki, rather than writing new services for the server — but I'll dig into it a bit more, I'm sure I could figure it out). The web service is supposed to handle rendering and caching.

AIUI Shellbox is an RPC-style service that proxies a shell command to another machine; you don't need a web service specific to your extension to use it. From what I can gather from skimming the existing files, you'd need to create a Helm chart for a new Kubernetes app (example), add a DNS record (e.g. rODNS2bd188027801: Add *.svc.{codfw,eqiad}.wmnet entries for new Shellboxes), load balancer config (e.g. rOPUPec8270c1abbf: Add LVS for new Shellboxes: media, syntaxhighlight & timeline), add ProductionServices config (rOMWCaeaf74c3ba55: ProductionServices: Add new Shellboxes), add Shellbox URL config (rOMWC31a92c4d5bd2: Have SyntaxHighlight use Shellbox service on group0 wikis). Or maybe you can use the existing shellbox instance (not sure what are the rules around that) and then you only need the last two steps.

If you want to use your own web service instead of Shellbox, the steps would be roughly similar, just more effort to create a chart from scratch, set up the CI pipeline etc.

That's true, and I'm not sure what can be done. I think GraphViz is probably the widest-supported syntax, because it's pretty old, but maybe Mermaid is more popular these days. Mermaid gets away from the troubles of calling 3rd party rendering programs, but means graphs can't be as used in some situations (e.g. exporting to some formats). My idea with Diagrams was to just add support for everything. :-)

GraphViz notation is not super user-friendly (although way more capable), and it's server-side so you don't get a responsive editor interface like you do with Mermaid, but something is better than nothing and I'm not sure getting Mermaid security-reviewed would be realistic.

In T183689#7666295, @Samwilson wrote:

use GraphViz or PlantUML: requires shelling out (both full-JS GraphViz implementations, Dagre and viz.js, are unmaintained), so we'd have to turn one of them into a ShellBox-compatible web service. Both tools are packaged for Debian; PlantUML is Java-based while Graphviz is native so I assume the latter is preferred. Extension:GraphViz is on Gerrit; unmaintained and needs to be fixed to even work with current MediaWiki but it's small extension so that's probably not too much effort.

The Diagrams extension already has a web service; I'm not sure what it'd take to make it ShellBox-compatible (mw:Shellbox and other docs seem to be more about setting up the server and using command in MediaWiki, rather than writing new services for the server — but I'll dig into it a bit more, I'm sure I could figure it out). The web service is supposed to handle rendering and caching.

You're looking for https://www.mediawiki.org/wiki/Manual:BoxedCommand - there are some example conversion commits linked from there. I did a brief skim of the extension code, I would suggest copying the FileRepo, etc. interactions from Score.

In T183689#7666328, @Tgr wrote:

In T183689#7666295, @Samwilson wrote:

AIUI Shellbox is an RPC-style service that proxies a shell command to another machine; you don't need a web service specific to your extension to use it. From what I can gather from skimming the existing files, you'd need to create a Helm chart for a new Kubernetes app (example), add a DNS record (e.g. rODNS2bd188027801: Add *.svc.{codfw,eqiad}.wmnet entries for new Shellboxes), load balancer config (e.g. rOPUPec8270c1abbf: Add LVS for new Shellboxes: media, syntaxhighlight & timeline), add ProductionServices config (rOMWCaeaf74c3ba55: ProductionServices: Add new Shellboxes), add Shellbox URL config (rOMWC31a92c4d5bd2: Have SyntaxHighlight use Shellbox service on group0 wikis). Or maybe you can use the existing shellbox instance (not sure what are the rules around that) and then you only need the last two steps.

Ideally whatever needs to be installed in the container can be installed as a Debian package. This isn't a strict requirement, it just makes things easier. I would say that using a separate Shellbox instance would be ideal.

If you want to use your own web service instead of Shellbox, the steps would be roughly similar, just more effort to create a chart from scratch, set up the CI pipeline etc.

We have a pretty good workflow for deploying nodejs services and a decent enough workflow for PHP apps. But I think using Shellbox provides a bit more containment and you don't have to write your own HTTP interface since you can use BoxedCommand.

Thanks for the info, and I'll definitely check out what Score is doing and copy it.

In T183689#7662311, @Samwilson wrote:

The Diagrams extension is probably not ready for deployment on Wikimedia wikis. It's main issue is that it needs a secondary diagram-rendering web service to be hosted somewhere, and I'm not sure how that'd work in WMF production. The extension does have the option (actually the default) to render locally, but that requires shelling out to run the external programs, so isn't the best way.

This is the same problem the Graph extension ran into with Graphoid. The RFC I started around that shone a light on all the technical problems we'd need to solve: T249419: RFC: Provide a mechanism by which we can render data visualizations on the server

the plantum author arnaud roques thinks about a greenfield approach to rewrite plantuml in haxe : https://github.com/plantuml/haxe00 . mindmaps are working. haxe's closed relate is lua. haxe can be transpiled into javascript, lua, php7 amongst others, its main usage is in games.

@Bugreporter2: This is not a request to enable a specific existing extension via an existing configuration setting.

In T183689#10940977, @Aklapper wrote:

@Bugreporter2: This is not a request to enable a specific existing extension via an existing configuration setting.

You're technically right, I think. It seems to be a request for a new extension that duplicates all (or at least most) of the functions of other extensions.