@MarkTraceur I filed this request - you should probably be the one to manage the request given you'll be project managing the contractor, but let's discuss. Feel free to edit the request, as I may be not thinking holistically enough...or I may be thinking too holistically!

In no particular order:

• Hiring is ongoing, but we're close. Start date looks like it's still a few weeks out, however.
• Based on advice we've gotten so far, I don't believe syntactic changes will be required. In theory, the system will Just Work™. If changes are needed, we don't know what they are yet, but I'd expect that to be part of the report as work is closed and before deployment occurs.
• If we want to roll this out to particular wikis piecemeal, I'd love some help figuring out which ones should go first. I believe I've heard there are counts for graph usage per-wiki floating around somewhere, which may help.

In T236892#5689943, @MarkTraceur wrote:

• If we want to roll this out to particular wikis piecemeal, I'd love some help figuring out which ones should go first. I believe I've heard there are counts for graph usage per-wiki floating around somewhere, which may help.

Link: https://phabricator.wikimedia.org/T211881#4878324

There's some more good data tidbits in there. I'll look more into ordering release next week.

Most of this work will be scoped to broadcasting the change once we've determined impact and rollout.

I think we'll need someone with better analytics skills than I have to help look at this, but I've been going over the data in the task link above. As stated there, only about 10 wikis are using most of the graphoid service. The largest that will be impacted, enwiki, is interesting. During the two-week measured frame it got the most hits for graphoid at over 21k requests. However, as @dr0ptp4kt found in T211881#5065149...

There's a grand total of 182 pages bearing <graph> in the source according to that primitive search query (that includes a low level of non-executing documentation).
113 are in User namespace. 39 of those invocations are in Article namespace. 8 are in Wikipedia namespace. 5 of them are in Template namespace; 4 of those 5 templates have just one article associated with them bearing the <graph> tag. The other 17 other usages of the tag are scattered about.

The most important offender likely being https://en.wikipedia.org/wiki/Template:Graph:Street_map_with_marks, which is used in nearly 1.2k articles.

I imagine the situation is like this for the other ten wikis. It'd be very helpful in messaging to know if indeed there are a few templates that are widely propagating <graph>, and if they can be updated or otherwise alerted to this change. It'd be helpful to be more targeted here in the communications so as to not avoid an unnecessary panic when the impact might be small.

@Keegan this task would welcome an update. TY!

Once a contractor is hired and onboarded to the project, we'll start the outreach to the few places actively using this tag. Still very much a WIP.

In T236892#5729653, @Keegan wrote:

I think we'll need someone with better analytics skills than I have to help look at this, but I've been going over the data in the task link above. As stated there, only about 10 wikis are using most of the graphoid service. The largest that will be impacted, enwiki, is interesting. During the two-week measured frame it got the most hits for graphoid at over 21k requests. However, as @dr0ptp4kt found in T211881#5065149...

There's a grand total of 182 pages bearing <graph> in the source according to that primitive search query (that includes a low level of non-executing documentation).
113 are in User namespace. 39 of those invocations are in Article namespace. 8 are in Wikipedia namespace. 5 of them are in Template namespace; 4 of those 5 templates have just one article associated with them bearing the <graph> tag. The other 17 other usages of the tag are scattered about.

The most important offender likely being https://en.wikipedia.org/wiki/Template:Graph:Street_map_with_marks, which is used in nearly 1.2k articles.

I imagine the situation is like this for the other ten wikis. It'd be very helpful in messaging to know if indeed there are a few templates that are widely propagating <graph>, and if they can be updated or otherwise alerted to this change. It'd be helpful to be more targeted here in the communications so as to not avoid an unnecessary panic when the impact might be small.

In case its useful, For an entirely different purpose, I did a (mostly superficial) analysis of how the graph extension is used on en wikipedia at https://www.mediawiki.org/wiki/User:Bawolff/Reflections_on_graphs#How_is_the_Graphs_extension_used_currently

Moving to CommRel-Specialists-Support (Apr-Jun-2020), assuming this is not done yet.

I've emailed tech ambassadors and made a first post at https://en.wikipedia.org/wiki/Template_talk:Graph:Chart

More outreach is in the works, including Tech News.

I’m not sure what “client-side only JavaScript feature” means. Is it that JavaScript will load for all graphs (in contrast to the current situation where it’s loaded only on-demand and not loaded for simple graphs at all)? Or is it that graphs will be rendered only on client side, without any server-side (PNG) fallback? Latter case would mean that Grade C browsers (and other browsers incapable of executing JavaScript, e.g. users with unreliable internet connection or users with NoScript) wouldn’t get any graphs, which is against the promise set out on the compatibility page (see link) that “content is presented in a readable manner”. I really hope this won’t happen.

@Tacsipacsi, that's correct, it would be rendered only on the clients with sufficient JavaScript support. It's not ideal, but it's a compromise to avoid graphs being disabled altogether. Archiving of the PNGs to the File: namespace is one possible way to cut over content meantime where that's preferred over the client rendered approach.

I think even disabling graphs for all would still be a better solution than disabling them only for some. Imagine that the article says “the change over the years is shown on the below graph:”—and there’s no graph. There is a clear promise on https://www.mediawiki.org/wiki/Compatibility that the content will be presented for Grade C browsers in a readable manner; this must be followed by all WMF software.

This has been added to Tech/News/2020/21.

In T236892#6135375, @Tacsipacsi wrote:

I think even disabling graphs for all would still be a better solution than disabling them only for some. Imagine that the article says “the change over the years is shown on the below graph:”—and there’s no graph. There is a clear promise on https://www.mediawiki.org/wiki/Compatibility that the content will be presented for Grade C browsers in a readable manner; this must be followed by all WMF software.

This feature is a Javascript feature to begin with, whether rendered on the server or in the browser, so compatibility as outlined is not applicable here.

However, the underlying point is understood, and there's active discussion about in T249419 about how to replace the service. Please consider this change to be but one piece of the work to re-do graphs and make them more robust, while at the same time deprecating parts of the architecture that are holding up progress.

I've just translated this for Tech news and I'm confused about what this even means. To users with even less knowledge, this sounds like you're about to break loads of stuff and boil the CPUs in everyone's phones and tablets.

Users who don't understand all the tech may think that every graph, even static SVG files, will now require JS. Sure, I understand that won't be the case, but Tech news isn't read by just me. And even I don't understand exactly what is covered by this, why this has to happen and how badly my phone will overheat because of it.

Agree with AlexisJazz, about need for more explanation. How is client-side JS more maintainable than server-side?

A graphing feature seems that it would be very useful, but I’d never heard of it. Wondering why usage is so low.

In T236892#6185989, @Pelagic wrote:

Agree with AlexisJazz, about need for more explanation. How is client-side JS more maintainable than server-side?

Client-side rendering is handled through the browser, and for an individual the cost is very, very low because you're only rending for yourself; CPU usage is a fraction of a percent, there's straight up no danger to individuals.

Scale this up to having the server handling all the requests all the time, and it's the central rendering place that's in danger falling over. It's reached an unsustainable point.

In T236892#6135287, @dr0ptp4kt wrote:

@Tacsipacsi, that's correct, it would be rendered only on the clients with sufficient JavaScript support. It's not ideal, but it's a compromise to avoid graphs being disabled altogether. Archiving of the PNGs to the File: namespace is one possible way to cut over content meantime where that's preferred over the client rendered approach.

I just want to point out that this task is very unclear, and therefore the notice to users has been ineffective. As Tacsipacsi notes, "client-side only JavaScript" is incomprehensible jargon. If I understand correctly, the real meaning of this task is "Deletion and discontinuation of static/PNG images so far produced and embedded in pages for graphs".

Getting rid of the static images has all sorts of wide-ranging consequences, which are pretty clear once you ask yourself what clients rely on static images and don't run JavaScript. However, because this has never been stated clearly, I wonder whether the consequences have been properly assessed.

Removing static images of graphs is a deal breaker for Kiwix, because graphs simply cannot be rendered client-side in a fully offline manner (they can pull data from Commons, Wikidata, maps, etc.). Given that T249419: RFC: Render data visualizations on the server appears to be progressing, I'd think it makes sense to set some deadline on that, and if it doesn't get resolved to SRE/etc. statisfaction, remove the Graph feature from content namespaces/wikis due to lack of maintainership.

In T236892#6186035, @Keegan wrote:

In T236892#6185989, @Pelagic wrote:

Agree with AlexisJazz, about need for more explanation. How is client-side JS more maintainable than server-side?

Client-side rendering is handled through the browser, and for an individual the cost is very, very low because you're only rending for yourself; CPU usage is a fraction of a percent, there's straight up no danger to individuals.

Scale this up to having the server handling all the requests all the time, and it's the central rendering place that's in danger falling over. It's reached an unsustainable point.

So whatever this is, it is producing something tailored personally to the user. The question is why.

Looks like we're playing Guess Who? or something.

I've asked someone from Engineering to clarify some things here, I think there's some misunderstandings but I'm not deep enough into the technical terms to answer on my own.

Hi! Looks like while I was looking in other directions, this garnered a fair bit of attention. I'm sorry to say we may have had some wires crossed in explaining the motivation and reality of these changes, so let me try and clarify a few things quickly:

During the planning stages (approximately 12 months ago), we explored the possibility of "simply" fixing the service to use a newer version, but decided that it was an unrealistic expectation. We opted to turn the service off instead, switching the extension to client-side-only. I think that decision was well-documented elsewhere. Graphoid itself is a problem when it comes to maintenance...Apart from it being a complicated and difficult-to-maintain piece of infrastructure, it also was meant to be a temporary solution for non-JavaScript clients that couldn't render the graphs on their own. Beyond that, as I understand it, the old version of node.js required for Graphoid and its requisite libraries was blocking an upgrade of a particular set of servers, which has stalled another team for upwards of 18 months now (according to T211881#5332195).

To address the more substantive point about the need for static image fallbacks, we (read: Seddon) have added a special option in the <graph> tag in afb3e90caee17aac4d54f60ec47e01f9016c83bd that allows manual setting of a fallback image. This didn't get communicated well, which I'm sorry for, but it is present. It will require conscious decisions on the parts of graph creators and article editors to identify important graphs, render them, and add the attributes to the various tags. However, it is possible to create fallbacks.

In summary: We were incredibly aware, going into this project, that we were planning to make a giant trade-off, and we were additionally aware that there may be further, unforeseen trade-offs (trades-off?) down the line. Those trade-offs may seem slightly different now, given 6-9 months of hindsight and other activity, but at the moment unless we want to scrap the plan entirely and leave the outdated servers and other infrastructure in place (which we'd need a really compelling argument to support), we don't have an alternative. I don't see that being the case here.

Thanks, Mark. And my apologies, folks, I didn't catch what the final decisions were on architecture between the go-ahead for this in mid-May and where we wound up. We'll work on making sure documentation is in order for setting fallback images, and get the word out about it to communities. I think this affects around ten to twelve wikis that actively use the graph tag in some way, I'll go back over the data.

@Keegan @MarkTraceur Can you provide a query for cirrus search to find affected articles?

In T236892#6301702, @AlexisJazz wrote:

@Keegan @MarkTraceur Can you provide a query for cirrus search to find affected articles?

I think this works:

all: insource:"<graph>" insource:/"<graph>"/

Found at T211881#5065149

The review in that comment is fairly informative from an enwiki standpoint, which shows its usage is limited in the article space and is mainly propagated through a few templates and talk page usage.

A preliminary look at the other wikis (~10) that use the graph tag finds similar results.

In T236892#6301847, @Keegan wrote:

In T236892#6301702, @AlexisJazz wrote:

@Keegan @MarkTraceur Can you provide a query for cirrus search to find affected articles?

I think this works:

all: insource:"<graph>" insource:/"<graph>"/

Found at T211881#5065149

The review in that comment is fairly informative from an enwiki standpoint, which shows its usage is limited in the article space and is mainly propagated through a few templates and talk page usage.

A preliminary look at the other wikis (~10) that use the graph tag finds similar results.

@Keegan @dr0ptp4kt No. This is not complete. That query is not case-insensitive so <Graph> won't match, {{#tag:graph won't match and possibly the worst offender, <graph mode=interactive won't match.

In cases where mode=interactive is not used, there is no reason to use Javascript. You should be caching these. The interaction in the interactive ones doesn't seem to work very well. This thing for example: https://en.wikipedia.org/wiki/Template:Global_Heat_Maps_by_Year confuses the hell out of me. But that's the JS version. I don't know if it worked any better before.

Oh crap, I just broke it entirely. The whole map is blue now and the current year is gone. The scale at the bottom is still there, but flatlined. Can't do anything now. Having this constant jquery spinner until you activate it is also miserable. If it worked any better, I suggest you turn server side rendering back on and actually consult with the community for a solution. I'm not saying server side rendering should remain indefinitely, but ask the community.

As an example, there is https://en.wikipedia.org/wiki/List_of_most_expensive_paintings#Interactive_graph but the table below it is a better way to display the data. The graph could be replaced by a static SVG to provide some idea of the bigger picture while the table offers the details.

In T236892#6302246, @AlexisJazz wrote:

In T236892#6301847, @Keegan wrote:

In T236892#6301702, @AlexisJazz wrote:

@Keegan @MarkTraceur Can you provide a query for cirrus search to find affected articles?

I think this works:

all: insource:"<graph>" insource:/"<graph>"/

Found at T211881#5065149

The review in that comment is fairly informative from an enwiki standpoint, which shows its usage is limited in the article space and is mainly propagated through a few templates and talk page usage.

A preliminary look at the other wikis (~10) that use the graph tag finds similar results.

@Keegan @dr0ptp4kt No. This is not complete. That query is not case-insensitive so <Graph> won't match, {{#tag:graph won't match and possibly the worst offender, <graph mode=interactive won't match.

In cases where mode=interactive is not used, there is no reason to use Javascript. You should be caching these. The interaction in the interactive ones doesn't seem to work very well. This thing for example: https://en.wikipedia.org/wiki/Template:Global_Heat_Maps_by_Year confuses the hell out of me. But that's the JS version. I don't know if it worked any better before.

Oh crap, I just broke it entirely. The whole map is blue now and the current year is gone. The scale at the bottom is still there, but flatlined. Can't do anything now. Having this constant jquery spinner until you activate it is also miserable. If it worked any better, I suggest you turn server side rendering back on and actually consult with the community for a solution. I'm not saying server side rendering should remain indefinitely, but ask the community.

@AlexisJazz What you're bringing up here isn't the task at hand, server-to-client-side rendering, and are instead the overall problems with the Graphoid service. Where the graph is rendered, whether the server or the client, is immaterial to the problems you're describing. It's the implementation of graphs themselves.

I'll refer you to read over T211881: graphoid: Code stewardship request where engineering is discussing the very questions you are asking is well: why are graphs being done this way in the first place? Why are fallback images not autogenerated? And many more questions over the course of the past 3-4 years now, at this point blocking further development of other server resources.

What's important for the communities from the standpoint of where the graph is rendered is in having a fallback image if a user doesn't have Javascript. This is what's been sorted out as the deprecation of server-rendering has been developed over the course of the past few months, and the next steps to moving forward on this is proper documentation for setting fallbacks.

Documentation for setting fallback images on Commons is up: https://www.mediawiki.org/wiki/Extension:Graph#User_defined_fallback

I need to work on getting this translated and distributed over the next week or so.

Hi @Jseddon , did I miss comms going out? Or is closing this a way of saying you'll be dealing with those yourself? TYVM!

OK, this was https://lists.wikimedia.org/pipermail/wikitech-ambassadors/2020-November/002385.html .