=Motivation=
[[ https://en.wikipedia.org/wiki/2019%E2%80%9320_coronavirus_pandemic | Documenting the spread of COVID-19 ]] disease has exposed difficulties collaborating on data and rendering data visualizations. This latter problem is the motivation of this RFC, evidence can be found in T248707, T118783, and others. These front-end problems are related to the graphoid service in a way that not everybody agrees is necessary, so I will lay out the need for both the Graph extension and Graphoid as I see it. This part of the RFC is more complicated and subjective, so discussion and updates are especially welcome.
To draw a graph we need data and a function that does the transformation into visual objects. On top of that we can layer interactivity so people can play with the new representation. Data is usually a few hundred KB of text, but it can get up to tens of MB or more if you're pulling in TopoJSON to render maps and getting multiple dimensions per geographic object. Here are some product research questions and my assumed answers until better data is available:
- how much data will our larger datasets have? Up to 10MB
- how many people choose to interact with a graph? Less than 1%
- how many different representations of data do we need? 99% of graphs will probably be maps, bar graphs, line graphs, and (::shudder::) pie charts
To show why these guesses matter, let's assume a graph rendering a world map like the linked article above. And let's walk through rendering this client-side-only or with the aid of a service. Let's assume 1 million daily views.
- Service pulls in 1MB of data, renders a 60KB image. 99% of clients get the 60KB image and 1% get 1MB of rendering code plus 1MB of data. Bandwidth to serve: 80 GB / client CPU time: 14 hours
- Without a service, each client would pull in 2MB and wait 2-60 seconds depending on their processing power, let's say 5 seconds on average. Bandwidh to serve: 2 TB / client CPU time: 2 months
This makes the case for pre-rendering graphs on the server pretty easy to make, based on just one article. So, let's move on to the existing solution, Graphoid. This service tried to do everything that Vega is capable of, including pulling in and transforming data from external APIs. For this reason, it cut through our infrastructure in ways that left some scars. Problems are documented in the stewardship request, T211881, and mentioned in other RFCs and tasks T119043, T98940, etc. I hope this RFC can be a place where we centralize these discussions. These are my takeaways but do feel free to edit and add your own:
- Stewardship request: T211881. The service integrates too deeply into page rendering. It created a situation where it has to know about the varnish caches, page property caches, parser caches, complicated timelines from when an edit happens to how it affects each cache, etc. And therefore folks working in those areas need to know about Graphoid as well. The next version should try to be as independent and functional as possible. To @akosiaris's point, this will also allow teams to work independently especially now that schedules are reduced. Also, another key point there is testability and performance monitoring. Graphoid calls out to mediawiki in a circular way so testing it in isolation can be very tricky.
- thoughts from @Bawolff: https://www.mediawiki.org/wiki/User:Bawolff/Reflections_on_graphs. My takeaway here is that we can and should support other rendering engines outside of Vega. Especially given the assumption about diversity of charts above.
- initial motivation from @Yurik: https://meta.wikimedia.org/wiki/User:Yurik/I_Dream_of_Content. This vision was always great. The solution should not sacrifice the general idea that interactive and visual content are important.
The problems with Graphoid are:
- hard to maintain because of tight coupling with many and complex parts of our overall system
- hard to test because of calls back to mediawiki API
- lots of errors connecting to external APIs for data, this makes SRE hard
The requirements are:
- as isolated from page rendering as possible
- takes all required parameters up front so as to not have to make calls to Mediawiki
- takes all data up front so as to not need external APIs
- updating when new data is available or graph is changed should not be made too complicated by the other simplifications
- flexible as to the graph rendering engine. Anything that can render on nodejs should be allowed, not just Vega.
**Affected Components**: Graphoid
**Initial Implementation**: @milimetric (me) with @Pchelolo for code reviews and @akosiaris for "hey you're still doing it wrong"s (and the blessing of @Nuria, at least for now :))
**Code Steward**: @milimetric until I find a team that enthusiastically wants to support this going forward
==Existing plans and the Graph Extension==
As of this writing, the Graphoid service is due to be undeployed. This proposal does not seek to block that work, indeed it seems like a good idea to undeploy the current service, make the graph extension capable of working client-side-only, and support it with a more stand-alone service going forward. The graph extension currently supports only Vega and the new service will be flexible. This is where we should keep in touch and do the product work to understand the needs of our communities so we offer the right features.
=Proposed Solution=Solution A=
Do this in two phases. Phase 1 would be the simplest possible lambda service:
- Input =>
- Graph spec. For vega, this would be the json definition, and it would include the data so there is some work here to deal with specs that include URIs
- Fully realized data, including artifacts like TopoJSON, country codes, etc
- Rendering Engine. For example: vega1, vega5, vega-lite1, or bar-chart (the implications of that last option are a big topic so initially just vega*)
- Rendered Image options. Size, format, etc.
- Output <= rendered graph image
A graph extension updated to work with this service would have to go through this process:
- realize all the data required by calling all the APIs and URLs referenced in the spec provided by the editor, including any transformations
- update the spec to refer to the data directly and post along with the other inputs to the service. This will be the hardest part, and will have to be done client-side before saving. It's a new type of interaction for editors (I think?) but it will deliver many more advantages in terms of support for graphs overall, due to simplified maintenance
- save the resulting image URL with the page. Here we probably want to save the original Vega spec before materializing the data, because it's more succinct. But we could also not save it at all and punt this part to Phase 2 and open a discussion about a new MCR slot. Not saving it at all means more work for editors to draft and save it on their own. Because of this complexity, we may want to delay integration with the graph extension until Phase 2
In Phase 2, we do work around the service to allow a future graph extension to integrate easily with it. We need to solve how we store the graph definition, data, and how to store and purge the resulting images. As I understand from @Pchelolo the evolution here will benefit a lot from lessons learned with Mathoid. This implementation should closely track the latest thoughts on Mathoid and take advantage of existing mechanisms. Some goals would be:
- UX of editors. Graph definitions should be easy to edit and validate client-side. Building templates on top of common types of graphs should be easy.
- It should be easy to copy and render old versions of graph definitions along with the data they referenced, if the APIs or URLs used allow it. This is where we may need to open up discussion about storing these artifacts in an MCR slot.
(...still drafting this, just hit "Create" because editing phab tasks in the popup seems unnecessarily cruel)
Tagging other people that might be interested in this discussion:
- @kaldari is working on a bot to update some of the most needed and critical graphs, that bot could use this service before even the graph extension
- @Tnegrin is project managing the immediate response to graphoid/graph extension problems
- @Abit is leading a weekly sync
- @Seddon who is working on making the graph extension client-side-only