Page MenuHomePhabricator

CONSULTATION/PLAN: Managing Complex State and GUI on MediaWiki (e.g. for Wikidata/Wikibase UI)
Open, MediumPublic

Description

The Wikidata team is considering to use a MVVM/Single-State solution for Wikidata’s UI. What are requirements and concerns would be important to consider?

Context: Wikidata’s current UI is built on jQuery UI. Since jQueryUI shall be faded out, we are looking at possible future frameworks or paradigms to build our UI on. Our needs are:

  • Having a sustainable foundation
  • Being able to handle complex state dependencies (simplest are like: "if element x is in edit mode, set element y to saving mode")
  • A solution that is easy to learn for beginners and easy to read and reason about for our engineers.

State management and data/event propagation goes beyond of what OOUI can provide, as far as I (Jan) know. So an obvious candidate was looking into reactive ModelViewViewModel solutions like the React or Vue library, which combine with their respective single state libs, namely redux and vuex. (The MVVM would manage the DOM, Templates, event propagation, component composition, while the single state lib would manage the state/model and its connection to the backend)

We see the following possible advantages:

  • Better modularization
  • understandability of our code, in particular reasoning about event- and data-flow
  • better separation of concerns and testability for:
    • HTML templates
    • Component interactivity
    • Data manipulation
    • connection to backend-API
  • If we use a well documented framework, learning to contribute is much easier compared to software for which there is only auto-generated code-level-docs

Such idea poses the following questions:

  • Should we introduce a new dependency like a framework as Vue?
    • What would be the process of introducing such a dependency (if we agree on one)?
    • Can we agree on this (or another?) paradigm for managing complex UIs, so that it is not a Wikidata-only solution, but could be used by other Wikimedia projects in the future, too?
  • How will this work with OOUIjs? OOUI seems to be mainly responsible for creating DOM elements and this actions are usually owned by the MVVM framework.
    • One can use hooks to use libraries like OOUI and such, but it feels like having the same functionality twice.
    • A possible solution would be using OOUI styles and markup but leaving DOM creation to the framework.
    • One could also combine OOUIjs with a single state lib like Redux. (as suggested by @Jhernandez) However, I don't know how well suited OOUI is to be combined e.g. with Redux.

FAQ on reactive/single state frameworks

  1. Does using a reactive/single state, React-like solution mean we need to write mixed JS/CSS/HTML in a new syntax? (aka JSX)? → No, it is possible, but for many frameworks (e.g. Vue), HTML templates are used.
  1. Does that mean that people coming from Object oriented languages will need to learn a whole new paradigm – reactive, pure-functional programming? → While there are some elements of functional programming used in react-like-frameworks, I would (subjectively) say that few additional, totally new knowledge is needed and most can be covered by "take parameters, work with them, return values; don't manipulate non-local values"
  1. How does DOM access work? Does this mean no jQuery? DOM can be still be directly accessed. Libraries like jQuery can still be reused (even if they might not be necessary in many points any more). However, to change data or dom persistently, you need to tell the library (which is not unusual, afaic)
  1. Don't such libraries need much tooling? → It depends… e.g. Vue can be used without a loader and does not need an extra JSX compiler. It is often used with a loader (webpack, mostly), because many people seem to prefer it and it enables you to structure your code in single-component-files, which I admit, while disliking the dependency, is very comfortable. (thanks to @Jdrewniak for asking this)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Jan_Dittrich renamed this task from CONSULTATION/PLAN: Managing Complex State and GUI on Mediawiki (e.g. for WIkidata/Wikibase UI) to CONSULTATION/PLAN: Managing Complex State and GUI on Mediawiki (e.g. for Wikidata/Wikibase UI) .Feb 7 2017, 11:16 AM
Legoktm renamed this task from CONSULTATION/PLAN: Managing Complex State and GUI on Mediawiki (e.g. for Wikidata/Wikibase UI) to CONSULTATION/PLAN: Managing Complex State and GUI on MediaWiki (e.g. for Wikidata/Wikibase UI) .Feb 7 2017, 11:56 AM

@Jan_Dittrich Something that hasn't been brought to the table in our last week's Front-end-Standards-Group meeting as the question was targeted mainly on certain technical measurment. But should be taken into consideration in your decision: OOjs UI comes out of the box with strong support of internationalization (i18n, among others extra focus of RTL language support) and incorporating accessibility best practices, both things where other libraries might fall short.

@Volker_E: Yes, good point. Such things are probably a concern of the templates one feeds to react or vue – I suppose for the libs themselves it is "not their department".

One could see how much of it one could take over (e.g. to possible future templates) and how much is deeply integrated using OO(JS).

@Jan_Dittrich this is exciting. We're also looking at Vue.js for the Wikistats 2.0 rewrite. We have to write a prototype and make sure it performs the way we need, so we haven't decided yet but we have similar questions and concerns. We are coming from knockout, so we have a lot of experience with MVVM. I can give some highlights here, but first I want to say:

If you decide to go with Vue.js, I will happily volunteer as much time as I can every week to work on this UI with you. I've been waiting and hoping that our front ends embrace a more mainstream library so I can contribute, and I'm certain that many other developers out there in the world of Open Source feel the same. I have respect for particular pieces of OOJS, such as the sophisticated i18n and RTL work, but it's fighting an impossible battle of attracting real wide scale Open Source collaboration. Vue is modular, and plugins that handle i18n exist. Ideally we can make these better by incorporating the OOJS work in this area. Both are licensed under MIT, so that's not a problem. I think we could of course also use the standard style as defined by @Volker_E and examine how tricky it would be to integrate OOJS UI (not trivial, which is one of the problems: lack of modularity).

From our experience with MVVM and reactive programming, here are my main take-aways:

  • modularity and testability is fantastic. Having a pure view model that just manages data is inherently easy to test. You set observable properties and check that computed properties are set as expected.
  • the mentality is different, but closer to the spirit of "manage data, not code". This can take some time getting used to, and it's especially tricky in Knockout because it's not opinionated at all. This was one of the main reasons for us to consider moving to Vue. More opinionated frameworks, if careful, can be easier to learn because there are not as many wrong choices to make.
  • there is plenty of support on sites like stackoverflow. People sometimes worry that if they don't use React or the other most popular framework of the day they won't be able to find answers to their questions. But even knockout, which has a fairly small user base, has a very nice community of people that will help you out and a good database of topics that have already been covered. Knockout and vue.js are also just so simple that you can read the source and understand how they work pretty quickly.

And finally my quick thoughts on your FAQ:

  1. Does using a reactive/single state, React-like solution mean we need to write mixed JS/CSS/HTML in a new syntax? (aka JSX)? → Html templates are possible, as already stated, but Vue.js "mixed" single file components are much more cleanly separated than React's JSX approach. I dislike JSX aesthetically but have liked Vue's single file components so far. They have good integration with vim which helps.
  2. Does that mean that people coming from Object oriented languages will need to learn a whole new paradigm – reactive, pure-functional programming? → I would add to the already good answer that being very conscious of function scope is useful. For example, arrow functions can't be used in some situations in Vue.js because their context is the parent context and that can be undesirable when creating a view model and expecting the context to be that view model.
  3. How does DOM access work? Does this mean no jQuery? → Honestly, you will love this after a short while. You can of course access the DOM if you wish, and Vue.js has a nice rich component lifecycle that lets you hook into different phases of rendering, where you have access to the DOM. But it is really relaxing to separate DOM access from data management, it's the main reason MVVM is popular.
  4. Don't such libraries need much tooling? → answer given is good. Knockout definitely depends on a lot more tooling than Vue, and I think the Vue creators aimed at directly addressing this problem. There are well thought-out plugins for routing and validation. Though I will say the validation plugins had some bugs, so that's something to consider: some of the ecosystem is still catching up to Vue 2.0.

@Milimetric: That is great to hear that you also consider such a lib! Particularly, I am happy that you can mainly confirm (and add to) my assumptions about programming, testing etc. (I do mainly UX and some JS and I got into the topic since UI changes are imho to hard to make with our current approach)

Notes from the frontend-standardization group meeting (22.Feb '17) , where we discussed the issue:

Notes by Jan Dittrich

TECHNICAL OPTIONS

  • Seemingly consensus for using a Single-Shared-State like Redux.

WAYS TO GO:

  • No Single Shared State, self-build MVC: Is what Visual editor does. Only briefly mentioned.
  • Single-Shared-State + OOUI: was tried, results seem to be mixed. It works, but it seems to have been a bit tedious since it requires "manual" checking what should be updated and what not (that seems to be the reason why vue/react have virtual DOM diffing!). Here is a small codepen: https://codepen.io/j4n/pen/dNKeKm
  • Single-Shared-State + Vue: Vue is said to be easier to understand and to code than react (subjective)
  • Single-Shared-State + …React-like: There are other, more react-like libs. The advantage over vue is that there is a ecosystem of things that work like react, thus reducing the dependency to one lib.

So, the React-y way that would make sense would be:

  • Using a single shared state like Redux (or a API compatible)
  • Using a react like lib
  • Using HTML-like templates which compile to the pretty standard JSX

Thus, we would use 3 layers, each being fairly clear cut. All having an established API and each being swappable in case our solution comes out of favor.

Or going with vue which has less of an ecosystem of vue/vuex compatible libraries but is well liked for being clean and dev and designer friendly – if if it gains momentum in the Mediwiki-ecosystem it would be balanced out by this (Still, vue/react are fairly similar, so swapping them would need rewrites, but the concepts would stay)

  • Single-Shared-State + OOUI: was tried, results seem to be mixed. It works, but it seems to have been a bit tedious since it requires "manual" checking what should be updated and what not (that seems to be the reason why vue/react have virtual DOM diffing!). Here is a small codepen: http://codepen.io/j4n/pen/

Codepen link returning 404. Could you please update?

@Jan_Dittrich: I read your notes and I think your choice between React ecosystem and Vue ecosystem is a good one, no matter which way you go. I prefer Vue but that's totally subjective, so I would recommend doing a small project in each to see how your team feels.

I'd like to add that if you like Vue's reactive programming and want more of that in the React ecosystem, you should check out Mobx, which is the reactive version of Redux. I think there are objective reasons that reactive programming is superior, as I mentioned before. And in terms of switching between Vue and React, if you use Mobx it's that much easier.

It would be really interesting in seeing Wikidata built with Vue.js - if you were to take that approach would you build something on top of RESTBase or would you build on top of MediaWiki's stack (and replicate functionality where needed?) I'm quite interested in approaches with the former.. particularly given the rise of progressive web apps and interest in many of the applications that @Magnus has built. In case you were not aware I replicated the entirety of MobileFrontend in a React app and it wasn't too difficult and in many ways it was liberating - https://trending.wmflabs.org/.

One of the common things I hear about us not adopting frameworks is that we'll make the wrong decision and migrating is hard, but the good thing about common frameworks is that usually someone has already experienced migrating from one to another and documented their experience (e.g. Vue.js to React.js).

Is this something you are looking to resolve immediately, or could this be something to collaborate and experiment on in the May hackathon (I'd be up for exploring such a thing)?

@Jdlrobson:

I'm quite interested in approaches with the former...

We have not taken any decisions yet. Thanks for helping us by sharing your experiences and giving the trending. example.

…experienced migrating from one to another and documented their experience (e.g. Vue.js to React.js).

That is useful and reassuring!

Is this something you are looking to resolve immediately, or could this be something to collaborate and experiment on in the May hackathon

I am pretty sure that we won’t have is set until the hackathon, so I think we can happily hack on that in May :-)

Made a vue.js-based demo browser for Wikidata:
https://tools.wmflabs.org/reasonator/vuejs_demo/#/item/Q42
This uses the live MW/wikibase CSS, and replicates the document structure in quite some detail.
~300 lines HTML, ~350 lines JavaScript, both reasonably formatted. Not too shabby.
No editing yet ;-)
Code repo is here:
https://bitbucket.org/magnusmanske/reasonator/src/dcddab64e01524ae6a30894c12483e4de867fc57/public_html/vuejs_demo/?at=master

@Magnus That is awesome work ! I'd love to hear an overview of your impressions

A point that I'm still wondering about, and that I have not seen addressed in this ticket nor meeting notes yet.. Server side rendering. I'm not saying we need full non-JS support for everything, but we definitely need fallbacks in some places, and we shouldn't be writing the same HTML twice. So how would we go about delivering say a recent changes list, and then progressively enhance that with Vue, to be come a pageable/scrollable dynamic list ?

Magnus, that's great, also interested in your thoughts. SSR on vue: https://vuejs.org/v2/guide/ssr.html. Seems like some basic support for vue ssr is possible with php's V8 engine, not sure if we're considering that or just de-coupling and running on node (I share Jon's thoughts on this being a solid approach).

but we definitely need fallbacks in some places, and we shouldn't be writing the same HTML twice

There is server side rendering for vue and react, so I assume we can use the same HTML in both places. I have no extended experience with Server Side Editing, though.

My React prototype has server side rendering for anonymous users (view-source:https://nomad.wmflabs.org/wiki/San%20Francisco) - logged in users just use JS, so this is not a problem and there would be no need to write HTML twice as @Jan_Dittrich points out. @TheDJ practically speaking this could be achieved by setting up a REST endpoint that returns HTML e.g. https://en.wikipedia.org/api/rest_v1/page/html/San_Francisco - only that HTML would include script/style tags (possibly to ResourceLoader urls or static urls). For JS rendering you could either use the same endpoint or a json version e.g. https://en.wikipedia.org/api/rest_v1/page/json/San_Francisco

@Magnus that is indeed awesome and demonstrates how easy this can be. Add the Wikidata shell around that and you won't know the difference about what you're using - but I imagine it can be a lot snappier. I'm looking forward to perusing the code. The benefit of using JS only is it lends itself better to service workers - so offline experiences and offline editing are real possibilities.

Really want to spend some solid time hacking on this kind of experience!

It does indeed look like vuejs can do server-side rendering, at least for viewing items (editing would require JS, I presume). We should probably check how much non-JS editing is done on Wikidata at the moment; I'd assume it's not a lot...
I have not worked with vuejs SSR; one reason I became interested in vuejs is precisely that it doesn't require server-side setup.
I have begun adding edit functions. String editing only, and it can't save yet, but one can see the structures in the <template>s and code slowly taking shape...

Some notes on this endeavour (all IMHO):

  • vuejs learning curve is not steep, much less than react
  • vuejs speed is excellent
  • Separation of HTML (including simple display logic) and complex component JS code makes the components much more maintainable and reusable
  • Right now, I have one file for all HTML and templates, and one for all the JS. "Proper" server setup would allow to split those into one ".vue" file per component, containing the respective <template> and JS. That would massively improve readability, dependencies, compartmentalisation etc. A service such as webpack can be used to join all these files into a single one, similar to the MW resource loader
  • I have not tried to use vuejs i18n components. I used my own translation JS library in the new mix'n'match, and that integrates quite well
  • vuejs works best/fastest as a single-page application, which clashes with the MW reload-each-page approach. vuejs still works with that, though

Great points, Magnus. As for .vue files and webpack setup, there is vue-cli which generates some common configuration according to configurable templates. So for example, if you want Vue with webpack, hot reload, .vue transpiling, etc. you can do:

$ npm install -g vue-cli
$ vue init webpack my-project

More info here: https://github.com/vuejs-templates/webpack. I found this handy when just starting because I didn't want to slow down to learn both vue and webpack at the same time.

In our team, we currently discussed the server side rendering and its open questions. The most as-in-the-book way is doing the server side rendering with the node libraries that are usually used (like nuxt.js or so). This would mean:

  • One would need to run a node server for rendering. We currently don't have the infrastructure as far as I am concerned
  • We also don't seem to have a caching infrastructure for node rendered pages (?)

(concerns as told by @daniel)

Anyone with input on these issues? Is that an open question for you, too or do you circumvent the problems somehow?

In our team, we currently discussed the server side rendering and its open questions. The most as-in-the-book way is doing the server side rendering with the node libraries that are usually used (like nuxt.js or so). This would mean:

  • One would need to run a node server for rendering. We currently don't have the infrastructure as far as I am concerned
  • We also don't seem to have a caching infrastructure for node rendered pages (?)

(concerns as told by @daniel)

Anyone with input on these issues? Is that an open question for you, too or do you circumvent the problems somehow?

AFAIK we do have infrastructure for running node servers with proper caching, as part of the rest services layer.

If you create endpoints that return structured data you could consume those from the PHP backend or the JS client.

@GWicke should have more information about the viability of these.

AFAIK we do have infrastructure for running node servers with proper caching, as part of the rest services layer.
If you create endpoints that return structured data you could consume those from the PHP backend or the JS client.

We have node based rendering infrastructure, but as far as I know, we cannot use them to directly serve page views. Because a) all the skin/chrome stuff is done in PHP and b) we currently only use node when editing, and don't have the server capacity to handle several orders of magnitude more hits that would be needed for serving page views.

So, we can use node.js to render stuff, but we still need to loop it through PHP to add chrome. We also need caching for this on the same level as the parser cache. That could be done on the HTTP level between PHP and node. Does this exist?

I still think that this kind of thing is viable right now only for an editing interface. For full page views, this needs more work, see T111588: RFC: API-driven web front-end.

FYI There's some interesting discussion going on here - https://news.ycombinator.com/item?id=14057148 - about React vs Vue.js vs other frameworks.

AFAIK we do have infrastructure for running node servers with proper caching, as part of the rest services layer.
If you create endpoints that return structured data you could consume those from the PHP backend or the JS client.

We have node based rendering infrastructure, but as far as I know, we cannot use them to directly serve page views. Because a) all the skin/chrome stuff is done in PHP and b) we currently only use node when editing, and don't have the server capacity to handle several orders of magnitude more hits that would be needed for serving page views.

We aren't using node for UI rendering at scale, but at the API level the REST API is seeing 15 minute averages of more than 6k requests per second most days. This is comparable to the action API: https://grafana-admin.wikimedia.org/dashboard/db/api-summary?orgId=1&from=now-30d&to=now

The vast majority of this traffic is driven by read views (math, page previews, apps) and data retrieval. Cache hit rate is already ~95%, and rising with traffic levels.

So, we can use node.js to render stuff, but we still need to loop it through PHP to add chrome.

This depends on the roll-out strategy. The standard chrome elements are actually not *that* hard to replace. Long tail special-case interfaces rendered in the content area like special pages etc will take longer, or might not make sense to re-do at all. See https://phabricator.wikimedia.org/T114596. In any case, unless we actually want to add the Chrome in PHP, I don't see a reason to proxy requests through PHP.

We also need caching for this on the same level as the parser cache. That could be done on the HTTP level between PHP and node. Does this exist?

We touched on this in the mail thread. Most API responses and server-side UI renders for anonymous users would be cached in the CDN / Varnish.

Hello! Just letting you know that you can schedule this session now for the hackathon. Instructions and scheduling can be done here:
https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2017/Program#How_to_schedule_breakout_sessions

I've proposed we have session 2 at 13:00 Saturday and put it on the calendar.
For anyone interested I propose we meet Friday (today) in Café Wien @ 11.30am to have an initial conversation. Does that work for everyone?
Update: We will meet at 2pm in Heuriger

Lydia_Pintscher moved this task from incoming to ready to go on the Wikidata board.

Recently we've decided on trying out to implement Vue template rendering in PHP and here what we have so far: https://github.com/wmde/php-vuejs-templating

Currently, solution supports very limited set of features, and in the long run won't be 100% compatible as soon as Vue uses JS eval of template values, but it seems like it is not that hard to implement most commonly used "features".

You're welcome to have a look and express your opinion!

Thanks! @Milimetric, do you know who might be interested in PHP based server side rendering of (vue) templates at the WMF?

@Jan_Dittrich: It seems like the mobile folks are leaning towards preact, but it's worth pinging them. I don't know of anyone else and we stay away from both mediawiki and php for the most part. We do have some extension work coming up soon though, and it's great to see that this option is available. Thank you very much to WMDE for blazing the trail!

@Milimetric, thanks for the update! I tried to get some info on the current decisions around vue/preact on T167041, particularly because I am unsure how sustainable preact is, particularly the staying-compatible-to-react part (but no answers yet)

Yes, I agreed with you when you raised this issue at the hackathon. It's riskier to depend on a smaller library. It might not be as big of a deal here because preact and react are compatible, so if the performance on mobile is a big deal, may be worth considering.

@Jan_Dittrich yeah I don’t think any one is looking at doing server side rendering in PHP. Most of the work is being looked at using JS.

You can talk to @Jhernandez to verify

@Fjalapeno: Good to know. I think @daniel had some concerns about rendering in JS – I sadly can't tell what they are because I am not deep into the infrastructural concerns here.

I wonder – did any of the teams/people involved in this ticket make any further decisions in the last months or had success in refactoring (core?) code to enable previously difficult things in relation to MVVM Frameworks and Server Side Rendering in PHP and/or Node?

Nothing has happened in the last weeks, all tickets related to this are going to get a once over and cleanup in the coming week so we can look at where we are and move forward.

So, an update to this ticket. WikibaseLexeme is deployed to production using a handful of vuejs templates & widgets with PHP based server side rendering using https://github.com/wmde/php-vuejs-templating

We are now looking to move forward with deciding if PHP SSR using something in V8JS would make sense or if we should instead go for a node service (nuxt).

There are various ticket open around our efforts, some new, some old, and they can currently be found at Wikidata-Frontend

@Fjalapeno: Good to know. I think @daniel had some concerns about rendering in JS – I sadly can't tell what they are because I am not deep into the infrastructural concerns here.

@daniel do you have a list of concerns anywhere?

@daniel do you have a list of concerns anywhere?

I think that is what T167134 is about.

@daniel do you have a list of concerns anywhere?

I think that is what T167134 is about.

T167134 (Blockers for using the vue.js framework for Wikibase on the Wikimedia cluster) has almost no content/comments itself. It has some sub tickets, most of them closed. The only that are still open are

  • "Define browser compatibility goals for Wikibase and wikidata.org" (https://phabricator.wikimedia.org/T167137), which I do not see as tied to vue.js (it is a problem that exists independent of vue and it already open now)
  • "Evaluate OOUI integration with vue.js" (T167145) which opens a possibility, but is a possible way to solve some problems, but has no definition of done.

So as far as I am concerned, we have currently no written-down blockers.

@daniel do you have a list of concerns anywhere?

I think that is what T167134 is about.

So as far as I am concerned, we have currently no written-down blockers.

I seem to remember @daniel had some concerns around the topics of i18n, l10n and resourceloader, but I can't find them documented anywhere.

My team (reading web) as part of T195473 is planning to use webpack which I'd strongly advise as a precursor to using Vue.js with a complicated UI. We still plan to ship CSS and i18n via ResourceLoader, but are likely to eventually move away from RL for client side templates. Let me know if you would you like to chat with us about that sometime.

I seem to remember @daniel had some concerns around the topics of i18n, l10n and resourceloader, but I can't find them documented anywhere.

Those seem solvable. But maybe it's time to write down requirements. Just to make sure we can measure the maturity of any new work and compare it with current state and desired state.

I seem to remember @daniel had some concerns around the topics of i18n, l10n and resourceloader, but I can't find them documented anywhere.

I breain dumped here: https://www.mediawiki.org/wiki/User:Daniel_Kinzler_(WMDE)/I18n-JS

Basically, if the messages only use a subset of wikitext than can be (or has been) implemented in pure JS, all is well. As soon as we need to call back to PHP for full wikitext processing, we run into scalability problems.