Page MenuHomePhabricator

[EPIC] IP Masking: StructuredDiscussions (Flow)/LiquidThreads Community discussion
Closed, ResolvedPublic

Description

TL;DR

(When we mention FLow, the moves are also valid for LiquidThreads (LQT))

Background

Flow is a complex piece of software that was never quite finished, fits poorly into the MediaWiki architecture due to its many (eventually un-utilized) layers of abstraction, lacks key features (such as search and solid moderation) and has many disruptive bugs, and is rejected by most Wikimedia communities. While there are communities that are happy with it, the value it provides isn't commensurate with its maintenance cost, much less with the expected future maintenance cost once there are major changes to Parsoid or VisualEditor. DiscussionTools now provides a similarly user-friendly UI, without all the problems. We should invest into undeploying Flow - it will be a significant effort, but keeping it in production would take more effort eventually.

Talk pages were long seen as a usability pain point (see the talk pages consultation's historical overview). The LiquidThreads extension was created back when the movement had very limited technical resources; plans to improve it eventually evolved into replacing it with an ambitious workflow system, Flow (later renamed StructuredDiscussions) which would have different modules for different types of talk page activities such as threaded discussion, voting or requests.

The eventual release only supported threaded discussion, and while it was a huge usability improvement for users unfamiliar with wikitext, it broke a number of workflows that power users considered very important, such as moderation or talk page refactoring. As a result, the rollout of Flow met fierce opposition from many wiki communities, and deployments were halted (and in some cases reversed). The WMF paused development in 2015 and instituted a freeze in new deployments of Flow a few years later. a few communities still use Flow.

In 2019, the WMF organized the talk pages consultation to look at the future of talk pages. The consultation yielded the Talk pages project. That project produced the DiscussionTools extension, which gained wide community adoption and is now the default on all wikis.

Community members have initiated discussions to replace Flow pages with DiscussionTools on https://www.mediawiki.org/ and at translatewiki.net.

Problems

Maintenance burden

Maintaining Flow is a drain on WMF resources:

  • It is a large and complex codebase, with approximately 36,000 lines of code.
  • The code is complicated to reason about and difficult to work with. Some of the original authors are no longer at WMF.
  • As it provides an alternative talk page implementation, all features that need to be interact with talk pages need a separate Flow and wikitext implementation. On the frontend side this is somewhat managed via mw.messagePoster, on the server side there isn't an off-the-shelf way to do it.
  • Subtle bugs relating to database replication issues periodically cause user talk pages to become broken (see e.g. T308907). Someone from Growth then has to run a maintenance script to fix the problem. Fixing the underlying issue is not trivial.
  • Flow accounts for 25 open issues on the production error workboard. There have been 111 Flow production error tasks in total.
  • There are 16 open Flow tasks tagged with Security, including some crippling limitations to moderation functionality. The non-trivial data flow makes it nearly impossible to make sure proper escaping is done before building SQL queries, making it prone to SQL injection.
  • There are ~1200 open tasks on the Flow workboard, with about a hundred of them having high priority.
  • Other teams are blocked on overdue maintenance needed in Flow: Content Transformers needs stored Flow HTML to be updated to the newest Parsoid version (T209120, T124837). Given its shaky change list integration (see below), it will require a lot of extra work for patrolling or moderation changes (a priority area for the next year).
  • Flow uses patterns and libraries not found elsewhere in MediaWiki, increasing maintenance cost. For example, it uses Pimple rather than MediaWikiServices for dependency injection (T150350), and it uses lightncandy for its templates, unlike any other extension.
Architecture burden

Because Flow was intended to be very generic and support all kinds of different workflows, the codebase ended up with many layers of abstractions (which eventually weren't used since only one workflow was implemented), making it hard to understand and maintain. Because it envisioned sharing talk pages between multiple wikis (this mostly got implemented but wasn't deployed), it had to invent its own alternative revision system. In addition, its authors tried to innovate in directions that the wider MediaWiki community didn't pick up on (e.g. using MySQL as a noSQL database), so Flow ended up being something of an alien object wedged into MediaWiki. Not using the revision table means Flow has to reimplement every single feature related to changes lists (page history / contributions / RC / watchlist) and moderation that other extensions get for free.

UX burden

Two competing interfaces for the same use case, where the user cannot chose which one to use and eventually has to learn both (wikis that use Flow don't use it on all talk pages), adds to the mental burden of using the site. While at a high level Flow is in many ways closer to user expectations of how a discussion system should look, the implementation is unfinished, lacks key features (such as searching or legible links), has many severe bugs and usability issues (e.g. T324416) and integrates with moderation tools poorly.

Risk of emergency undeployment

Undeploying Flow would be a significant export and a lengthy project. However, if we don't expend that effort and then find ourselves in a situation where we need to undeploy Flow anyway and can't take several weeks to do it (e.g. a hard-to-fix security issue is found), the effects would be quite catastrophic. All Flow content and all Flow-related log entries and page histories would become inaccessible, Flow-related entries would disappear from user contributions. If Flow gets reenabled later, some of the content would be corrupted or lost (as Flow needs to keep its own database in sync with e.g. page moves).

Flow and LQT usage

We gathered initial data about Flow usage to help guide the community consultation (T345484). Some key insights:

  • French Wikipedia is by far the largest Flow user. It has twice as many Flow posts than Mediawiki.org (the second largest user). French Wikipedia combined with Mediawiki.org has more Flow edits than all of the other wikis together. Further data on Flow usage is available here.

In August 2023, on average:

  • DiscussionTools is used about 18,780 times per day.
  • Flow is used about 250 times per day.
  • LiquidThreads is used less than once per day.

LiquidThreads is enabled on only on 5 Wikimedia wikis. Flow is enabled on 48 wikis. DiscussionTools is enabled everywhere.

List of wiki where the tools are deployed:

See also

Questions:

Acceptance Criteria:

  • Initiate a community conversation with some of the main communities using Flow (see T345484: Data on usage of StructuredDiscussions (Flow))
    • Explain the complexity, problems with Flow, and our need to limit our maintenance burden so we can support other work. See which communities are receptive to moving to DiscussionTools. (We can't yet promise a timeline on this, but we will work with Engineering leadership to find the resources to support this work).
    • Gather feedback on Growth's proposed IP Masking approach to handling Flow boards: Logged out traffic can't post to Flow discussion boards until they have either logged into an account or created a Temporary account (by editing any non-Flow board). We will provide an informative message for logged out traffic that attempts to edit Flow boards, we will provide further details about logging in or creating a temp account.

Details

Due Date
Dec 29 2023, 8:00 AM

Related Objects

StatusSubtypeAssignedTask
Resolvedkostajh
DeclinedNone
In ProgressNiharika
OpenNone
OpenKStoller-WMF
OpenNone
OpenNone
OpenNone
ResolvedTrizek-WMF
ResolvedUrbanecm_WMF
ResolvedTrizek-WMF
ResolvedTrizek-WMF
ResolvedTrizek-WMF
ResolvedSgs
ResolvedSgs
ResolvedEtonkovidova
DeclinedNone
OpenNone
ResolvedUrbanecm_WMF
ResolvedTrizek-WMF
ResolvedTrizek-WMF
ResolvedUrbanecm_WMF

Event Timeline

Gather feedback on Growth's proposed IP Masking approach to handling Flow boards: Logged out traffic can't post to Flow discussion boards until they have either logged into an account or created a Temporary account (by editing any non-Flow board). We will provide an informative message for logged out traffic that attempts to edit Flow boards

Alternatively (if we do not properly fix it at Flow):

  • Introduce a new special page Special:CreateTemporaryAccount, which should at least be usable if blocked IP can edit their own talk page
  • If a unregistered user try to edit Flow board, either (1) direct them to that page so that a temporary account can be created; or (2) invoke such special page automatically before edits are saved (probably once user open the edit interface)
Trizek-WMF changed the task status from Open to Stalled.Sep 12 2023, 3:19 PM

Stalling it until Data is available (T345484).

In T346108#9158863, @Bugreporter wrote:
If a unregistered user try to edit Flow board, either (1) direct them to that page so that a temporary account can be created; or (2) invoke such special page automatically before edits are saved (probably once user open the edit interface)

Yes, I should probably make that more explicit in this task. I think we should definitely include further details about logging in or creating a temp account for any logged out user who attempts to post in Flow.

@Trizek-WMF Would it help to have a quick design mockup of what that would look like when facilitating this discussion?

@Trizek-WMF Would it help to have a quick design mockup of what that would look like when facilitating this discussion?

I think we should have two phases for this conversation.

  1. Explain our intention, and check which communities are receptive to moving to DiscussionTools, and which would like to explore other solutions. We keep the door open to the latter, but we can also be clear and honest about the fact that it will require time and energy.
  2. Provide other solutions if needed, and discuss them.

We might not need step 2, which would reduce the amount of work regarding this topic.

KStoller-WMF changed the task status from Stalled to Open.Sep 13 2023, 5:57 PM

Moving this task back to open, since an initial summary of Flow usage is here.

@Trizek-WMF if you have further questions or need a larger data set for any of the lists, please follow up in the related task: T345484: Data on usage of StructuredDiscussions (Flow)

Adding key insights from T345484: Data on usage of StructuredDiscussions (Flow):

French Wikipedia is by far the largest Flow user. It has twice as many Flow posts than Mediawiki.org (the second largest user). French Wikipedia combined with Mediawiki.org has more Flow edits than all of the other wikis together. Further data on Flow usage is available here.

In August 2023, on average:

DiscussionTools is used about 18,780 times per day.
Flow is used about 250 times per day.
LiquidThreads is used less than once per day.

LiquidThreads is enabled on only on 5 Wikimedia wikis. Flow is enabled on 48 wikis. DiscussionTools is enabled everywhere.

KStoller-WMF changed the task status from Open to In Progress.EditedSep 22 2023, 6:14 PM

Moving to In Progress, as we are drafting a communication plan. The actual community discussion has not yet started.

KStoller-WMF renamed this task from IP Masking: StructuredDiscussions (Flow) Community discussion to [EPIC] IP Masking: StructuredDiscussions (Flow) Community discussion.Oct 11 2023, 8:29 PM
KStoller-WMF moved this task from Inbox to Epics on the Growth-Team board.
Trizek-WMF renamed this task from [EPIC] IP Masking: StructuredDiscussions (Flow) Community discussion to [EPIC] IP Masking: StructuredDiscussions (Flow)/LiquidThreads Community discussion.Oct 26 2023, 3:29 PM
Trizek-WMF updated the task description. (Show Details)

There's an earlier discussion of how to disable Flow in T96301, which includes some discussion of how to dump flow into wikitext.

Trizek-WMF changed the task status from In Progress to Stalled.Feb 19 2024, 1:59 PM

I am waiting for the two last tasks to be finish before resolving this one.