Page MenuHomePhabricator

How to measure "Wiki projects requesting beta features and other early deployments"
Closed, DeclinedPublic

Description

In our annual plan we define a metric about "Wiki projects requesting beta features and other early deployments".

We need to discuss how to implement this, sooner than later, and decide what to do next.

Event Timeline

It's late; please help me understand the scope better:

Wiki projects requesting beta features and other early deployments

What does "requesting beta features" mean? Making a beta feature not beta anymore but a default feature on a certain wiki after a consultation with that wiki community? Or making previously entirely unavailable features available under Special:Preferences#mw-prefsection-betafeatures on certain wikis for opt-in?

This tracking would cover deployments under Wikimedia-Site-requests / Wikimedia-Extension-setup but only deployments that are 'helped with' by Community-Relations-Support and hence would not cover changes like https://phabricator.wikimedia.org/T177989 or https://phabricator.wikimedia.org/T173639#3676752 ?

https://www.mediawiki.org/wiki/Technical_Collaboration_Guidance/Community_decisions#Deployment_on_their_wikis says:

When a feature is being released to the wikis through publicly documented deployment waves, communities can influence the best timing for the deployment of a product or major feature in their wikis:

  • Communities excited about a new feature or product can request to be included in the first waves, after showing proof of interest in their community discussion channel.

The idea is: in terms of community engagement, having several communities interested in becoming early adopters of new features is good; having none is bad.

One problem is that Wikimedia Foundation teams don't have a common process for early deployments, and even CLs don't have a common process to look for early adopters. Sometimes projects are chosen based on data or connections without making a broad call for everybody, sometimes such broad calls are made.

The problem is in fact that we haven't discussed whether a common process is appropriate, desirable. Maybe we should start there.

Another factor to consider is that, while "some early adopters" is clearly better than "no early adopters", more is not necessarily better. Teams tend to prefer 2-5 projects, and having 25 projects willing to be early adopters would not be good news necessarily. Still, from a community engagement perspective, having 25 projects interested would be good news indeed, regardless of whether they are included in the early waves or not.

We need to solve these questions about concepts before attempting to actually count early adopters.

(One thing that I just realised (feel free to ignore if you think it's not relevant) is that ATM the early stages of rollout may be invisible to the communities if we rollout one wiki per time (because it isn't an item relevant enough for Tech News). On one hand this is good because it protects them from unwanted attention, such as cross-wiki feature haters wanting to influence feature reception. OTOH we can certainly come up with a mechanism for highlighting how "brave" those early adopters were, what they and us learned, etc.)

In a TCG inspired process, hate/blockers should be identified in the planning / prototyping phase, and no big surprises should be feared at deployment time. I'd rather go for a systematic and predictable process, where an invitation to join a beta release is announced to all wiki projects. If the team only wants a limited numbers of betatesters or has preferences like "RTL languages", "non-Latin scripts", "at least 100 very active contributors"... they could say so in the invitation itself.

Clearly, we can only measure the requests we are aware of. Something we want to consider is whether not having any involvement at all in such requests still counts, and I suspect it does.
For the "how" part, I guess we could start tracking how many agree after the specific related recommendation/request that we make to tech ambassadors and on Tech News when a product team is ready for it.

The intention is to measure communities volunteering to be early adopters in relation to a specific deployment plan. These scenarios would be very different:

  • Audiences team launches a call for early adopters, no project volunteers and in fact 12 request explicitly to be in latter waves.
  • The same, but 12 projects want to be among the early adopters and no project asks to have their deployment postponed to later waves.

However, for this to happen we would need to have a more consistent approach to such announcements. Perhaps I am idealizing the situation and we will never get there? Or maybe we don't even need to get there if in such situations a couple of projects suffice as early adopters?

Aklapper: I think it has to mean making beta features available in mw-prefsection-beta first. Few wikis are comfortable having features rolled out without extensive testing at the wiki in question. This has caused problems before. Three examples I can think of are: date auto-formatting (which also caused auto-linking – this was a huge debate at en.Wikipedia, eventually leading to an ArbCom case); and deployment of Flow as feature-complete and read-to-use, which some wikiprojects at the same site then implemented for their talk pages, which led to similar controversy; and similar issues with Visual Editor. Making VE opt-in at the user level has been key to getting some communities to accept anything about it, and to work with the devs in ironing out problems with it instead of just objecting to it being used at all (new users at a site like en.WP generally don't know it's available, so it is mostly clueful editors who are testing it and finding out what its limitations and flaws are).

Qgil: WMF's "communities can ..." language isn't really very helpful in the context, because they're not hive minds. The reality of the situation is that various sub-communities at each wiki have different levels of interest in and potential acceptance of any new feature addition, removal, or change. Even if an RfC is held about something, it may only represent a sharply limited subset of editors. Just relying on Tech News to attract interest (more on this below) is also self-selecting and self-limiting to a certain micro-community of "nerds". Secondly, I'm skeptical that sheer number of early-adopter projects is very important; what seem more relevant are "size" (activity level) of each project, and their diversity. E.g., it's more important to get input from de.Wikipedia than from et.Wikipedia, and of more value to have input from a major Wikipedia, from Commons, from WikiData, and from a major Wiktionary, than from four Wikipedias. You get more stress-testing out of bigger projects, while from more diverse projects you get broader use-case data (including feedback on problems that were not predicted).

Elitre: Notwithstanding my caveat above about RfCs, such polling processes are probably the best indication that a project has sufficient critical mass of interested and non-hostile editors for a beta feature to be made available. It may just be a matter of making sure such discussions are opened (in a well-written manner), then widely "advertised" on-wiki, and the results accurately reported back. I'm skeptical that expecting editors to come to to meta, Phabricator, or some other site advertised by Tech News will be sufficient. Even if the poll is not localized to a particular wiki, it needs significant per-wiki promotion to get attention drawn to it. Maybe this will be a key role for the Tech Ambassadors and Community Liaisons?

On to some more political and PR matters:

As a new Tech Ambassador, there are some concerns I expect to be trying to mediate (and which I may share to an extent). The dominant one is that much WMF attention and dev time is aimed at "new gizmos" when some core problems in basic functionality have been reported for a decade or longer and remain unresolved (e.g. T6521 is high on the list of editor concern and causing real problems and serious disputes, but apparently arousing very little dev or Foundation interest). The more Phab time is spent on "chrome" and the less on making the basic functionality of the system work better, the less interest and faith the "communities" collectively have in what's being developed and offered to them. Even making the basic look-and-feel of the key sites like Wikipedia, Wiktionary, and Commons [I mean that in a public userbase sense, not a value judgement one] seem more 2017 and less 2004 is probably a higher priority to the average editor and reader than many of the development projects that are ongoing for new beta features. I have many of those features (and Gadgets that used to be Beta features) turned on, but few of them make a crucial difference to either my editing or reading experiences; I seriously doubt I'm unusual in this.

It's also not clear that everyone with "{WMF)" in their username who posts about such things to places like https://en.wikipedia.org/wiki/WP:Village_pump is there to listen rather than just tell when it comes to what WMF is planning. Just the fact that part of the above discussion is about "cross-wiki feature haters" is illustrative of the issue. That kind of characterization isn't helpful, both in that it's dismissive of the concerns raised, and it's a false dichotomy – for every feature that someone hates, they probably love a dozen, and would love to see a dozen more. The editors are not your enemies, they just don't think every idea that every dev or every WMF employee comes up with is automatically golden.

PS: One thing that's unclear even to tech-oriented Wikipedians and such, on average, is exactly how much input and influence the Foundation has over what the devs do. There's a general perception, for example, that WMF – with all its tens of millions of dollars in reserve – can throw some money at a problem or a want and get results. It's not well-understood to what extent this idea is valid, nor how to affect the process. Most of the coding work on MediaWiki and related software appears to be volunteer labor, though WMF has clearly devoted actual money to some projects, like Flow.

At the end we have stopped trying to measure "Wiki projects requesting beta features and other early deployments". Declining.