https://www.mediawiki.org/wiki/MediaWiki_Developer_Summit_2015#Schedule
Tuesday 27, 9:00am
The existing ways of releasing user-facing features have proved to have severe limitations during the Media Viewer release. As someone who went through that storm, I feel like many of the organizational issues we encountered haven't been addressed yet. This is a critical issue, as years of hard work can potentially get a very hostile response when the release date comes, and unprepared engineers can find themselves with a big crisis to handle with many hotfixes to produce in a short period of time.
While some areas have been improved since, such as the introduction of the design research team as a way to validate changes, much still has to be discussed. Here are some ideas of topics to talk about:
- How could the community be better involved in the design, development and launch of user-facing features?
- What is the role of WMF engineering when it comes to introducing new features?
- What process should be followed when things go wrong (unpopular release)?
- Gradual release: small wikis first and large wikis last doesn't work. Something better is needed, what should it be?
- Does the regular deployment train process worsen crisis situations? If so, how could it be improved?
- How can success be measured for a feature deployment? A/B testing? Microsurveys?
- Should the user-facing feature release process be standardized, or does it need to be tailored for each feature?
//We will use the Media Viewer release as a concrete example for the discussion. Here's a summary of what happened and the conclusions drawn from that experience://
**What we had planned:**
- Good unit testing and browser testing coverage
- Beta testing
- In-feature surveys to measure user reaction to launch
- Regular community outreach (IRC, mailing list, wiki, "community champions", etc.) during the beta and launch period
- Releasing to small wikis first, large wikis last
**What went well:**
- We received extremely useful feedback that was acted on from the beta testing and community outreach.
- Every wiki except commons, dewiki and enwiki seemed to like the feature as it was at launch time
- Barely any regression when we had to fix things in a hurry thanks to test coverage
**What went wrong:**
- Scope creep. We ended up removing a ton of features that were fully built
- Commons, dewiki and enwiki are entirely different from smaller wikis. User acceptance on smaller wikis is irrelevant when dealing with those larger ones.
- Community outreach had unsifficient attendance and the people who were disgruntled at launch time probably missed those earlier outreach efforts.
- Metrics work was hurried due to pressure to launch on set date as well as trying to shove as many features into the produt as possible at the last minute. As a result success metrics were almost entirely missing
- Surveys were inadequately worded and were used for quantitative decision-making (for lack of success metrics), when they should have been qualitative only
- Things not going well made all layers of the organization intervene, which was very distracting for the team and introduced conflicting signals
- When we got around to doing user testing research, the product performed inadequately
- The entire team was drawn into the community backlash
**What could be improved:**
- Gradual release that can be rolled back easily
- Better user acceptance measurement
- Better safeguards against scope creep. Engineers saying no wasn't enough
- Metrics work is an area that engineering had to fight for getting done. The Product group should be as interested in this part of the project as the engineers are
- No success metric = no go, regardless of set launch date
- No user research = no go, regardless of set launch date
- Don't set fixed launch dates. Targeting a quarter is realistic, targeting a specific date makes us cut corners that we can't always fix after the launch (eg. key metrics not tracked before and after)
- Better fencing engineers from community backlash when it happens. Holding that many all-hands-on-deck meetings was counter-productive.
- More effective community engagement. More influential community members should be invited into the early stages