Page MenuHomePhabricator

Developing community norms for vital bots and tools
Closed, ResolvedPublic

Tokens
"Love" token, awarded by Qgil."Like" token, awarded by cscott."Like" token, awarded by Tgr."Like" token, awarded by JJMC89."Like" token, awarded by Krenair."Baby Tequila" token, awarded by fgiunchedi."Hungry Hippo" token, awarded by Capt_Swing."Love" token, awarded by MusikAnimal.
Assigned To
Authored By
bd808, Oct 27 2016

Description

Type of activity: Pre-scheduled session
Main topic: Building on Wikimedia services: APIs and Developer Resources

The problem

Bots and tools are a vital resource for many on-wiki content creation and curation activities. A typical bot/tool project begins life as a way for a motivated Wikimedia community member to make some on-wiki task easier (or possible). These individuals are "scratching their own itch" in the best tradition of open source development. Many of these projects have a short lifecycle due to factors such as loss of interest by the maintainer, insurmountable technical hurdles, or discovery of a better means to manage the original problem. Others however become popular and tightly integrated in the workflows of one or more on-wiki communities.

Popular tools and bots become de facto production software needed to keep the wikis healthy and happy. Their roots as weekend projects from motivated volunteers brought them their success, but ultimately pose a risk to their end users. Life happens and a single developer project is in perpetual danger of abandonment. Adopting basic FLOSS project practices and following some general rules of professional software and systems management can help protect the software and the wikis.

Expected outcome

  • Discuss potential "best practices" to promote to bot and tool maintainters directly.
  • Discuss features/services that Tool Labs could provide to make following the recommended practices easier.
  • Discuss how to reach bot and tool maintainers on and off wiki
  • Discuss how to promote best practices to on-wiki groups that are affected when tools and bots die due to lack of active maintainership

Current status of the discussion

Gave a talk at WikiConf NA 2016. Published presentation artifacts on Wikitech.

Links

Event Timeline

bd808 created this task.Oct 27 2016, 3:13 PM
Restricted Application added a project: User-bd808. · View Herald TranscriptOct 27 2016, 3:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Anomie added a subscriber: Anomie.Nov 15 2016, 5:10 PM
MusikAnimal added a subscriber: MusikAnimal.
Siznax added a subscriber: Siznax.Nov 15 2016, 7:49 PM
Restricted Application added a project: Cloud-Services. · View Herald TranscriptNov 15 2016, 9:32 PM
bd808 updated the task description. (Show Details)Nov 15 2016, 9:45 PM
Qgil moved this task from To triage to Team radar on the Developer-Advocacy board.
fgiunchedi added a subscriber: fgiunchedi.

Related research: When the Levee Breaks: Without Bots, What Happens to Wikipedia’s Quality Control Processes?

I maintain several bots and tools and I agree that we need to have a documented plan in place here. There are a lot of indispensable tools out there (not mine!) that keep our wikis humming along, and woe betide us if we lose them. We stand to lose a hell of a lot of critical functionality and efficiency if these tools go down. But the critical roles these tools fill are complex, diverse, and often invisible, meaning we have no good way of knowing what the scale of downstream impacts if a key tool goes offline.

bd808 moved this task from Triage to In Progress on the Cloud-Services board.Nov 24 2016, 9:42 PM
bd808 added a subscriber: Qgil.Nov 29 2016, 3:53 AM

@Qgil I'm guessing that my 4 tokens + 1 comment is a pretty weak argument for a main room topic at this stage?

Don't forget 7 subscribers! :)

If unsure, we can keep it "on track" for now. @bd808 what is important is that you start prioritizing proposals in your area, just in (the likely) case that there are not enough pre-scheduled slots for all of them.

bd808 added a subscriber: chasemp.Dec 8 2016, 8:49 PM

In a full 90 minute slot, @chasemp and I would expand the scope of this to cover:

  • Planning for the Tool Labs volunteer committee to enforce the Right to fork and Abandoned Tool takeover policies
  • New services/features that could be build to help that committee succeed
  • Discussion of ways to contact the existing Tool Labs maintainers in addition to the labs-l/labs-announce channels
bd808 moved this task from Backlog to Ready on the Community-Tech-Tool-Labs board.Dec 15 2016, 5:47 PM
Tgr awarded a token.Dec 23 2016, 1:33 AM
Tarrow added a subscriber: Tarrow.Jan 4 2017, 10:54 AM
cscott awarded a token.Jan 5 2017, 9:57 PM
cscott added a subscriber: cscott.Jan 5 2017, 10:03 PM

At the editing team off-site we also raised the idea of improving our frameworks so that gadgets/bots/extensions/labs/production are more similar, so that it is easier to take community-developed code and "productize" it when it becomes important to maintain... and vice-versa, to allow community members to adopt WMF tools which have dropped off the "vital to keep running" list.

Right now the entry point for many community developers is a client-side javascript gadget, which then would have to be rewritten from scratch in python to turn it into a pywikibot, and then rewritten again as a PHP extension (or standalone node.js service?) if it was going to going to be moved to production. It ought to be possibly to grease some wheels here to make it easier for community code to "be born", "grow up" in labs, spend some time being maintained by WMF in production, and then eventually "get old" and be retired out to labs and then community maintenance again.

bd808 added a comment.Jan 5 2017, 11:00 PM

Right now the entry point for many community developers is a client-side javascript gadget, which then would have to be rewritten from scratch in python to turn it into a pywikibot, and then rewritten again as a PHP extension (or standalone node.js service?) if it was going to going to be moved to production. It ought to be possibly to grease some wheels here to make it easier for community code to "be born", "grow up" in labs, spend some time being maintained by WMF in production, and then eventually "get old" and be retired out to labs and then community maintenance again.

That's a grand vision to think about. I'd love to hear more about a concrete path towards it.

There is a really big difference between the size of problems that are being solved by various tools, gadgets, bots, and scripts hosted in Tool Labs. Some have long term maintainers and others are the result of a weekend or hackathon blitz. Practically none of them are built to withstand the massive scale of anon wiki traffic, and honestly that seems fair to me since MediaWiki itself only survives it with a complex mesh of caches that have been implemented over the years. If deploying a tool was as hard as getting code into MediaWiki's core repo or operations/puppet.git we wouldn't have many tools. Many of the major obstacles to WMF production deployment are things like security review wait times, packaging guidelines, configuration management, and deployment tooling challenges which are mostly orthogonal to the idea of software framework improvements.

I am very interested in helping define and promote practices that make keeping a useful tool running easier. Today my ideas mostly lean towards promoting core FLOSS software practices like licensing and version control as baby steps. The next tier after that includes explaining the utility of separating config and code, and of using and contributing to shared libraries/frameworks rather than starting over from scratch for each project.

Right now the entry point for many community developers is a client-side javascript gadget, which then would have to be rewritten from scratch in python to turn it into a pywikibot, and then rewritten again as a PHP extension (or standalone node.js service?) if it was going to going to be moved to production. It ought to be possibly to grease some wheels here

Does that idea make sense? A client-side JS gadget is probably UI to make an interactive editing task easier, which wouldn't make much sense as a bot in Python or a nodejs service and as a MediaWiki extension would probably just be some boilerplate to just load the same JS. A bot running in Python might or might not be doing something that makes any sense to do server-side as a PHP MediaWiki extension, but if it does make sense it's likely the way it goes about it should be completely different: the pywikibot is likely a long-running process making many API calls, while a PHP MediaWiki extension would have trouble being long-running (use the job queue instead?) and doing things via the API instead of shared backend logic is a bad code smell. Even an extension and a service run and access data in different ways.

Server-side code, whether PHP or nodejs, also has rather different security concerns compared to a browser-based JS gadget or a pywikibot bot: the gadget has relatively few concerns, the pywikibot's concern is mainly in keeping its password private, while the PHP or nodejs doesn't have to care about a password but does need to avoid CSRF and (for the PHP at least) SQL injection.

Moving things between WMF and community "ownership" is worthy of thought, but we should probably keep the consideration along the lines of moving the apples from one place to another without turning them into goats at the same time. For example, JS gadget into extension with minimal PHP boilerplate to load the same JS, or service running on Labs infrastructure to service running on production infrastructure, or changing the "maintainer" field on an extension from WMF to community.

bd808 moved this task from To Do to In Dev/Progress on the User-bd808 board.Jan 7 2017, 12:21 AM

To the owner of this session: Here is the link to the session guidelines page: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Session_Guidelines. We encourage you to recruit Note-taker(s) 2(min) and 3(max), Remote Moderator, and Advocate (optional) on the spot before the beginning of your session. Instructions about each role player's task are outlined in the guidelines. The physical version of the role cards will be made available in all the session rooms. Good luck prepping, see you at the summit! :)

Note-taker(s) of this session: Follow the instructions here: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Session_Guidelines#NOTE-TAKER.28S.29 After the session, DO NOT FORGET to copy the relevant notes and summary into a new wiki page following the template here: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Your_Session and also link this from the All Session Notes page: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/All_Session_Notes. The EtherPad links are also now linked from the Schedule page (https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Schedule) for you!

bd808 moved this task from In Dev/Progress to Done on the User-bd808 board.Jan 17 2017, 4:16 PM

@bd808 are there any other remaining action items on this task? If not, I would close this as resolved!

bd808 added a comment.Feb 10 2017, 5:36 PM

@bd808 are there any other remaining action items on this task? If not, I would close this as resolved!

I've 'been meaning to' put that slides and video up on Commons and leaving this open in the hope that I will eventually remember to do so.

scfc triaged this task as Low priority.Feb 16 2017, 9:34 PM
scfc moved this task from Triage to Backlog on the Toolforge board.
Qgil awarded a token.Nov 13 2017, 10:17 AM