### What is the problem?
The Scoring Platform team works with Wikimedia projects to deploy machine learning models on their projects in their language. These initiative often succeed at engaging the communities at all stages, from initial investigation to initial deployment to even quality control. This is not only good practice, it is required: without participation of the communities, we do not have the language assets needed to deploy models.**Operation People-Understand-ORES**
ORES currently supports ~50 wikis. https://tools.wmflabs.org/ores-support-checklist/ We are interested in supporting more wikis over time, as well as having conversations with our communities about quality control, especially considering we are currently working on the Jade extension as the audit mechanism for ORES- What are the things everyone in the project should know?
-- ORES is a backend service that does useful stuff for editors with algorithms
-- Killer use-cases are counter-vandalism and measuring article quality
-- ORES works in 58 wikis and is expanding by 2-3 wikis per quarter
-- The #scoring-platform-team (SPT) is very understaffed with no dedicated product support
--- The SPT doesn't build any end-user facing tools -- just the backend that lets them use predictions.
### How can we help you?
I am interested in having a community relations specialist as a consultant to help us plan our outreach to communities around ORES and Jade. For the next few months this will mostly be occasional consulting around strategy and help us check our assumptions around community practices. Once we backfill our senior engineer role we will probably begin making plans for deployment, which will likely mean a more active role for the specialist. Our target Wikipedias are Arabic, Catalan, Persian, and Finnish (contingent on them approving this, that is).**Project details**
Also, I think in general a specialist could make sure our community communication practice is up to code, especially since it is so important to what we do.- What do you want to achieve (goals)?
-- ORES usage go up across wikis -- people understand what it is for and are able to use it
-- Feedback loop to hear from communities. Opportunities to improve / something is not working.
- Who is your target audience (be specific)?
-- Patrollers, admins, and community organizers (WikiProject Organizers, Editathon organizers)
- How do you want your audience to feel?
-- Informed and Understood
- What medium do you have in mind and how do you want to distribute it?
-- Phabricator
--- PROS: Easier to streamline tasks
--- CONS: only in English
-- Mediawiki talk page:
--- PROS: more accessible
--- CONS: needs manual conversion to task on Phab.
-- ORES page on WP
--- PROS: Very close to target audience
--- CONS: not all WPs that deploy this tool have one. E.g.: https://it.wikipedia.org/wiki/Progetto:Patrolling/ORES
- Do you already have an idea of what you need to roll out your project? If so, please check all that apply (i.e. [x]):
-- [] Blog
-- [x] Page on wiki
-- [] Social media
-- [x] Email or MassMessage
-- [] Presentation
-- [] Video
-- [] Infographic
-- [] Diagram
-- [] Art piece
-- [] Other: _______
-- [x] I don’t know
- How are you measuring your success (key performance indicators)?
-- Usage of ORES-powered tools (e.g. new filters for recent changes, huggle, RTRC, etc.)
-- Improvements in model fitness (e.g. accuracy, precision, recall, ROC AUC, PR AUC)
- What organizational goal is this linked to? (please link to current Annual Plan details)
-- Knowledge Integrity, Machine Learning Infrastructure
- When is this due?
-- Planning: FY20 Q1
-- Production: n/a
-- Launch: FY20 Q2
- Please provide additional context & direction:
-- Previous attempts:
--- https://www.mediawiki.org/wiki/ORES/Get_support
--- https://tools.wmflabs.org/ores-support-checklist/
--- {T182054} (Failed to start)
--- {T164331} (Failed to get clear buy-in from Growth)
-- Relevant visual material:
--- One presentation on good ORES feedback: https://www.youtube.com/watch?v=rsFmqYxtt9w#t=30m00s
-- Existing ideas:
---
- Who is leading this project on your team?
-- Aaron Halfaker - Principal Research Scientist - ahalfaker@wikimedia.org
### What does success look like?-----
1. In the short term, we will have some strategy for engaging the target wikis. Ideally they will all approve of Jade or at least have actionable feedback for us.
2. In the longer term, Jade is deployed to some pilot wikis. It doesn't have to be the original four target wikis or even four wikis necessarily.### What is the problem?
3The Scoring Platform team works with Wikimedia projects to deploy machine learning models on their projects in their language. These initiative often succeed at engaging the communities at all stages, from initial investigation to initial deployment to even quality control. We have more efficient team procedures around reaching out to communitiesThis is not only good practice, workingit is required: with them to deploy modelsout participation of the communities, and working with themwe do not have the language assets needed to audit themdeploy models.
### What is your deadline?ORES currently supports ~50 wikis. https://tools.wmflabs.org/ores-support-checklist/ We are interested in supporting more wikis over time, as well as having conversations with our communities about quality control.
We would like an initial meeting sometime in the next two weeks.In some wikis (e.g. Hungarian Wikipedia) ORES remains unknown and underutilized by patrollers and other editors who would benefit. Also, We are also interested in pursuing more dedicated specialist resources in the next fiscal year.we never hear from many communities about how their models are working (or not working). We'd like to improve our ability to communicate and be communicated to. We'd also like to figure out a better process for sharing responsibility with product teams.