Page MenuHomePhabricator

Outreach campaign to raise awareness of Scoring Platform
Open, MediumPublic

Description

Operation People-Understand-ORES

  • What are the things everyone in the project should know?
    • ORES is a backend service that does useful stuff for editors with algorithms
    • Killer use-cases are counter-vandalism and measuring article quality
    • ORES works in 58 wikis and is expanding by 2-3 wikis per quarter
    • The Scoring-platform-team (SPT) is very understaffed with no dedicated product support
      • The SPT doesn't build any end-user facing tools -- just the backend that lets them use predictions.

Project details

  • What do you want to achieve (goals)?
    • ORES usage go up across wikis -- people understand what it is for and are able to use it
    • Feedback loop to hear from communities. Opportunities to improve / something is not working.
  • Who is your target audience (be specific)?
    • Patrollers, admins, and community organizers (WikiProject Organizers, Editathon organizers)
  • How do you want your audience to feel?
    • Informed and Understood
  • What medium do you have in mind and how do you want to distribute it?
    • Phabricator
      • PROS: Easier to streamline tasks
      • CONS: only in English
    • Mediawiki talk page:
      • PROS: more accessible
      • CONS: needs manual conversion to task on Phab.
    • ORES page on WP
  • Do you already have an idea of what you need to roll out your project? If so, please check all that apply (i.e. [x]):
    • Blog
    • Page on wiki
    • Social media
    • Email or MassMessage
    • Presentation
    • Video
    • Infographic
    • Diagram
    • Art piece
    • Other: _______
    • I don’t know
  • How are you measuring your success (key performance indicators)?
    • Usage of ORES-powered tools (e.g. new filters for recent changes, huggle, RTRC, etc.)
    • Improvements in model fitness (e.g. accuracy, precision, recall, ROC AUC, PR AUC)
  • What organizational goal is this linked to? (please link to current Annual Plan details)
    • Knowledge Integrity, Machine Learning Infrastructure
  • When is this due?
    • Planning: FY20 Q1
    • Production: n/a
    • Launch: FY20 Q2
  • Who is leading this project on your team?
    • Aaron Halfaker - Principal Research Scientist - ahalfaker@wikimedia.org

Comms plan


What is the problem?

The Scoring Platform team works with Wikimedia projects to deploy machine learning models on their projects in their language. These initiative often succeed at engaging the communities at all stages, from initial investigation to initial deployment to even quality control. This is not only good practice, it is required: without participation of the communities, we do not have the language assets needed to deploy models.

ORES currently supports ~50 wikis. https://tools.wmflabs.org/ores-support-checklist/ We are interested in supporting more wikis over time, as well as having conversations with our communities about quality control.

In some wikis (e.g. Hungarian Wikipedia) ORES remains unknown and underutilized by patrollers and other editors who would benefit. Also, we never hear from many communities about how their models are working (or not working). We'd like to improve our ability to communicate and be communicated to. We'd also like to figure out a better process for sharing responsibility with product teams.


Task checklist

  • Aaron: Get data on Wikis that are under-utilizing ORES (CC nate)
  • María: Ask Gergo to describe history of ORES and huwiki -- Specifically focusing on awareness. (Task link: ….)
  • Everyone: re-convene after Wikimania

Event Timeline

Harej created this task.Feb 27 2019, 8:52 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 27 2019, 8:52 AM
Elitre added a subscriber: Elitre.Mar 4 2019, 3:42 PM

Thanks for filing, triaging.

Elitre claimed this task.Mar 4 2019, 3:49 PM

(And to make sure I don't forget about re-assigning this, I temporarily assign it to myself to make sure it doesn't get lost anywhere.)

Johan claimed this task.Mar 4 2019, 4:14 PM
Restricted Application added a project: User-Johan. · View Herald TranscriptMar 4 2019, 4:14 PM
Johan added a comment.EditedMar 4 2019, 5:24 PM

I'll take a look at this. @Harej, I've scheduled a meeting on Monday next week.

Johan moved this task from Backlog to Do soon on the User-Johan board.Mar 4 2019, 5:24 PM
Johan moved this task from Do soon to Do now on the User-Johan board.Mar 21 2019, 1:56 PM

Met with @Harej. We talked about what we could do generally and he should come back with a more specific request.

Elitre triaged this task as Medium priority.Mar 26 2019, 3:20 PM
Johan added a comment.Apr 1 2019, 9:35 PM

@Harej No hurry from our side, take whatever time that makes sense for you, but I just wanted to check in and make sure we hadn't missed any communication from your side?

Harej added a comment.Apr 1 2019, 9:56 PM

You have not missed anything. I will update this task when there is a more precisely scoped project.

I talked to @Harej about what he had in mind for this work. It reminded me of the work I was doing with @Keegan to describe product communication practices around ORES year (or two?) ago. Maybe some of that would would be valuable for getting started here.

Johan moved this task from Do now to Do later on the User-Johan board.May 7 2019, 12:30 PM
Harej removed a subscriber: Harej.Jul 4 2019, 9:26 AM
Elitre added a subscriber: Harej.Jul 12 2019, 11:44 AM

You have not missed anything. I will update this task when there is a more precisely scoped project.

Hey! Since it's been a while, unless you have news for us, maybe this can be temporarily closed and reopened when the need arises.

Hey folks. I'm not sure what progress @Harej had made on this, but I'm taking over now.

Generally, we struggle with making sure that communities who receive ORES deployments know and understand that the relevant tools are available. Our own analysis (ping @Groceryheist) suggests that many different communities had no idea ORES was available. This suggests that there some breakdown in our communication strategy. I'd like to consult with someone from CR to identify and test some best practices for communicating before and after these software deployments.

Elitre added a comment.EditedJul 15 2019, 1:32 PM

Thanks, @Halfak. Can you reassess the task description and update it? (As the new FY has started this may need to be reevaluated and likely reassigned.) In particular we'll need details around your deadline. Thank you!

Elitre reassigned this task from Johan to Keegan.Jul 17 2019, 4:08 PM
Elitre added a subscriber: Johan.

\o/ @Groceryheist is back from travels on July 19th. I'll set up a kickoff meeting after that. So, probably early next week.

\o/ @Groceryheist is back from travels on July 19th. I'll set up a kickoff meeting after that. So, probably early next week.

@Halfak we'll be working with @mcruzWMF on this, so please make sure to invite her to the kickoff meeting.

OK sounds good.

Halfak updated the task description. (Show Details)

I just applied the template that @mcruzWMF linked me to the description of the task. Let me know if you have any questions.

mcruzWMF updated the task description. (Show Details)

It just occurred to me that this work might align with @Rfarrand's work on setting up smaller, geographically distributed technical events. I wonder if there's a good way that we might both support those events and build our relationships with a variety of technical contributors who might be interested in ORES.

It just occurred to me that this work might align with @Rfarrand's work on setting up smaller, geographically distributed technical events. I wonder if there's a good way that we might both support those events and build our relationships with a variety of technical contributors who might be interested in ORES.

This is an interesting idea, @Halfak. Can you please provide more information, so we can understand your reasoning better? These are the specific questions I have:

  • How are technical contributors involved in deploying ORES on a Wikimedia project? (connected to Goal: more implementations of ORES)
  • Can technical contributors benefit from ORES? How? (connected to Target audience, Motivation)
  • What would you like technical contributors to do when they learn about ORES? I see in the task description that one of the KPI you set is More ORES-powered tools. Is there something else that you are thinking about? (connected to Call to action)

I like this idea a lot. After we get more clarity on this, we can regroup about what incorporating this audience group means for this outreach campaign. One possible outcome is that we hold 2 different campaigns (because the target audience, motivation, and call to action are essentially different). Looking forward to your input!

How are technical contributors involved in deploying ORES on a Wikimedia project? (connected to Goal: more implementations of ORES)

Technical contributors use ORES in the bots, gadgets, and tools they build. See https://www.mediawiki.org/wiki/ORES/Applications and note that most uses of ORES are in volunteer developed tools. It's usually technical contributors (or contributors with technical skills) who we end up working with first in a project. This might be an artifact of how information travels. But I think it's also because the local technologists often perform a dual role of building tech and shepherding other tech deployments.

Can technical contributors benefit from ORES? How? (connected to Target audience, Motivation)

I think this was mostly answered in my post above. ORES primary users are developers and secondarily the end-users of the tools they build. It turns out some dev-users are product teams at the Foundation while others are technical volunteers. ORES is usually a hit at hackathons because it gives people new opportunities to UIs for important work in Wikipedia.

What would you like technical contributors to do when they learn about ORES? I see in the task description that one of the KPI you set is More ORES-powered tools. Is there something else that you are thinking about? (connected to Call to action)

I would like technical contributors to do a few things:

  • Creatively imagine useful scenarios for quality measurement, vandalism detection, topic routing, etc. and either build new tools or incorporate ORES predictions into the tools they have already.
  • Discuss the potential of ORES with their peers. Help people understand what predictions are good for and where they can't replace human judgment.
  • Give us expert feedback. We want feedback from any ORES end-user, but technical contributors are more able to write more clear bug reports and feature requests. E.g. T227153: On en.wikipedia, ref tags inserted by the shortened footnote template, {{sfn}}, are not counted in ORES features
mcruzWMF renamed this task from Community Relations Specialist support for Scoring Platform to Outreach campaign to raise awareness of Scoring Platform.Jul 30 2019, 6:23 PM

We certainly are building out a program for pre hackathon smaller technical events. Probably to pilot with the 2020 hackthon (Albania) or 2020 wikimania (Thailand).
If any ideas and pre-existing connections for small in person events arise around this effort in either of those regions around those time frames I would LOVE to talk more in depth about specifics.

Hey @Halfak , thanks for the details on how and why to target technical contributors. I think that entails a different outreach campaign because the target audience and the call to action (or expected engagement) are different.

I met with @Tgr today and I was able to flesh out the distribution plan for the original outreach campaign (targetting patrollers). Please take a look and let me know what you think: https://docs.google.com/document/d/1l5XsCm-kOZ5V135n52UeGQWH3Mh8yEXyKM4lVY8hG-Y/edit#heading=h.5p3qgix00thr

I commented on the linked task - while there may be points of contact etc., I wouldn't want the task at hand to lose its focus on a specific team's needs. CommRel may not be able to commit resources to grand plans at this time.

Thank you @mcruzWMF! I missed this in the storm of emails around Wikimania. I'll be reviewing now.

I just left a couple of notes. The plan generally looks good to me. My biggest question is a budget one.

@Halfak I am surprised again to hear that bugetary topics are an issue. They are frequently raised as a reason for not getting things done or not accomplished in a timely manner, but the WMF is currently very over funded by a large surplus.

On another note: As an end user of ORES (but definitely not IT or AI savvy), I would be happy to comment on how ORES is functioning for example at NPP and AfC on en.Wiki. We volunteers are not sufficiently privileged to participate at Wikimania conferences when it matters, but I will certainly be available in Bangkok - I live here.