@calbon do you have a preference on which models to try this out with?

I was thinking maybe the en.damaging editquality model and maybe the en.articlequality model, but am open to any other suggestions.

Let's try whichever one is the easiest to start out with. That'll give us a good idea of the lowest hanging fruit are.

The model cards paper recommends having the following sections:

• Model Details
  • Who built the model
  • Date developed
  • Model version
  • Model type/class
  • Link to paper/citation details
  • License
  • Contact info
• Intended Use
  • primary use-cases
  • primary users
• Factors
  • demographics
  • environmental conditions
  • technical attributes
• Metrics
  • fitness statistics / performance measures
  • decision thresholds (for acceptance/deployment/etc.)
  • variation approaches
• Evaluation Data
  • datasets
  • motivation
  • preprocessing
• Training Data
  • Same as evaluation.
  • If we can't share for some reason (PII etc.) then try to explain or describe things like distribution of some of the factors
• Quantitative Analysis
  • intersectional results
  • unitary results
• Ethical Considerations
• Caveats and Recommendations

Most of this data for ORES is scattered around github, mediawiki and wikitech, so making a model card article for an editquality model should be pretty straightforward

Let's start by manually creating a draft model card at https://www.mediawiki.org/wiki/Machine_Learning/Test_Model_Card
We can use the en.damaging editquality model, we can just start with gathering the Model Details section first.

I talked with @Isaac the other week and he showed me: https://en.wikipedia.org/wiki/User:HostBot/Social_media_traffic_report

This is a page that gets auto-updated by a bot as new information rolls in, so there is some precedence for automating articles on mediawiki.

It seems like we should be able to easily publish model cards in a similar manner, however the first step is figuring out where/how to source all the data in a repeatable manner.

This is a page that gets auto-updated by a bot as new information rolls in, so there is some precedence for automating articles on mediawiki.

I can try to help if more is needed but quick overview of how this works:

• We run a script on the servers that generates that data that we want to publish on a regular cadence
• When the data is prepared, we use some simple Python functions to format that data into wikitext (in our case, a large table with some explanatory text): code
• We then use the APIs to write that wikitext to the specified page. It makes the edit under my account using an owner-only OAuth token (see documentation)

Oh awesome thanks Isaac. Where do you host the Python code?

Where do you host the Python code?

This is a prototype so we're just running this off stat1004 using this code. There are two cronjobs: the first to run the job to generate the data and then a second to publish to the wikis. The separation is because the first accesses HDFS and so needs to kinit (which can be done automatically but the script has to be run from a more generic account then). The second must be run under my account though because it has to be able to access the file with the credentials for publishing to the wikis as my account is stored under a file that is read-only for me. Hacky but it has worked for almost a year! I suspect there are better solutions though but heads up that the permissions issue might arise for you too (and you might consider pursuing the creation of a more generic MLPlatformBot_(WMF) account to ensure better continuity and somewhat reduce the permissions issue). If you're curious, the crontab looks like this: slides.

office hours video recording on model cards

• Model title
• Model description
• Model created date
• Model version

Description

• Intented uses
• Intented users
• Out-of-scope*

Model Timeline

• explanation of major changes
• metrics
• [what do you call all the versions of a model]

Training data

• License

Interpretability Guide

Contact Us*

Difficulty tag to reproduce (as a barrier to entry)

I experimented with the model-card-toolkit module a couple weeks ago to see how well it would integrate with mediawiki.

Here are my notes:
I used model-card-toolkit to generate a model card for the enwiki-goodfaith revcoring model.
Script: https://phabricator.wikimedia.org/P16815
The script generates an html output that looks something like this:

When I upload the html to mediawiki, I get an article like this: https://www.mediawiki.org/wiki/User:ACraze_(WMF)/test/modelcard
There's definitely some formatting errors; we would need to create a jinja template that handles some of the formatting for links/p tag details and converts it to mw syntax if we decided to use the toolkit module.

One major thing I noticed, is that we will need to keep track of model metadata somewhere.A couple weeks ago @calbon mentioned the idea of possibly storing model metadata on wikidata, which could be pretty interesting. I captured the idea down in T286508: Investigate storing model metadata on Wikidata

Hi all! I just put together a memo synthesizing what dataset/model/service documentation could look like on WMF resources. For generalizability, I've called the documentation "algorithmic accountability sheets". The memo addresses what questions we should be asking for each component of an algorithm, as well as some thoughts on governance, metrics, deprecation, etc.

It should be publicly accessible for everyone to view and comment on. Let me know what you think either on the document or in this thread!

This is great thanks @Htriedman. One issue I wanted to discuss is what happens when a human editor changes something that was written programmatically by the model card "bot". Does the human editor's change get overwritten the next time the model card creator code is run?

Also, do the model cards go on Mediawiki or Wikitech?

@calbon good first question — yes, the model card bot should overwrite manual edits. In my mind, the canonical way to edit model cards should be by pushing some change to a config file for the card hosted on Gerrit/Gitlab. Then, changes will show up on the next run of the card generator.

To your second question: I don't really know where the cards should live! Would love to hear some pros/cons for each option from someone who has a better sense of the communities on each wiki.