Status update (July 11th, 2017)

Two outages with documentation. Revscoring 2.0 coming with better model information and "thresholds". New support for Romanian, Albanian, Tamil, Greek, and Bengali. We're officially welcoming @awight to the team!

Hey folks!

As of July 1st, we are officially the Scoring Platform team. We're welcoming Adam Wight (@awight) to the team officially.

The last ~month was very productive, but we had two major production issues. See 20170613-ORES and 20170623-ORES. As you will see below, there's a series of tasks that address problems that were related to these issues.

Despite dealing with production issues, we've been able to get a very substantial change to the revscoring library merged. This change will make accessing information about models (build environment, test statistics, scoring thresholds, etc.) much easier. This will cause a breaking change in ORES UI so we'll be making an announcement when we roll it out. Stay tuned.

We've also increased our language and model coverage substantially. We even built and deployed a totally new type of model to help out French Wikisource!

New team stuff

So with the new fiscal year, we're a new team. We're working on an announcement to be posted on the WMF blog. That should be coming out soon. See T169755. Most of the new team stuff focused on getting Adam all of the rights he needed to do ORES deploys and other work.

ORES downtime

We had two major downtime events with ORES. One of these (20170613-ORES) was not our fault, but we still set up better monitoring (T167830) so that, when it happens again, we can fix it more quickly. The second event (20170623-ORES) was due to a deeply problematic regular expression pattern that had ~ a 1 in a billion chance of causing catastrophic failure. We both fixed the regular expression (T168888) and fixed the timeout that didn't catch the out-of-control regex match (T168965)

New language support

We were lucky to have a lot of volunteers working with us this month so that allowed us to make a lot of progress towards expanding support to more wikis. Both the Albanian and Romanian Wikipedias finished their labeling campaigns so we'll be able to deploy advanced support to them soon (T163010, T156517). We now have some of the basic language assets for Tamil so we should be able to build up basic support for that Wikipedia soon (T166052). We also implemented an article quality model for Turkish Wikipedia (T164671) thanks to lots of work by @Mavrikant. We developed a new strategy for cross-language badword/informal detection and addressed some lingustic overlap between English and Hungarian Wikipedia (T167231, T165872). We implemented a page-level OCR model for French Wikisource (somewhat like article quality, but more about the quality of machine reader transcriptions) (T167196). Finally, we deployed the ORES Review Tool to French Wikipedia (T165044)

Data release -- Monthly Article Quality predictions (English Wikipedia)

This was a long time coming. We've got the data that allowed us to measure the coverage gap of articles about Women Scientists in Wikipedia hosted in labs (T146718). That means the table can be queried directly from Quarry. See this demo query.

New features for ORES/revscoring

Prompted by concerns raised by @Catrope from the Collaboration-Team-Triage, we have been working on a better way to represent information about a model (T162217): build environment, statistics, prediction thresholds, etc. We've even built a way to allow for querying the thresholds of a model that we refer to as "threshold optimizations". This refactoring gave us an opportunity to address some other outstanding wants with regards to revscoring -- e.g. storing more information about the build environment (T160223) and cleaning up our "tune" utility (T163711).

Wikilabels UX improvement & maintenance.

Thanks to @Jan_Dittrich and @Pginer-WMF's feedback, we've been working on addressing some user-experience issues. These were mostly fixes to language to make the functionality of the system more clear (T167079, T138736). We also brought Wikilabels down for a short period of time on Tuesday July 11th for scheduled database maintenance (T169933).

ORES Review Tool improvements

We finished up some patchsets that were blocked for a long time on some fixes to core MediaWiki. This allowed us to fixed highlighting in Special:RecentChanges and Special:Watchlist (T155903, T155930).

General ORES maintenance

We've done a bunch of maintenance to ORES to solve a variety of issues that cropped up. E.g. improving tests (T168007), solving a regression in the basic ORES ui (T149117), fixing our new precaching system (T168674) and enabling it to work with the new EventStreams feed (T166046).

Misc operations work, versions and styling

Written by Halfak on Jul 12 2017, 10:44 PM.
Principal Research Scientist
Zppix, Pginer-WMF, Catrope and 3 others
"Love" token, awarded by bkowshik."Love" token, awarded by Zppix.