Status Update (January 30, 2018)

Highlights

  • Deployed Revscoring 2.0. Each scoring model includes statistics that can be used to query and choose an appropriate threshold depending on the use case.
  • Rewrote ORES extension, improving code quality and test coverage. Failures will cause graceful degradation rather than breaking pages that rely on ORES.
  • GCI happened and some work has been done on wikilabels.
  • The ORES labs cluster has been migrated to Debian Stretch, and we're ready to migrate production clusters.
  • "draft topic" model is trained and it works. Support for the model in ORES is ongoing.
  • New languages, new campaigns, new models. We've deployed advanced edit quality models to Simple English, Spanish, and Swedish Wikipedia, Spanish Wikibooks, and basic edit quality to Icelandic Wikipedia and Spanish Wikiquote. Preliminary edit quality campaigns are finished for Hungarian and Serbian Wikipedia.
  • JADE (auditing system) work is continuing, we have a database schema designed, some code written for the backend service, and have planned an event-based architecture plus content-handled Jade and Jade_talk namespaces within MediaWiki.
  • Draftquality data is cached in the ORES extension and is made available to other extensions.

🤔

T166045: Scoring platform team FY18 Q1
T178428: Respond to press inquiry re. algorithms & bots in Wikipedia
T182823: Talk to reporter from OZY

Draft topic

T172321: Build mid-level WikiProject category training set
T179311: Generate mid-level WikiProject categories
T181166: Revscoring: Statistic for multilabel classification
T183580: class weights support for multilabel classification

JADE

T170954: Set up working group for JADE
T174685: Create list of ORES collaborators (focus on language asset helpers)
T175192: Design JADE scoring schema
T178101: Post about judgments/endorsements/preference
T178102: Post about suppression for JADE comments
T178103: Post about public analytics for JADE
T179298: Post about splitting "revision" and "edit" conceptually
T181098: Implement basic path structure for JADE (judgements)
T183598: Post about using MCR for JADE
Potential event-passing design, https://www.mediawiki.org/wiki/File:JADE_event_schema.svg

Infrastructure

T159105: ORES services should have vagrant roles
T168672: Grant AWight admin access to ORES pypi repos
T173251: Have CI merge research/ores/wheels changes
T174660: [Discuss] Moving from nosetests to pytest
T175651: Grafana has confusing or wrong scale for "scores errored" graph
T176914: Wire statistics into test model included with our Vagrant role.
T179862: Keep statistics about ores service hits for storing thresholds
T181067: Parallelize scap for ORES
T181544: Investigate scb1001 and scb1002 available memory graphs in Grafana
T184077: ORES MediaWiki-Vagrant roles should be ported to Stretch

Wikilabels

T139957: Add "info" URL to campaign data so that we can link to campaign page
T155116: Develop a backup strategy for campaigns/tasks/labels
T155440: Add notice to on-wiki labeling pages (e.g. en:WP:Labels) about deprecation.
T171768: Allow Wiki Labels API to list inactive campaigns
T175724: Oauth login does not return the user to the previous page
T175726: Error messages should not contain relative paths or error codes
T176331: Deploy edit quality models for eswikibooks
T178004: Add list of labelers to campaign stats (sort by labels submitted)
T179015: Introduce and create pytest for flask application of the wikilabels AI service
T179296: Remove usages of nosetest and replace it with pytest
T183068: Wikilabel interface in Hungarian has uninformative action buttons, due to translations not updating
T183196: Deploy wikilabels mid-December 2017

ORES

Migrating to stretch:

T182799: Make sure ORES is compatible with stretch
T184296: Convert CloudVPS instances to stretch.
T184765: Back up ores-misc-01 to ores-staging-01
T184766: Convert ores-misc-01 to stretch

Migrating to revscoring 2.0

T175180: Deploy ORES (revscoring 2.0)
T179712: ORES 500s when model_info lookup fails due to a key error
T179838: Update ORES deploy wheels with revscoring 2.0.9
T179296: Remove usages of nosetest and replace it with pytest

T163786: Make ORES documentation translatable
T174402: Review and fix file handle management in worker and celery processes
T175627: UK dictionary broken in production
T178668: /scores/<context> stopped working
T179064: ORES internal server error for edit with many added links
T179098: Deployment to canary causes an import error on docopt
T179509: id string in ORES does not include "features"
T179629: Add link to FAQ to ORES homepage
T179711: ORES 500 errors on a threshold lookup request
T179837: Deploy ORES early Nov 2017
T180115: [regression] ORES filters are not available on French Wikipedia anymore
T180496: Clean up ORES wheels Makefile
T181103: Announce the ORES FAQ
T181183: Improvements to ORES deployment documentation and process
T181187: ORES beta cluster config should be as close to production as possible
T182614: Investigate why ORES logs are being written to syslog despite explicit logging config. Fix.
T184276: Beta Cluster ORES celery worker dies
T184282: Move beta cluster ORES to its own machine
T185148: Update docs, monitoring, etc. for new labs servers

Editquality (Vandalism detection)

T167968: Complete edit quality campaign for Hungarian Wikipedia
T174558: Deploy damaging/goodfaith model for svwiki
T176134: Train & test damaging/goodfaith model for eswiki
T176332: Train/test edit quality models for eswikibooks
T177762: Edit quality campaign for es.wikiquote
T178108: Edit quality campaign for Serbian Wikipedia
T180686: Wikidata beta edit filters are showing every edit in watchlist as damaging
T181099: Train/test reverted model for Icelandic
T181848: Experiment with using English Wikipedia models on Simple English
T181849: Edit quality campaign for simple.wikipedia.org

Revscoring

New languages

T178524: Add language support for Icelandic
T182612: Implement language support for Catalan

T175180: Deploy ORES (revscoring 2.0)
T175627: UK dictionary broken in production
T177544: Revscoring 2.0 takes up too much memory
T177636: Reduce label_thresholds granularity
T179507: Compare coverage reports of migrating to pytest

ORES extension

Extension rewrite

T177421: Cached thresholds should be invalidated for new model versions.
T178792: ORESFetchScoreJob: RuntimeException No model available for [goodfaith]
T181334: Split Cache.php to different services
T181892: Rewrite Stats.php
T182111: Cached thresholds should be purged when model version is incremented
T182942: Tests should have covered regression in T182936
T183468: Deprecate CheckModelVersions and integrate it with the extension workflow
T183762: oresscores is not working at all
T184127: Add models when initializing the table
T184140: Increase coverage of ORES extension
T184142: Refactor Scoring.php
T184554: Deprecate Extension:ORES "beta" mode
T184775: How do I test my extension's maintenance scripts?

T154175: Clean up failure ratio monitoring and set up an alarm when it goes more than a certain threshold
T175053: Make RCFilters compatible with both the old and new thresholds APIs
T176183: Store draftquality data in ores extension
T176588: Query action on API returns the rvcontinue value that point to itself causes infinite loop
T179107: ORES service erroring, in a way that throws exceptions in Extension:ORES
T179430: ORES extension failing to parse scoring response
T179596: Enable draftquality model in ORES extension for enwiki
T179602: Rewind revscoring 1/2 compatibility hacks
T179830: Notice: Undefined property: stdClass::$ores_damaging_threshold in /srv/mediawiki/php-1.31.0-wmf.6/extensions/ORES/includes/Hooks.php on line 602
T179861: Collect all data for draftquality model in enwiki
T179862: Keep statistics about ores service hits for storing thresholds
T180026: Drop oresc_rev_predicted_model index
T180045: Review and deploy schema change on dropping oresc_rev_predicted_model index
T180450: ORES thresholds for Wikidata is too strict
T180633: ORES RC filters missing in beta cluster, fetching thresholds fails
T181006: Watchlist and RecentChanges failure due to ORES on frwiki and ruwiki
T181010: [Spike] Write reports about why Ext:ORES is helping cause server 500s and write tasks to fix
T181168: Replicate RC/WL failures in Beta
T181191: Make ORES-consuming pages more robust to ORES errors
T183862: Recent Changes is broken on Dutch Wikipedia Beta on Beta Cluster

Written by awight on Jan 30 2018, 7:24 PM.
User

Thanks for the heads-up. Keep up the good work.