Page MenuHomePhabricator

How can we test all the wiki/page combinations that can be affected by ORES?
Open, Needs TriagePublic

Description

This seems like a natural candidate for automated browser testing, but there are a lot of combinations. I don't know of any platform currently available at WMF that would let us test experimental code on *every* wiki.

We can affect:

  • Page save (job is enqueued, so this is usually safe to end-users).
  • Special:RecentChanges
  • Special:Contributions
  • Special:Watchlist
  • Special:Preferences

We're currently deployed to wikis listed in $wmgUseORES.

We plan to extend support to these wikis:
https://www.mediawiki.org/wiki/ORES#Support_table

Event Timeline

awight created this task.Dec 1 2017, 6:30 PM

I think we really need someone to do QA for this. Could we collab with @jmatazzoni for some QA time on beta during our deployments?

Hi guys. This sounds valuable but I'm not sure I understand the project or the request fully. Can you say a little more, please, about what you have in mind to do and what kind of help you need?

Halfak added a comment.Dec 6 2017, 7:41 PM

Essentially, it's possible that changes to ORES can affect RC Filters. We'd like Collab's help reviewing the filters functionality when we make deployments (our usual cadence is once every 3 weeks or so). This would involve checking on the functionality in Beta and reporting back if anything was broken.

mmodell added a subscriber: mmodell.Dec 6 2017, 8:07 PM

This seems like it would be at last partially testable with selenium running against beta cluster.

awight removed awight as the assignee of this task.Dec 11 2017, 3:51 PM
awight updated the task description. (Show Details)Dec 11 2017, 3:53 PM
  • We could do browser testing on a schedule or on a commit (in ORES service or Ext.)
  • We could have a QA specialist look at stuff in a browser for deployments
  • We could add a simple check of RC/WL/etc. for ORES Service Deployments and ORES Ext deployments.
Halfak added a subscriber: Catrope.

We talked to @jmatazzoni and @Catrope. They agreed that Collaboration should look into implementing these types of browser tests. So we (Scoring-platform-team) will keep monitoring. Note that manual checks are currently part of our deploy pattern.

awight removed a subscriber: awight.Mar 21 2019, 4:02 PM