Page MenuHomePhabricator
Create Task
Maniphest T221104

Disable failing ruby browser tests in daily browser tests
Closed, DeclinedPublic3 Estimated Story Points
Actions

Description

Tests to disable are ones causing the following failures:

  1. T213454 - RSpec::Expectations::ExpectationNotMetError
  2. T213433 - Watir::Wait::TimeoutError
  3. T213430 - The save has failed. (failed-save) (MediawikiApi::ApiError)

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Comment out more failing Ruby browser testsLucas Werkmeister (WMDE)mediawiki/extensions/Wikibasemaster+275 -270
Comment out lots of failing Ruby browser testsLucas Werkmeister (WMDE)mediawiki/extensions/Wikibasemaster+811 -790
Customize query in gerrit

Related Objects

Event Timeline

• alaa_wmde created this task.Apr 16 2019, 3:12 PM
Restricted Application added a project: Wikidata. · View Herald TranscriptApr 16 2019, 3:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• alaa_wmde renamed this task from Disable failing ruby browser tests to Disable failing ruby browser tests in daily browser tests.Apr 16 2019, 3:14 PM
• alaa_wmde set the point value for this task to 3.
• alaa_wmde moved this task from Ready to estimate to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.
• alaa_wmde edited projects, added: Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed: Wikidata-Campsite.
• alaa_wmde updated the task description. (Show Details)Apr 16 2019, 3:22 PM
• noarave claimed this task.Apr 17 2019, 9:16 AM
Restricted Application added a project: User-Noarave. · View Herald TranscriptApr 17 2019, 9:16 AM
• noarave moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Apr 17 2019, 9:16 AM
• noarave removed • noarave as the assignee of this task.Apr 17 2019, 10:52 AM
• noarave removed a project: User-Noarave.
• noarave moved this task from Doing to To Do (prioritised from top to bottom) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
• noarave subscribed.
Lucas_Werkmeister_WMDE claimed this task.Apr 17 2019, 4:39 PM
Lucas_Werkmeister_WMDE moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Lucas_Werkmeister_WMDE added a comment.Apr 17 2019, 4:42 PM

Recent list of failed tests @ https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/548/testReport/

Lucas_Werkmeister_WMDE added a comment.Apr 17 2019, 4:58 PM

Another list @ https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/547/testReport/ – contains some tests that weren’t failing in the other list and vice versa, so apparently this whole business is flaky as well, great :(

I guess I’ll just be very liberal in commenting out tests.

gerritbot added a comment.Apr 17 2019, 5:45 PM

Change 504622 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] Comment out lots of failing Ruby browser tests

https://gerrit.wikimedia.org/r/504622

gerritbot added a project: Patch-For-Review.Apr 17 2019, 5:45 PM
Lucas_Werkmeister_WMDE moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Apr 17 2019, 5:45 PM
gerritbot added a comment.Apr 18 2019, 3:21 PM

Change 504622 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Comment out lots of failing Ruby browser tests

https://gerrit.wikimedia.org/r/504622

Lucas_Werkmeister_WMDE moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Apr 18 2019, 3:50 PM

Moving to Verification – let’s see what the next daily CI run looks like…

Lucas_Werkmeister_WMDE moved this task from Test (Verification) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.EditedApr 23 2019, 11:56 AM

Okay, current status:

I’ll upload another patch.

gerritbot added a comment.Apr 23 2019, 12:06 PM

Change 505757 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] Comment out more failing Ruby browser tests

https://gerrit.wikimedia.org/r/505757

Lucas_Werkmeister_WMDE moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Apr 23 2019, 12:08 PM

To be honest, I’m starting to doubt the usefulness of this commenting-out spree. There are now more commented-out scenarios than ones left active, and the way that different test failures appear and vanish between test runs makes me think that most of the failures aren’t related to any individual test.

But let’s merge this and see what happens to the daily CI runs then, I suppose.

hoo moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Apr 24 2019, 8:20 PM
hoo subscribed.
In T221104#5130208, @Lucas_Werkmeister_WMDE wrote:

To be honest, I’m starting to doubt the usefulness of this commenting-out spree. There are now more commented-out scenarios than ones left active, and the way that different test failures appear and vanish between test runs makes me think that most of the failures aren’t related to any individual test.

Yikes… but that doesn't make any of the commented out tests more useful either…

gerritbot added a comment.Apr 24 2019, 9:09 PM

Change 505757 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Comment out more failing Ruby browser tests

https://gerrit.wikimedia.org/r/505757

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.3; 2019-04-30).Apr 24 2019, 10:00 PM
Lucas_Werkmeister_WMDE added a comment.EditedApr 26 2019, 10:21 AM

I propose we decline this task. It seems clear to me that it’s not the individual tests that are broken, and disabling more and more of them is just a prolonged game of whack-a-mole until we’ve reached the stable end state of having disabled every last ruby browser test.

We should rewrite the tests in Selenium (making sure that they actually pass) and ignore the ruby ones until we can completely remove them.

• alaa_wmde added a comment.EditedMay 1 2019, 6:05 AM

That's worse than we imagined .. I'm wondering where the flakiness is coming from, just to make sure we do not carry it over with us into the re-written wdio tests. Did you get to figure out a little more why they are flaky already?

I know browser tests can waste great amount of effort in getting them working and useful, and at the end of the tunnel one would wonder whether the benefit justifies the costs.

So let's do this:

  1. timebox this (4 hours?) to figure out where the flakiness coming from. Is it selenium (we use selenium for wdio too)? is it race-conditions between client and server state? or is it just cucumber/capybara?
  2. shift focus entirely to re-writing those tests in javascript using wdio, taking with us any useful learnings from 1.

@Lucas_Werkmeister_WMDE since you've put enough effort into this already, please make the decision on the above as you see fit

Lucas_Werkmeister_WMDE removed Lucas_Werkmeister_WMDE as the assignee of this task.May 2 2019, 11:54 AM
Lucas_Werkmeister_WMDE subscribed.

I didn’t put any time into investigating the failures because as I recall, we decided against that and instead wanted to just disable “the failing tests” (it now turns out there’s no such distinction, any test can apparently fail).

We can still do the timeboxed investigation, but since I don’t think any of the work I made so far will be useful there (it was pretty mechanical work), and I’m no longer on the campsite, I think someone else should do that.

WMDE-leszek subscribed.May 6 2019, 8:23 AM
WMDE-leszek closed this task as Declined.May 9 2019, 11:20 AM
WMDE-leszek moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Per discussion above, I'd call it declined as it does not seem to be sensible thing to do.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits