Page MenuHomePhabricator

[Bug] Special:SetLabel browser tests failing, but unable to run tests locally
Closed, InvalidPublic

Description

The same Wikidata tests are failing for days now: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/282/

Failing Scenarios:
cucumber tests/browser/features/aliases.feature:158 # Scenario: Save duplicated aliases
cucumber tests/browser/features/sitelinks.feature:36 # Scenario: Type site id
cucumber tests/browser/features/sitelinks.feature:183 # Scenario: Add multiple sitelinks
cucumber tests/browser/features/sitelinks.feature:195 # Scenario: Remove multiple sitelinks
cucumber tests/browser/features/sitelinks.feature:228 # Scenario: Add new sitelink to already referenced site
cucumber tests/browser/features/special_set_label.feature:12 # Scenario: Special:SetLabel page has all required elements
cucumber tests/browser/features/special_set_label.feature:31 # Scenario: Add a label
cucumber tests/browser/features/special_set_label.feature:43 # Scenario: Edit an existing label
cucumber tests/browser/features/special_set_label.feature:54 # Scenario: Edit using an invalid language fails
cucumber tests/browser/features/special_set_label.feature:64 # Scenario: Edit using an invalid ID fails

Some are still printing these hilarious "Uh-oh. Could not find link to Sauce Labs job URL."

The Special:SetLabel tests look like they are actually broken, which may be the case because https://gerrit.wikimedia.org/r/233423 actually changed the HTML of this page. But I'm entirely unable to do anything about this. I just wasted hours trying to run any test. I tried to run the test locally the same way I always did, but all I get is an empty Firefox window that does nothing. I followed https://phabricator.wikimedia.org/diffusion/EWBA/browse/master/tests/browser/README.md and https://github.com/wikimedia/mediawiki-selenium step by step but all I get is an empty Firefox window that does nothing.

This is by far not the first time I'm running into such problems. The test framework appears so fragile, brittle and impossible to debug, I find this unacceptable.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 1 2017, 9:30 AM
thiemowmde triaged this task as Unbreak Now! priority.Mar 1 2017, 9:30 AM
thiemowmde moved this task from Proposed to Monitoring on the Wikidata-Former-Sprint-Board board.
Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptMar 1 2017, 9:30 AM

@thiemowmde There were some major changes recently in the way Selenium works with Firefox. Without going into much detail, switching browser to Chrome should just work. Let me know if you need help. I am zeljkof in #wikimedia-releng on Freenode.

Also, I have recently updated Selenium related documentation: https://www.mediawiki.org/wiki/Selenium

all I get is an empty Firefox window that does nothing

Can you provide terminal output? That would help with debugging. Also your OS, Ruby and Firefox versions.

I tried to set export BROWSER=chrome, but it still opens Firefox. Console output is basically nothing:

thma@thma:/var/www/mediawiki/core/extensions/Wikibase/tests/browser$ bundle exec cucumber
# Wikidata UI tests
#
# Author:: Tobias Gritschacher (tobias.gritschacher@wikimedia.de)
# License:: GNU GPL v2+
#
# feature definition for item aliases tests
@chrome @firefox @internet_explorer_10 @internet_explorer_11 @local_config @test.wikidata.org @wikidata.beta.wmflabs.org
Feature: Edit aliases

  Background:                                    # features/aliases.feature:11

Then the empty Firefox window opens and nothing happens.

I did. I changed pretty much every single line in environments.yml, but nothing changed the fact that it runs Firefox instead of Chrome. Yes, I'm sure I edited the correct file because it started complaining when I put garbage in.

It's really hard to debug the problem here. Ping zeljkof at #wikimedia-releng and we can debug via Hangouts or something similar.

I have just updated the instructions on how to run tests from your machine targeting mediawiki-vagrant: https://www.mediawiki.org/wiki/Selenium/Target_MediaWiki-Vagrant_machine

Could you please check if you can run tests for mediawiki/core? If that works, maybe something is not configured properly for Wikibase.

thiemowmde closed this task as Invalid.EditedMar 6 2017, 12:04 PM
thiemowmde removed Tobi_WMDE_SW as the assignee of this task.

After uninstalling and reinstalling the entirety of Ruby, Bundler, as well as all Gems, and after an other hour of going insane (including messages like

Xvfb not found on your system (Headless::Exception)

and

undefined method `echo' for #<MediawikiSelenium::Environment:0xa27bcd0> (NoMethodError)

and

/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- mkmf (LoadError)

(the later one means – hold your breathe – that I have to install ruby1.9.1-dev instead of ruby1.9.1, which does make sooo much sense when I'm not developing Ruby itself) …

… browser tests run again on my machine. I have no idea why. They still don't run headless (see the first message above), but at least.

The fact that the same SetLabel tests are failing for many weeks now does have a simple reason: Neither http://wikidata.beta.wmflabs.org nor http://test.wikidata.org was updated for a long time. These tests fail because they are running on outdated HTML.

Change 341302 had a related patch set uploaded (by thiemowmde):
[mediawiki/extensions/Wikibase] Remove problematic OOUI dependency from browser tests

https://gerrit.wikimedia.org/r/341302

Change 341302 merged by jenkins-bot:
[mediawiki/extensions/Wikibase] Remove problematic OOUI dependency from browser tests

https://gerrit.wikimedia.org/r/341302