The job doesn't seem to be executing tests. Errors out with:

17:09:01 Use bundle show [gemname] to see where a bundled gem is installed.
17:09:01 + vendor/bin/bundle exec cucumber --color --tags @integration --tags '~@skip' --format pretty --format junit --out /mnt/jenkins-workspace/workspace/mwext-mw-selenium@2/log/junit
17:09:05 cannot load such file -- mediawiki_selenium/support (LoadError)
17:09:05 /mnt/jenkins-workspace/workspace/mwext-mw-selenium@2/src/extensions/RelatedArticles/tests/browser/features/support/env.rb:3:in `require'
17:09:05 /mnt/jenkins-workspace/workspace/mwext-mw-selenium@2/src/extensions/RelatedArticles/tests/browser/features/support/env.rb:3:in `<top (required)>'
17:09:05 /mnt/home/jenkins-deploy/.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/rb_support/rb_language.rb:95:in `load'
17:09:05 /mnt/home/jenkins-deploy/.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/rb_support/rb_language.rb:95:in `load_code_file'
17:09:05 /mnt/home/jenkins-deploy/.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/runtime/support_code.rb:180:in `load_file'
17:09:05 /mnt/home/jenkins-deploy/.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/runtime/support_code.rb:83:in `block in load_files!'
17:09:05 /mnt/home/jenkins-deploy/.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/runtime/support_code.rb:82:in `each'
17:09:05 /mnt/home/jenkins-deploy/.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/runtime/support_code.rb:82:in `load_files!'
17:09:05 /mnt/home/jenkins-deploy/.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/runtime.rb:184:in `load_step_definitions'
17:09:05 /mnt/home/jenkins-deploy/.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/runtime.rb:42:in `run!'
17:09:05 /mnt/home/jenkins-deploy/.gem/2.0.0/gems/cucumber-1.3.20/lib/cucumber/cli/main.rb:47:in `execute!'
17:09:05 /mnt/home/jenkins-deploy/.gem/2.0.0/gems/cucumber-1.3.20/bin/cucumber:13:in `<top (required)>'
17:09:05 /mnt/home/jenkins-deploy/.gem/2.0.0/bin/cucumber:23:in `load'
17:09:05 /mnt/home/jenkins-deploy/.gem/2.0.0/bin/cucumber:23:in `<main>'
17:09:05 Build step 'Execute shell' marked build as failure
17:09:05 [PostBuildScript] - Execution post build scripts.

Change 261208 had a related patch set uploaded (by Bmansurov):
Browser test check

https://gerrit.wikimedia.org/r/261208

Change 260766 abandoned by Bmansurov:
Browser test check

Reason:
switching to dev: https://gerrit.wikimedia.org/r/#/c/261208/

https://gerrit.wikimedia.org/r/260766

The error is gone once I switched to the dev branch. The tests are being skipped because of the '@extension-mobilefrontend' tag [1] that was introduced recently [2]. Should we remove that tag?

[1] https://integration.wikimedia.org/ci/job/mwext-mw-selenium/2695/console

[2] https://gerrit.wikimedia.org/r/#/c/257410/

@bmansurov looks like the 2nd point still needs work - "Skipped feature due to missing MediaWiki extensions: mobilefrontend" - but yes getting close! :)

Patch is up => https://gerrit.wikimedia.org/r/#/c/260764/

@jhobs correctly points out that RelatedArticles depends on MediaWiki-extensions-Cards on 260764.

Change 260764 merged by jenkins-bot:
Setup RelatedArticles browser tests job

https://gerrit.wikimedia.org/r/260764

Change 262716 had a related patch set uploaded (by Hashar):
Rake entry point and rubocop fix up

https://gerrit.wikimedia.org/r/262716

Browsertest for RelatedArticles is deployed at https://integration.wikimedia.org/ci/job/browsertests-RelatedArticles-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/ It fails though because the master branch does not have a Gemfile yet.

The first run was against https://gerrit.wikimedia.org/r/#/c/261208/ which has been crafted to fail intentionally.

The CI change also introduced the rake command as an entry point. It is defined in the dev branch via https://gerrit.wikimedia.org/r/262716 which also fix up rubocop. The browser test fails though :(

Thanks, @hashar. What can we do to get https://gerrit.wikimedia.org/r/262716 merged if browser tests are failing?

Oops eggs and chicken really. Since the browser tests are failing, maybe the job should be disabled for now, we can move it from test/gate-and-submit to experimental.

Change 262900 had a related patch set uploaded (by Hashar):
[RelatedArticles] browser tests are experimental

https://gerrit.wikimedia.org/r/262900

https://gerrit.wikimedia.org/r/#/c/262900/ moves the selenium tests to experimental. So to trigger them you will have to manually comment in Gerrit check experimental to the job triggered.

Change 262900 merged by jenkins-bot:
[RelatedArticles] browser tests are experimental

https://gerrit.wikimedia.org/r/262900

Change 262716 merged by jenkins-bot:
Rake entry point and rubocop fix up

https://gerrit.wikimedia.org/r/262716

261208 demonstrates that check experimental fails as we require it to. Now we have to do the hard task of making the tests pass, right?

@Jdlrobson says that it's not Ready for Signoff as the tests aren't actually passing.

In T120715#1925560, @phuedx wrote:

261208 demonstrates that check experimental fails as we require it to. Now we have to do the hard task of making the tests pass, right?

Yup. You probably want to run the browser tests on your local machine and fix them up. Once happy, propose the change to Gerrit and comment check experimental. If all goes fine Jenkins will achieve the same result: SUCCESS.

Then we can adjust CI configuration to have the browser test triggered automatically via the Zuul test pipeline. Deploy that and then recheck the change and CR+2 it.

Dan / Zeljko would be able to help with the browser tests. We have a quick meeting at 4pm UTC / 5pm CET today. You are welcome to join :-}

@zeljkofilipin, what may be causing an element not to be seen by Watir? I see the element on the page, but I'm still getting this error message:

@extension-betafeatures
Scenario: ReadMore is present when enabled as a BetaFeature # features/read_more.feature:7
  Given I am logged into the website                        # features/support/step_definitions/common_steps.rb:39
  And ReadMore is enabled as a beta feature                 # features/support/step_definitions/common_steps.rb:69
    timed out after 5 seconds, waiting for {:css=>"#mw-input-wpread-more", :tag_name=>"input", :type=>"checkbox"} to become present (Watir::Wait::TimeoutError)
    ./features/support/pages/special_preferences_page.rb:11:in `enable_read_more'
    ./features/support/step_definitions/common_steps.rb:70:in `/^ReadMore is enabled as a beta feature$/'
    features/read_more.feature:9:in `And ReadMore is enabled as a beta feature'
  And I am on the "Related Articles 1" page                 # features/support/step_definitions/common_steps.rb:43
  And page has fully loaded with ReadMore code              # features/support/step_definitions/common_steps.rb:47
  Then I must see ReadMore                                  # features/support/step_definitions/common_steps.rb:73

Change 261208 abandoned by Bmansurov:
Browser test check

Reason:
Not needed for now

https://gerrit.wikimedia.org/r/261208

Is BeatureFeatures extension set as a dependency?

@Jdlrobson, care to explain more? A dependency in where? Both the BetaFeatures extension is installed and we have @extension-betafeatures above the test.

In T120715#1930208, @bmansurov wrote:

@zeljkofilipin, what may be causing an element not to be seen by Watir?

@bmansurov: it depends. :)

How can I reproduce the problem? I do not see RelatedArticles role in mediawiki-vagrant. Feel free to ping zeljkof at #wikimedia-releng, I should be around early in your day.

I have found where the test fails: https://integration.wikimedia.org/ci/job/mwext-mw-selenium/3602/console

It is hard for me to debug since mediawiki-vagrant does not have RelatedArticles role.

In T120715#2008249, @zeljkofilipin wrote:

How can I reproduce the problem? I do not see RelatedArticles role in mediawiki-vagrant. Feel free to ping zeljkof at #wikimedia-releng, I should be around early in your day.

@zeljkofilipin You just need to run the tests. It's the same error you mentioned in your patch at https://gerrit.wikimedia.org/r/#/c/269943/

Looks like Vector skin is not available in CI (screenshot, video): https://integration.wikimedia.org/ci/job/mwext-mw-selenium/3602/

When I run the last scenario targeting beta, this is what I get:

$ MEDIAWIKI_ENVIRONMENT=beta MEDIAWIKI_PASSWORD=not-the-real-one bundle exec cucumber features/read_more.feature:29
@staging @integration
Feature: ReadMore

  Background:                                      # features/read_more.feature:3
    Given RelatedArticles test pages are installed # features/support/step_definitions/common_steps.rb:1

  @extension-betafeatures
  Scenario: ReadMore is present when enabled as a BetaFeature # features/read_more.feature:29
    Given I am logged into the website                        # features/support/step_definitions/common_steps.rb:39
    And ReadMore is enabled as a beta feature                 # features/support/step_definitions/common_steps.rb:69
      timed out after 5 seconds, waiting for {:css=>"#preftab-betafeatures"} to become present (Watir::Wait::TimeoutError)
      ./features/support/pages/special_preferences_page.rb:10:in `enable_read_more'
      ./features/support/step_definitions/common_steps.rb:70:in `/^ReadMore is enabled as a beta feature$/'
      features/read_more.feature:31:in `And ReadMore is enabled as a beta feature'
    And I am on the "Related Articles 1" page                 # features/support/step_definitions/common_steps.rb:43
    And page has fully loaded with ReadMore code              # features/support/step_definitions/common_steps.rb:47
    Then I must see ReadMore                                  # features/support/step_definitions/common_steps.rb:73

Failing Scenarios:
cucumber features/read_more.feature:29 # Scenario: ReadMore is present when enabled as a BetaFeature

1 scenario (1 failed)
6 steps (1 failed, 3 skipped, 2 passed)
0m33.032s

Change 269971 had a related patch set uploaded (by Bmansurov):
Fix the failing browser test

https://gerrit.wikimedia.org/r/269971

Change 269995 had a related patch set uploaded (by Zfilipin):
WIP cleaning up Selenium tests

https://gerrit.wikimedia.org/r/269995

Change 269971 abandoned by Bmansurov:
Fix the failing browser test

Reason:
See the alternative patch above.

https://gerrit.wikimedia.org/r/269971

Change 270309 had a related patch set uploaded (by Zfilipin):
Separated RelatedArticles Selenium jobs to desktop and mobile

https://gerrit.wikimedia.org/r/270309

Change 269995 merged by jenkins-bot:
Fixed Selenium tests

https://gerrit.wikimedia.org/r/269995

Change 270309 abandoned by Zfilipin:
WIP Separated RelatedArticles Selenium jobs to desktop and mobile

Reason:
Better solution at https://gerrit.wikimedia.org/r/#/c/274365/

https://gerrit.wikimedia.org/r/270309

What happened here? It doesn't seem this ever happened (cc @Jhernandez )

@bmansurov @zeljkofilipin Any thoughts on the state of this task?

Given that we may remove ReadMore from wikis, I say we don't invest any more time in this task. That said, I'm happy to know what the state of this is. @zeljkofilipin?

In T120715#2486137, @bmansurov wrote:

Given that we may remove ReadMore from wikis,

Err, context please? Is there a discussion about this somewhere?

In T120715#2486141, @Legoktm wrote:

Err, context please? Is there a discussion about this somewhere?

T126826

@bmansurov AFAIK that's about desktop only actually. It seems like it didn't have the same engagement.

I think that the idea is to follow through in mobile web. I believe there's been research on the more-like queries to address community concerns with the addition of a no-boost popularity and things are moving along somewhere else.

That's why I'm vouching to enable them, if they prove certain features that we're going to have in beta/stable in some platform.

In T120715#2483192, @Jhernandez wrote:

@bmansurov @zeljkofilipin Any thoughts on the state of this task?

As far as I can see, mwext-mw-selenium job is still in experimental pipeline for RelatedArticles

I have started the job (with check experimental comment) at 298664.

The job is running relatively fine (mwext-mw-selenium/8423/console). There is only one failure (out of 4 scenarios):

(...)
(...) cucumber --color --tags @integration --tags '~@skip' (...)
(...)
00:00:50.938   @en.m.wikipedia.beta.wmflabs.org @extension-mobilefrontend
00:00:50.938   Scenario: ReadMore is present in minerva beta    # features/read_more.feature:14

00:01:32.159     Given I am using the mobile site               # features/support/step_definitions/common_steps.rb:15
00:01:32.159     And I am in mobile beta mode                   # features/support/step_definitions/common_steps.rb:26
00:01:32.159     And I am on the "Related Articles 1" page      # features/support/step_definitions/common_steps.rb:39
00:01:32.160     And page has fully loaded with ReadMore code   # features/support/step_definitions/common_steps.rb:43
00:01:32.160       timed out after 30 seconds (Watir::Wait::TimeoutError)
00:01:32.160       ./features/support/step_definitions/common_steps.rb:45:in `block (2 levels) in <top (required)>'
00:01:32.160       ./features/support/step_definitions/common_steps.rb:44:in `/^page has fully loaded with ReadMore code$/'
00:01:32.160       features/read_more.feature:18:in `And page has fully loaded with ReadMore code'
(...)
00:01:57.860 Failing Scenarios:
00:01:57.861 cucumber features/read_more.feature:14 # Scenario: ReadMore is present in minerva beta
00:01:57.861 
00:01:57.861 4 scenarios (1 failed, 3 passed)
00:01:57.861 22 steps (1 failed, 2 skipped, 19 passed)
00:01:57.861 1m23.892s
(...)

To fix the job, you can fix the failure, or remove the scenario from the test run. When the job is green, you can move the job from experimental pipeline.

(Please note that I will be on vacation from next week. I will reply as soon as I get back. @hashar can help while I am away.)

Given that misleading information above it's worth pointing out that wikivoyage have are using this extension in production.There are no plans to completely disable. Please be careful with strong statements like this @bmansurov.

Change 310139 had a related patch set uploaded (by Jdlrobson):
Beta should be available to test suite.

https://gerrit.wikimedia.org/r/310139

Change 310141 had a related patch set uploaded (by Jdlrobson):
Run RelatedArticles browser tests on every commit

https://gerrit.wikimedia.org/r/310141

Browser tests run via check experimental are now apassing as demonstrated here because of this patch

Once https://gerrit.wikimedia.org/r/310139 is merged we can make the necessary config change and wrap this up.

Change 310139 merged by jenkins-bot:
Beta should be available to test suite.

https://gerrit.wikimedia.org/r/310139

We just need to review/merge https://gerrit.wikimedia.org/r/#/c/310141/ now to add the necessary protection.

Change 310141 merged by jenkins-bot:
Run RelatedArticles browser tests on every commit

https://gerrit.wikimedia.org/r/310141

https://gerrit.wikimedia.org/r/#/c/310141/ got merged

CI has been updated to trigger the Selenium job when a patch is proposed or get CR+2.

I did a recheck against the last merged change https://gerrit.wikimedia.org/r/#/c/310050/ and against an open change https://gerrit.wikimedia.org/r/#/c/310381/ . Both pass just fine \o/

Congratulations.