selenium-CirrusSearch-jessie does not run any tests
Closed, ResolvedPublic

Description

Since #97 selenium-CirrusSearch-jessie does not run any tests.

From #107:

00:02:13.115 + cd /home/jenkins/workspace/selenium-CirrusSearch-jessie/src
...
00:01:05.696 + ./node_modules/.bin/wdio tests/selenium/wdio.conf.js --spec extensions/CirrusSearch/tests/selenium/specs/smoke_test.js
...
00:01:06.405 pattern /home/jenkins/workspace/selenium-CirrusSearch-jessie/src/extensions/VisualEditor/modules/ve-mw/tests/selenium/specs/**/*.js did not match any file
00:01:06.406 pattern /home/jenkins/workspace/selenium-CirrusSearch-jessie/src/skins/*/tests/selenium/specs/**/*.js did not match any file

Done

🏝 444609 Selenium: create selenium-daily NPM script

Restricted Application added a subscriber: Aklapper. Β· View Herald Transcript

#96 had a different problem, resolved by 427362

00:01:16.922 + ./node_modules/.bin/wdio tests/selenium/wdio.conf.jenkins.js --spec extensions/CirrusSearch/tests/selenium/specs/smoke_test.js
...
00:01:17.784 Failed loading configuration file: /home/jenkins/workspace/selenium-CirrusSearch-jessie/src/tests/selenium/wdio.conf.jenkins.js

A couple of similar jobs are working just fine.

selenium-Popups-jessie/205/console

00:01:17.899 + cd /home/jenkins/workspace/selenium-Popups-jessie/src
...
00:01:17.899 + ./node_modules/.bin/wdio tests/selenium/wdio.conf.js --spec extensions/Popups/tests/selenium/specs/previews.js
....
00:01:18.676 pattern /home/jenkins/workspace/selenium-Popups-jessie/src/extensions/VisualEditor/modules/ve-mw/tests/selenium/specs/**/*.js did not match any file
00:01:18.677 pattern /home/jenkins/workspace/selenium-Popups-jessie/src/skins/*/tests/selenium/specs/**/*.js did not match any file
00:01:18.809 pattern ./extensions/CirrusSearch/tests/selenium/specs/**/*.js did not match any file
00:01:19.441 pattern /home/jenkins/workspace/selenium-Popups-jessie/src/extensions/VisualEditor/modules/ve-mw/tests/selenium/specs/**/*.js did not match any file
00:01:19.441 pattern /home/jenkins/workspace/selenium-Popups-jessie/src/skins/*/tests/selenium/specs/**/*.js did not match any file
...
00:01:30.301 [chrome #0-0] Spec: /home/jenkins/workspace/selenium-Popups-jessie/src/extensions/Popups/tests/selenium/specs/previews.js
00:01:30.301 [chrome #0-0] Running: chrome
00:01:30.301 [chrome #0-0]
00:01:30.301 [chrome #0-0]   Dwelling on a valid link
00:01:30.302 [chrome #0-0]
00:01:30.302 [chrome #0-0]   Dwelling on a valid link
00:01:30.302 [chrome #0-0]       βœ“ I should see a preview
00:01:30.302 [chrome #0-0]       βœ“ Abandoning link hides preview
00:01:30.303 [chrome #0-0]
00:01:30.303 [chrome #0-0]
00:01:30.303 [chrome #0-0] 2 passing (11s)
...
00:01:30.335 2 passing (11.60s)

selenium-RelatedArticles-jessie/313/console

00:01:16.853 + cd /home/jenkins/workspace/selenium-RelatedArticles-jessie/src
...
00:01:16.854 + ./node_modules/.bin/wdio tests/selenium/wdio.conf.js --spec extensions/RelatedArticles/tests/selenium/specs/readmore.js
...
00:01:17.754 pattern /home/jenkins/workspace/selenium-RelatedArticles-jessie/src/extensions/VisualEditor/modules/ve-mw/tests/selenium/specs/**/*.js did not match any file
00:01:17.754 pattern /home/jenkins/workspace/selenium-RelatedArticles-jessie/src/skins/*/tests/selenium/specs/**/*.js did not match any file
00:01:17.934 pattern ./extensions/CirrusSearch/tests/selenium/specs/**/*.js did not match any file
00:01:18.648 pattern /home/jenkins/workspace/selenium-RelatedArticles-jessie/src/extensions/VisualEditor/modules/ve-mw/tests/selenium/specs/**/*.js did not match any file
00:01:18.649 pattern /home/jenkins/workspace/selenium-RelatedArticles-jessie/src/skins/*/tests/selenium/specs/**/*.js did not match any file
...
00:01:24.718 [chrome #0-0] Spec: /home/jenkins/workspace/selenium-RelatedArticles-jessie/src/extensions/RelatedArticles/tests/selenium/specs/readmore.js
00:01:24.719 [chrome #0-0] Running: chrome
00:01:24.719 [chrome #0-0]
00:01:24.719 [chrome #0-0]   ReadMore
00:01:24.719 [chrome #0-0]
00:01:24.719 [chrome #0-0]   ReadMore
00:01:24.719 [chrome #0-0]       βœ“ ReadMore is not present on Vector
00:01:24.720 [chrome #0-0]       βœ“ ReadMore is present in Minerva
00:01:24.720 [chrome #0-0]
00:01:24.720 [chrome #0-0]
00:01:24.720 [chrome #0-0] 2 passing (6s)
...
00:01:24.750 2 passing (7.00s)

The problem is reproducible outside of CI, it fails on my machine too.

~/Documents/gerrit/mediawiki/vagrant/mediawiki$ export MEDIAWIKI_PASSWORD=not-the-real-one
~/Documents/gerrit/mediawiki/vagrant/mediawiki$ export MEDIAWIKI_USER='Selenium user'
~/Documents/gerrit/mediawiki/vagrant/mediawiki$ export MW_SERVER=https://en.wikipedia.beta.wmflabs.org:443
~/Documents/gerrit/mediawiki/vagrant/mediawiki$ export MW_SCRIPT_PATH=/w
~/Documents/gerrit/mediawiki/vagrant/mediawiki$ ./node_modules/.bin/wdio tests/selenium/wdio.conf.js --spec extensions/Popups/tests/selenium/specs/previews.js
pattern /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/VisualEditor/modules/ve-mw/tests/selenium/specs/**/*.js did not match any file
pattern /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/*/tests/selenium/specs/**/*.js did not match any file
pattern /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/VisualEditor/modules/ve-mw/tests/selenium/specs/**/*.js did not match any file
pattern /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/*/tests/selenium/specs/**/*.js did not match any file
...
[chrome #0-0] Spec: /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/Popups/tests/selenium/specs/previews.js
[chrome #0-0] Running: chrome
[chrome #0-0]
[chrome #0-0]   Dwelling on a valid link
[chrome #0-0]
[chrome #0-0]   Dwelling on a valid link
[chrome #0-0]       βœ“ I should see a preview
[chrome #0-0]       βœ“ Abandoning link hides preview
[chrome #0-0]
[chrome #0-0]
[chrome #0-0] 2 passing (12s)
[chrome #0-0]

Wrote xunit report to [./log/].

2 passing (12.30s)
~/Documents/gerrit/mediawiki/vagrant/mediawiki$ ./node_modules/.bin/wdio tests/selenium/wdio.conf.js --spec extensions/RelatedArticles/tests/selenium/specs/readmore.js 
pattern /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/VisualEditor/modules/ve-mw/tests/selenium/specs/**/*.js did not match any file
pattern /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/*/tests/selenium/specs/**/*.js did not match any file
pattern /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/VisualEditor/modules/ve-mw/tests/selenium/specs/**/*.js did not match any file
pattern /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/*/tests/selenium/specs/**/*.js did not match any file
[16:38:57] [S] [MWBOT] Login successful: Selenium user@https://en.wikipedia.beta.wmflabs.org:443/w
------------------------------------------------------------------
[chrome #0-0] Session ID: a5bbf8c7789e2ce77ac0e4bd5b9f1a28
[chrome #0-0] Spec: /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/RelatedArticles/tests/selenium/specs/readmore.js
[chrome #0-0] Running: chrome
[chrome #0-0]
[chrome #0-0]   ReadMore
[chrome #0-0]
[chrome #0-0]   ReadMore
[chrome #0-0]       βœ“ ReadMore is not present on Vector
[chrome #0-0]       βœ“ ReadMore is present in Minerva
[chrome #0-0]
[chrome #0-0]
[chrome #0-0] 2 passing (10s)
[chrome #0-0]

Wrote xunit report to [./log/].

2 passing (11.00s)
~/Documents/gerrit/mediawiki/vagrant/mediawiki$ ./node_modules/.bin/wdio tests/selenium/wdio.conf.js --spec extensions/CirrusSearch/tests/selenium/specs/smoke_test.js 
pattern /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/VisualEditor/modules/ve-mw/tests/selenium/specs/**/*.js did not match any file
pattern /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/*/tests/selenium/specs/**/*.js did not match any file
zeljkofilipin added a comment.EditedApr 27 2018, 2:46 PM

It's because of this in wdio.conf.js introduced in 424763.

 // Patterns to exclude.
 exclude: [
   './extensions/CirrusSearch/tests/selenium/specs/**/*.js'
],
zeljkofilipin added a comment.EditedApr 27 2018, 3:18 PM

A simple solution would be to overwrite exclude with and empty array (see 429435):

wdio.smoke.js

const merge = require( 'deepmerge' ),
	wdioConf = require( './wdio.conf.js' );

// Overwrite default settings
exports.config = merge( wdioConf.config, {
	exclude: []
} );

but looks like it does not work, exclude is still [ './extensions/CirrusSearch/tests/selenium/specs/**/*.js' ]

~/Documents/gerrit/mediawiki/vagrant/mediawiki$ ./node_modules/.bin/wdio tests/selenium/wdio.smoke.js --spec extensions/CirrusSearch/tests/selenium/specs/smoke_test.js 
pattern /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/VisualEditor/modules/ve-mw/tests/selenium/specs/**/*.js did not match any file
pattern /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/*/tests/selenium/specs/**/*.js did not match any file
[ { maxInstances: 1,
    browserName: 'chrome',
    chromeOptions: { args: [Array] } } ]
{ host: '127.0.0.1',
  port: 4444,
  sync: true,
  specs: 
   [ '/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/extensions/CirrusSearch/tests/selenium/specs/smoke_test.js' ],
  suites: {},
  exclude: [ './extensions/CirrusSearch/tests/selenium/specs/**/*.js' ],
  logLevel: 'error',
  coloredLogs: true,
  deprecationWarnings: true,
  baseUrl: 'https://en.wikipedia.beta.wmflabs.org:443/w',
  bail: 0,
  waitforInterval: 500,
  waitforTimeout: 10000,
  framework: 'mocha',
  reporters: [ 'spec', 'junit' ],
  reporterOptions: { junit: { outputDir: './log/' } },
  maxInstances: 1,
  maxInstancesPerCapability: 100,
  connectionRetryTimeout: 90000,
  connectionRetryCount: 3,
  debug: false,
  execArgv: null,
  mochaOpts: { timeout: 20000, ui: 'bdd' },
  jasmineNodeOpts: { defaultTimeoutInterval: 10000 },
  onPrepare: [Function: NOOP],
  before: [],
  beforeSession: [],
  beforeSuite: [],
  beforeHook: [],
  beforeTest: [],
  beforeCommand: [],
  afterCommand: [],
  afterTest: [ [Function: bound afterTest] ],
  afterHook: [],
  afterSuite: [],
  afterSession: [],
  after: [],
  onComplete: [Function: onComplete],
  onError: [],
  onReload: [],
  beforeFeature: [],
  beforeScenario: [],
  beforeStep: [],
  afterFeature: [],
  afterScenario: [],
  afterStep: [],
  username: 'Selenium user',
  password: 'tnV9YtF5dpUUhd8',
  services: [ 'sauce' ],
  user: undefined,
  key: undefined,
  screenshotPath: './log/',
  spec: 'extensions/CirrusSearch/tests/selenium/specs/smoke_test.js',
  watch: false }

Change 429435 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] WIP selenium-CirrusSearch-jessie does not run any tests

https://gerrit.wikimedia.org/r/429435

zeljkofilipin added a subscriber: hashar.EditedApr 27 2018, 3:29 PM

A simple example seems to work fine (thanks @hashar):

// $ node

> const merge = require('deepmerge');
undefined

> let config = {a:1, b:2};
undefined

> config
{ a: 1, b: 2 }

> let configChange = {b:3};
undefined

> configChange
{ b: 3 }

> merge(config, configChange);
{ a: 1, b: 3 }
zeljkofilipin added a comment.EditedMay 4 2018, 4:21 PM
// $ node

> const merge = require('deepmerge');
undefined

> let config = {a:[1]};
undefined

> config
{ a: [ 1 ] }

> let configChange = {a:[2]};
undefined

> configChange
{ a: [ 2 ] }

> merge(config, configChange);
{ a: [ 1, 2 ] }

configChange does not override config, just adds to it.

From KyleAMathews/deepmerge#arraymerge

// $ node

> const merge = require('deepmerge');
undefined

> let config = {a:[1]};
undefined

> config
{ a: [ 1 ] }

> let configChange = {a:[2]};
undefined

> configChange
{ a: [ 2 ] }

> merge(config, configChange);
{ a: [ 1, 2 ] }

> function overwriteMerge(destinationArray, sourceArray, options) {
... 	return sourceArray
... }
undefined

> merge(config, configChange,{ arrayMerge: overwriteMerge });
{ a: [ 2 ] }
zeljkofilipin added a comment.EditedMay 4 2018, 5:16 PM

Created test job: selenium-CirrusSearch-jessie-429435

...
00:01:21.339 [chrome #0-0] Spec: /home/jenkins/workspace/selenium-CirrusSearch-jessie-429435/src/extensions/CirrusSearch/tests/selenium/specs/smoke_test.js
00:01:21.339 [chrome #0-0] Running: chrome
00:01:21.339 [chrome #0-0]
00:01:21.339 [chrome #0-0]   Smoke test for search
00:01:21.339 [chrome #0-0]
00:01:21.339 [chrome #0-0]   Smoke test for search
00:01:21.339 [chrome #0-0]       βœ“ Search suggestions
00:01:21.340 [chrome #0-0]       βœ“ Fill in search term and click search
00:01:21.340 [chrome #0-0]       βœ“ Search with accent yields result page with accent
00:01:21.340 [chrome #0-0]
00:01:21.340 [chrome #0-0]
00:01:21.340 [chrome #0-0] 3 passing (8s)
...

Change 430944 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] selenium: Do not exclude CirrusSearch when running smoke tests

https://gerrit.wikimedia.org/r/430944

zeljkofilipin removed zeljkofilipin as the assignee of this task.

Change 444609 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/extensions/CirrusSearch@master] Selenium: create selenium-daily NPM script

https://gerrit.wikimedia.org/r/444609

zeljkofilipin updated the task description. (Show Details)Jul 9 2018, 3:52 PM

Change 444609 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Selenium: Create selenium-daily NPM script

https://gerrit.wikimedia.org/r/444609

zeljkofilipin updated the task description. (Show Details)

Change 429435 abandoned by Zfilipin:
selenium: Do not exclude CirrusSearch when running smoke tests

Reason:
Resolved by T188742

https://gerrit.wikimedia.org/r/429435

Change 430944 abandoned by Zfilipin:
selenium: Do not exclude CirrusSearch when running smoke tests

Reason:
Resolved by T188742

https://gerrit.wikimedia.org/r/430944