It looks like we need to change the passwords of selenium-user-beta and selenium-user-production (proactively) under https://integration.wikimedia.org/ci/credentials/store/system/domain/selenium/

In T204569#4590313, @Reedy wrote:

It looks like we need to change the passwords of selenium-user-beta and selenium-user-production (proactively) under https://integration.wikimedia.org/ci/credentials/store/system/domain/selenium/

No. Those are only for Jenkins jobs that run daily and target beta/production, selenium-daily-beta-PROJECT, selenium-PROJECT, selenium-PROJECT-chrome. Quibble jobs don't use beta/production password.

I think this is the last time we had a similar problem: T188520: Use a more complex password for WikiAdmin in selenium tests.

The last time password was updated in integration/jenkins/415312.

Change 461138 had a related patch set uploaded (by Reedy; owner: Reedy):
[mediawiki/libs/PasswordBlacklist@master] Add testwikijenkinspass to passwords to test not blacklisted

https://gerrit.wikimedia.org/r/461138

This is currently in bin/mw-set-env.sh

# MediaWiki admin user
# Used by mw-install* scripts for creating the account,
# but also by used by selenium tests for logging-in
export MEDIAWIKI_USER="WikiAdmin"
export MEDIAWIKI_PASSWORD="testwikijenkinspass"

My patch isn't going to fix it, just testing whether the blacklist is blacklisting that password... I've added it to the core test too for completeness

In T204569#4590313, @Reedy wrote:

It looks like we need to change the passwords of selenium-user-beta and selenium-user-production (proactively) under https://integration.wikimedia.org/ci/credentials/store/system/domain/selenium/

So this was a guess of what was happening... Unfortunately the error messages are vague as to telling me why anything is failing

So... Error: Unrecognised hook ["after each" hook for "should be deletable"] for suite [Page] ???

14:49:07 [14:49:07] [E] [MWBOT] Login failed: WikiAdmin@http://127.0.0.1:9412/
14:49:08 
14:49:08 	Screenshot: /workspace/log/should-be-re-creatable.png
14:49:08 
14:49:08 
14:49:08 	Screenshot: /workspace/log/should-be-re-creatable.png
14:49:08 
14:49:10 [14:49:10] [E] [MWBOT] Login failed: WikiAdmin@http://127.0.0.1:9412/
14:49:10 [14:49:10] [E] [MWBOT] Login failed: WikiAdmin@http://127.0.0.1:9412/
14:49:10 (node:2517) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 4): TypeError: Cannot read property 'currentRetry' of undefined
14:49:11 
14:49:11 	Screenshot: /workspace/log/should-have-history-%40daily.png
14:49:11 
14:49:11 
14:49:11 	Screenshot: /workspace/log/should-have-history-%40daily.png
14:49:11 
14:49:11 /workspace/src/node_modules/webdriverio/build/lib/utils/ReporterStats.js:362
14:49:11             if (!suiteStats.hooks[uid]) throw Error(`Unrecognised hook [${runner.title}] for suite [${runner.parent}]`);
14:49:11                                         ^
14:49:11 
14:49:11 Error: Unrecognised hook ["after each" hook for "should be deletable"] for suite [Page]
14:49:11     at Error (native)
14:49:11     at ReporterStats.getHookStats (/workspace/src/node_modules/webdriverio/build/lib/utils/ReporterStats.js:362:47)
14:49:11     at ReporterStats.hookEnd (/workspace/src/node_modules/webdriverio/build/lib/utils/ReporterStats.js:332:34)
14:49:11     at BaseReporter.<anonymous> (/workspace/src/node_modules/webdriverio/build/lib/utils/BaseReporter.js:147:25)
14:49:11     at emitOne (events.js:96:13)
14:49:11     at BaseReporter.emit (events.js:188:7)
14:49:11     at BaseReporter.handleEvent (/workspace/src/node_modules/webdriverio/build/lib/utils/BaseReporter.js:300:27)
14:49:11     at Launcher.messageHandler (/workspace/src/node_modules/webdriverio/build/lib/launcher.js:688:28)
14:49:11     at emitTwo (events.js:106:13)
14:49:11     at ChildProcess.emit (events.js:191:7)
14:49:11

Ok... So

https://gerrit.wikimedia.org/r/#/c/mediawiki/libs/PasswordBlacklist/+/461138/ confirms the password blacklist isn't rejecting testwikijenkinspass as is supposed to be used based on https://gerrit.wikimedia.org/r/#/c/integration/jenkins/+/415312/5/bin/mw-set-env.sh

https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/414603/ adds the same password to core tests, and that doesn't fail either...

On top of the core patch, I've made another to core, disabling the password blacklisting checks for all the groups - https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/461175/ the tests pass. Narrowed to just for sysops, and that's causing the failure

So either it's causing an unintended browser test failure (for some reason), or the password being used by the failing test isn't the one we think it is for the sysop user (ie it's not actually testwikijenkinspass, it's some other blacklisted password)

18:38:53 [chrome #0-1] Could not login: Aborted

This error is kinda useless all around

Then there's errors like...

18:38:53 (node:1451) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 4): TypeError: Cannot read property 'isPending' of undefined

I just found this in CirrusSearch. Certainly looks supicious - https://gerrit.wikimedia.org/g/mediawiki/extensions/CirrusSearch/+/58fac2e4440f927c84ee0c29ad084774850d940f/tests/integration/config/wdio.conf.jenkins.js#12

https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CirrusSearch/+/461190/ removes the overrides

wmf-quibble-core-vendor-mysql-hhvm-docker FAILURE in 13m 07s
mediawiki-quibble-composer-mysql-php70-docker FAILURE in 5m 54s
mediawiki-quibble-vendor-mysql-hhvm-docker FAILURE in 4m 20s
mediawiki-quibble-vendor-mysql-php70-docker FAILURE in 7m 07s

integration/jenkins.git and bin/mw-set-env.sh were used with the legacy jobs running on Nodepool. They are deprecated and not much used anymore. Instead the logic to install MediaWiki and run tests has been ported to Quibble, it uses the dummy password testpass.

https://github.com/wikimedia/integration-quibble/blob/3da498c/quibble/mediawiki/maintenance.py#L47

def install(args, mwdir=None):
    log = logging.getLogger('mw.maintenance.install')

    cmd = ['php', 'maintenance/install.php']
    cmd.extend(args)
    cmd.extend([
        '--with-extensions',  # T189567
        '--pass=testpass',
        'TestWiki',
        'WikiAdmin'
    ])
    log.info(' '.join(cmd))

The Selenium tests are also being launched by Quibble, the password is set via the environment variable MEDIAWIKI_PASSWORD:

def run_webdriver(mwdir, display, port=9412):
    webdriver_env = {}
    webdriver_env.update(os.environ)
    webdriver_env.update({
        'MW_SERVER': 'http://127.0.0.1:%s' % port,
        'MW_SCRIPT_PATH': '/',
        'FORCE_COLOR': '1',  # for 'supports-color'
        'MEDIAWIKI_USER': 'WikiAdmin',      #  /
        'MEDIAWIKI_PASSWORD': 'testpass',   # <------------- here! -----------------
        'DISPLAY': display,                 #  \
    })

So I guess we can change the password in Quibble and that will solve it. Need a patch for integration/quibble, test it is actually working, then we cut a version, update the Docker containers and refresh all Jenkins jobs. Should be testable with:

$ ZUUL_URL=https://gerrit.wikimedia.org/r/p ZUUL_PROJECT=mediawiki/core ZUUL_REF=refs/changes/414603/24 quibble --run=selenium

Change 461195 had a related patch set uploaded (by Reedy; owner: Reedy):
[integration/quibble@master] Use stronger password in quibble related browser tests

https://gerrit.wikimedia.org/r/461195

Thanks @hashar!

Patch above to update the passwords there... :)

Change 461138 merged by jenkins-bot:
[mediawiki/libs/PasswordBlacklist@master] Add testwikijenkinspass to passwords to test not blacklisted

https://gerrit.wikimedia.org/r/461138

Change 465606 had a related patch set uploaded (by Reedy; owner: Reedy):
[integration/config@master] Rebuild quibble to bring in I79c015b2a8a029eeb8746c1fad201b6e801935b3

https://gerrit.wikimedia.org/r/465606

Change 461195 merged by jenkins-bot:
[integration/quibble@master] Use stronger password in quibble related browser tests

https://gerrit.wikimedia.org/r/461195

Change 465606 merged by jenkins-bot:
[integration/config@master] Bump Quibble images to 0.0.27

https://gerrit.wikimedia.org/r/465606