@zeljkofilipin: How can I see which environment variables are being set during a build? I'd like to verify that mediawiki_captcha_bypass_password is set and correct.

Build #268 is the last one that is green:

Success Build #268 (Jan 11, 2017 12:22:59 PM)

Failures have started with #269:

Failed Build #269 (Jan 12, 2017 12:22:59 PM)

Job configuration history is unfortunately empty :( so we can not see if anything changed there:

https://integration.wikimedia.org/ci/view/Selenium/job/selenium-GettingStarted/jobConfigHistory/

MEDIAWIKI_CAPTCHA_BYPASS_PASSWORD is set up in job-templates-selenium.yaml.

https://phabricator.wikimedia.org/diffusion/CICF/browse/master/jjb/job-templates-selenium.yaml;16e78b71e9a7b0101317a4a4011ba3a91ffbeb24$144-145

You can see it in the job configuration under Bindings:

https://integration.wikimedia.org/ci/view/Selenium/job/selenium-GettingStarted/configure

The value is visible here: https://office.wikimedia.org/wiki/Selenium_passwords

You can update it in Jenkins credential store:

https://integration.wikimedia.org/ci/credentials/store/system/domain/selenium/credential/mediawiki-captcha-bypass-password/update

The variable is not visible in the job console output:

https://integration.wikimedia.org/ci/view/Selenium/job/selenium-GettingStarted/297/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/console

In T155797#3008920, @phuedx wrote:

@zeljkofilipin: How can I see which environment variables are being set during a build?

I see I did not answer the question. I am not sure if there is a list of environment variables per job or build. Maybe @hashar would know.

In T155797#3008920, @phuedx wrote:

I'd like to verify that mediawiki_captcha_bypass_password is set and correct.

This is doable. You can edit and existing job, or even better, copy a job, and add echo $MEDIAWIKI_CAPTCHA_BYPASS_PASSWORD to the shell script that runs the tests. Run the job, copy/paste the output and delete the build and/or job immediately.

There is a small security risk, since console output will have the password. The risk can be minimized by deleting the build/job immediately after it finishes. (Our Jenkins installation is wide open. There might be a way to make a build/job private.)

There are some instructions on how to copy a job. You need to do just a few of the first steps:

https://www.mediawiki.org/wiki/Reading/Web/QA#Simulating_browser_test_run_on_an_unmerged_patchset.

I have added echo $MEDIAWIKI_CAPTCHA_BYPASS_PASSWORD just above rake selenium in a copy of selenium-GettingStarted job:

echo $MEDIAWIKI_CAPTCHA_BYPASS_PASSWORD

# run the tests
"$WORKSPACE"/vendor/bin/bundle exec rake selenium

The output showed that indeed the password is not the same as at the office wiki. I have updated the password in Jenkins credential store (pasted the value from office wiki) and restarted the selenium-GettingStarted job. The job still fails.

@phuedx I think MEDIAWIKI_CAPTCHA_BYPASS_PASSWORD is not correct. Could you please make sure both office wiki and Jenkins have the correct password?

Change 336770 had a related patch set uploaded (by Phuedx):
Fill required field when bypassing CAPTCHA

https://gerrit.wikimedia.org/r/336770

In theory if one want to decrypt a credential password:

Head to contint1001 and find the encrypted credential in /var/lib/jenkins/credentials.xml

Head to Jenkins Groovy console https://integration.wikimedia.org/ci/script and fill in:

println(hudson.util.Secret.decrypt("CRYPTIC SECRET HERE"))

Press Run and that should gives you the credential.

Change 336770 merged by jenkins-bot:
Fill required field when bypassing CAPTCHA

https://gerrit.wikimedia.org/r/336770

Note that while trying to figure this out, I did update the mediawiki_captcha_bypass_password credential in Jenkins to match the value of $wmgCaptchaPassword in /srv/mediawiki-staging/private/PrivateSettings.php on tin, which I thought was the correct deployment server. It didn't make the build pass so obviously I was wrong. I reset the credential immediately after the build failed.

Why would PrivateSettings.php be out of sync?

In T155797#3012540, @phuedx wrote:

Note that while trying to figure this out, I did update the mediawiki_captcha_bypass_password credential in Jenkins to match the value of $wmgCaptchaPassword in /srv/mediawiki-staging/private/PrivateSettings.php on tin, which I thought was the correct deployment server. It didn't make the build pass so obviously I was wrong. I reset the credential immediately after the build failed.

Why would PrivateSettings.php be out of sync?

Do you mean actual tin, or deployment-tin?

This runs on Beta Cluster, so it would be deployment-tin. tin is production.

In T155797#3129733, @Mattflaschen-WMF wrote:

Do you mean actual tin, or deployment-tin?

This runs on Beta Cluster, so it would be deployment-tin. tin is production.

It's highly likely that I ssh'd to the wrong server.