Do we really need a browser test for this? Those are the most expensive to develop and maintain and the least helpful when they break.

I can see the need for browser tests for the authorization dialog as that's nontrivial UI and essential functionality; for anything else, I would just use integration tests that create / recall some objects.

The problem is that to test it, we need a request token, so we need a consumer in the DB and the code to generate the request token by hitting /initialize... which all gets complicated.

One thing we could do is set up a test consumer and a test application page which can just live on beta/testwiki/vagrant permanently (https://gerrit.wikimedia.org/r/#/c/210036/ does something like that for vagrant). The browser test would navigate to the application page, press a button, get redirected to the authorization dialog, verify it's there, press OK, verify it ended ap on the app's confirmation page. That's a bit of work and the browser test would be somewhat fragile (as it would test both OAuth and the test app) but very simple.

At any rate, browser tests are usually the highest-hanging fruit, so starting with integration tests is definitely the right approach.

T105387 is another good example of an outage that should have been caught by a smoke test.

A python smoke test looks something like this:

import requests
from requests_oauthlib import OAuth1
from jose import jwt
auth = OAuth1(consumer_key, consumer_secret, access_token, access_secret)
r = requests.get('https://en.wikipedia.beta.wmflabs.org/w/api.php', params={'format': 'json', 'action': 'query', 'meta': 'userinfo'}, auth=auth)
data = r.json()
assert(data['query']['userinfo']['name'] == username)
r = requests.get('https://en.wikipedia.beta.wmflabs.org/w/index.php', params={'title': 'Special:OAuth/identify'}, auth=auth)
data = jwt.decode(r.text, consumer_secret, options={'verify_signature': False, 'verify_aud': False, 'verify_iat': False, 'verify_exp': False, 'verify_nbf': False, 'verify_iss': False, 'verify_sub': False, 'verify_jti': False})
assert(data['username'] == username)

Testing the authorization flow will require a proper browser test.

In T78314#2747466, @Tgr wrote:

Testing the authorization flow will require a proper browser test.

If you are creating a new browser/selenium test(s), I would recommend webdriverio (node.js):

https://www.mediawiki.org/wiki/Selenium/Node.js
https://github.com/zeljkofilipin/mediawiki-webdriverio
https://gerrit.wikimedia.org/r/#/c/328191/
https://gerrit.wikimedia.org/r/#/c/337602/

Let me know if you want to learn more, or need help.

I have proposed a skill share session for Vienna hackathon T159945: Selenium/WebdriverIO tests in JavaScript/Node.js if you would like to pair in person.