The entire mobile site is without browser test coverage.
Over the last few weeks the beta cluster was in read only mode. I am told this is no longer the case, but we are consistently seeing failures [[ https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-MinervaNeue/ | in our browser tests ]] stating "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)". I've spent much of day trying to find out the cause of this without success, so I'm turning to Phabricator and hopefully a wider audience to track down the source of this problem. The strange thing is, is that this is inconsistent - sometimes the same scenarios calling exactly the same API queries pass!
It's been impossible to replicate this issue locally so far but the issue can be traced to the mediawiki_selenium Gem (we are using 1.7.3). The HTTP API request being made that triggers the MediawikiApi::ApiError is a simple edit action which look a little like this:
https://en.wikipedia.beta.wmflabs.org/wiki/Special:ApiSandbox#action=edit&format=json&title=Selenium%20search%20test&text=Test%20is%20used%20by%20Selenium%20web%20driver&token=a58125cfef2cf4b200e4953376ace77d5bfd8ba0%2B%5C
```
The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)
/mnt/home/jenkins-deploy/.gem/2.1.0/gems/mediawiki_api-0.7.0/lib/mediawiki_api/client.rb:211:in `send_request'
/mnt/home/jenkins-deploy/.gem/2.1.0/gems/mediawiki_api-0.7.0/lib/mediawiki_api/client.rb:232:in `raw_action'
/mnt/home/jenkins-deploy/.gem/2.1.0/gems/mediawiki_api-0.7.0/lib/mediawiki_api/client.rb:36:in `action'
/mnt/home/jenkins-deploy/.gem/2.1.0/gems/mediawiki_api-0.7.0/lib/mediawiki_api/client.rb:111:in `edit'
/mnt/home/jenkins-deploy/.gem/2.1.0/gems/mediawiki_api-0.7.0/lib/mediawiki_api/client.rb:103:in `create_page'
```
Given repeating the request can be done so without the error, the only possible explanation I can think of is that the read-only state has been cached somewhere in the beta cluster stack and only applies to the Selenium user that's being used to run the browser tests.
This is high priority from web side as every day this remains broken we are without test coverage for a key part of the site and we are out of our depth here.
= Questions
[ ] Is there somebody who understands the mediawiki_api Ruby package and can help provide some more detailed debug statements of what's happening in the browser tests?
[ ] Are there any reasons the API might die with a read-only error after read-only mode after a wiki has transitioned back to write mode?