@zeljkofilipin and @Anomie you seem like the right experts to ask about my 2 questions. Can you shed any light on what could be going wrong here?

Oddly enough.. I ran the exact same tests locally and aside from some known issues with toasts "I should see a toast notification " the tests all passed and I did not hit the read only issue. Does Quibble have some kind of caching layer?

FWIW from a Beta-Cluster-Infrastructure perspective, editing appears to be working both in the UI and API. If you can find the HTTP request that actually gets sent to the beta cluster (preferably in the form of a curl command or similar) that triggers this, even inconsistently, I can take a look, but I don't know enough about your client setup to debug from here right now.

In T210557#4779906, @Krenair wrote:

FWIW from a Beta-Cluster-Infrastructure perspective, editing appears to be working both in the UI and API. If you can find the HTTP request that actually gets sent to the beta cluster (preferably in the form of a curl command or similar) that triggers this, even inconsistently, I can take a look, but I don't know enough about your client setup to debug from here right now.

Not sure how exactly these tests run... but I suspect either Saucelabs or the driver that runs these test has some kind of caching in place. I have not been able to replicate against the beta cluster via running the tests from the command line or manually executing the API queries. I suspect the browser testing infrastructure here is what's at fault... with some more debugging or time we should be able to get to the bottom of this.

Removing MediaWiki-Action-API, there doesn't seem to be anything here related to that tag.

If replica databases are lagged, MediaWiki will automatically switch to read-only mode, perhaps you're running into that. This particularly applies if your test user is flagged as a bot. You might check the logson deployment-fluorine02.deployment-prep.eqiad.wmflabs or https://logstash-beta.wmflabs.org/ after a failing run to see if anything interesting was logged. You might also have your test framework log the 'readonlyreason' field from the API error response, which should give you a better idea of why exactly it's read-only.

I noticed that the Selenium Sauce Labs jobs seem to pass when on saucelabs-03 and fail on saucelabs-01. Can we try running a couple selenium-MinervaNeue jobs on saucelabs-03?

Thanks @Anomie for that insight - that's super helpful and after looking into logstash-beta today it looks like this is exactly what's happening.

RelEng there is clearly something wrong with the beta cluster post migration. I've updated the description with the logstash URL. Given the impact this is happening on testing (and the severe regressions we've already seen this week e.g. T210670) I've bumped this to unbreak now.

Over the last few weeks the beta cluster was in read only mode.

This isn't true, afaict. There was the migration, but that started and ended last week (with some smaller follow-up this week).

But let's not quibble (get it) about that assertion, the Beta Cluster is indeed experiencing more read-only mode since the migration:

https://logstash-beta.wmflabs.org/goto/9075ea98170e327ef430c74cd55e83e0

That ^ is the same query as what @Jdlrobson added but viewing the past 2 weeks, which shows nothing before the migration, the obvious bump when it was migrating, and a continued low level after.

https://logstash-beta.wmflabs.org/goto/9075ea98170e327ef430c74cd55e83e0 (this is the same query as what @Jdlrobson added but viewing the past 2 weeks, which shows nothing before the migration, the obvious bump when it was migrating, and a continued low level after.

I'm asking people in RelEng, but this may need more direct mediawiki db interaction experience than we have.

Can we please try to find out what requests are being sent that trigger it?

Looking at SHOW SLAVE STATUS\G on deployment-db04 shows no current lag according to seconds_behind_master, but I'm looking into where we might see its historical values. I thought there was a prometheus collector set up for beta's maria DBs, but I'm not whether it monitors master/slave replication status.

In the meantime, @Krenair has set up a cronjob that monitors seconds_behind_master so we'll have that in case this happens again soon.

I'm also wondering how MediaWiki detects replication lag. Does it look at all DB hosts? And which MaridDB variables does it inspect? Is it possible that it's looking at deployment-db03 and errantly detecting it as a lagged replica?

Looking at SHOW SLAVE STATUS deployment-db03 does indicate that it's half configured as a replica, just not replicating due to it pointing to itself as master and an IO failure.

root@BETA[(none)]> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State:
                  Master_Host: deployment-db03
                  Master_User:
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File:
          Read_Master_Log_Pos: 4
               Relay_Log_File: deployment-db03-relay-bin.000005
                Relay_Log_Pos: 4
        Relay_Master_Log_File:
             Slave_IO_Running: No
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 0
              Relay_Log_Space: 248
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 1593
                Last_IO_Error: Fatal error: Invalid (empty) username when attempting to connect to the master server. Connection attempt te
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 0
               Master_SSL_Crl:
           Master_SSL_Crlpath:
                   Using_Gtid: No
                  Gtid_IO_Pos:

Sorry, I posted the wrong server's output. :) Editing...

In T210557#4791663, @dduvall wrote:

Note that I'm still seeing errors occurring that seem related to missing redis servers ("Failed connecting to redis server at deployment-redis05.deployment-prep.eqiad.wmflabs") but they seem related to the redis_lock configuration and should have additional tasks filed.

(T210301: Status of deployment-redis0[56]?)