Possibly related to T324120: Unable to save edits on beta cluster.

Also possibly related to T324121: Unable to logout from beta cluster

From deployment-mwlog01.deployment-prep.eqiad1.wikimedia.cloud:/srv/mw-logobjectcache.log

2022-11-30 18:37:45.509493 [157d7dc9746d33541496eca2] deployment-mwmaint02 jawiki 1.40.0-alpha objectcache ERROR: DBError: Cannot access the database: Access denied for user 'wikiadmin'@'172.16.%' to database 'mainstash' (172.16.0.238:3306)

There do not seem to be any MediaWiki user grants for the "mainstash" database. Is this a new thing that is only partially setup?

root@BETA[mysql]> select host, db from db where user = 'wikiadmin';
+----------+-------------+
| host     | db          |
+----------+-------------+
| 10.%     | %wik%       |
| 10.%     | centralauth |
| 172.16.% | %wik%       |
| 172.16.% | centralauth |
+----------+-------------+
4 rows in set (0.001 sec)

root@BETA[mysql]> select host, db from db where user = 'wikiuser';
+----------+-------------+
| host     | db          |
+----------+-------------+
| 10.%     | %a%         |
| 10.%     | %wik%       |
| 10.%     | centralauth |
| 172.16.% | %a%         |
| 172.16.% | %wik%       |
+----------+-------------+
5 rows in set (0.001 sec)

In T324128#8432856, @bd808 wrote:

There do not seem to be any MediaWiki user grants for the "mainstash" database. Is this a new thing that is only partially setup?

The GRANT SELECT, INSERT, UPDATE, DELETE ON '%a%'.* TO 'wikiuser'@'172.16.%' grant covers "mainstash".

Cassandra was down, OOM-killed by the kernel. I've resized the VM (from 2G to 4G memory) and the service is back up. The VM was (is!) very memory constrained, but nonetheless I wouldn't have expected this, so hopefully there isn't something else going on. I'll continue to monitor it.

In T324128#8542805, @Silvan_WMDE wrote:

Looks like the issue is back since 13.01.2023, 18:03:00 - WikibaseLexeme selenium tests are failing due to this.

It seems as though Cassandra continues to be killed by the kernel (OOM), so I've opened T327521 to look into this more properly. What seems to be different this time is that after 17 instances of Cassandra being killed, and 17 subsequent restarts by puppet-agent (since December 6), Kask failed to recover due to errors associated with its connection pool. I've opened T327524 to track this.

It's back up now; I'll try to keep an eye on it, but if you notice any more failures, let me know.