Change Details

I'm seeing quite a few read-only errors in fatalmonitor. I didn't hold up the train because the failures don't appear to be related to the deployment of `1.32.0-wmf.15` (and the error rate hasn't changed after deploying to group2) These are apparently all coming from `labtestweb2001` ```name=channel:exception [f812f324c280f1d54cec7be6] [no req] JobQueueError from line 828 of /srv/mediawiki/php-1.32.0-wmf.15/includes/jobqueue/JobQueueDB.php: Wikimedia\Rdbms\DBQueryError: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? ` Query: UPDATE `job` SET job_token = 'f06fded40378d34ed3a7621fd30d58d0',job_token_timestamp = '20180802191202',job_attempts = job_attempts+1 WHERE job_cmd = 'cirrusSearchCheckerJob' AND job_id = '3487' AND job_token = ''` Function: JobQueueDB::claimRandom Error: 1290 The MariaDB server is running with the --read-only option so it cannot execute this statement` (10.192.32.8) ``` ```name=channel:memcached Memcached error for key "WANCache:m:global:Wikimedia\Rdbms\LoadBalancer:server-read-only:db2037" on server "127.0.0.1:11213": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY ``` See https://logstash.wikimedia.org/goto/d53c9f8625b96c9107ba83dcc72c97ac

I'm seeing quite a few read-only errors in fatalmonitor. I didn't hold up the train because the failures don't appear to be related to the deployment of `1.32.0-wmf.15` (and the error rate hasn't changed after deploying to group2) These are apparently all coming from `labtestweb2001` ```name=channel:exception [f812f324c280f1d54cec7be6] [no req] JobQueueError from line 828 of /srv/mediawiki/php-1.32.0-wmf.15/includes/jobqueue/JobQueueDB.php: Wikimedia\Rdbms\DBQueryError: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? ` Query: UPDATE `job` SET job_token = 'f06fded40378d34ed3a7621fd30d58d0',job_token_timestamp = '20180802191202',job_attempts = job_attempts+1 WHERE job_cmd = 'cirrusSearchCheckerJob' AND job_id = '3487' AND job_token = ''` Function: JobQueueDB::claimRandom Error: 1290 The MariaDB server is running with the --read-only option so it cannot execute this statement` (10.192.32.8) ``` See https://logstash.wikimedia.org/goto/d53c9f8625b96c9107ba83dcc72c97ac

I'm seeing quite a few read-only errors in fatalmonitor. I didn't hold up the train because the failures don't appear to be related to the deployment of `1.32.0-wmf.15` (and the error rate hasn't changed after deploying to group2) These are apparently all coming from `labtestweb2001` ```name=channel:exception [f812f324c280f1d54cec7be6] [no req] JobQueueError from line 828 of /srv/mediawiki/php-1.32.0-wmf.15/includes/jobqueue/JobQueueDB.php: Wikimedia\Rdbms\DBQueryError: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? ` Query: UPDATE `job` SET job_token = 'f06fded40378d34ed3a7621fd30d58d0',job_token_timestamp = '20180802191202',job_attempts = job_attempts+1 WHERE job_cmd = 'cirrusSearchCheckerJob' AND job_id = '3487' AND job_token = ''` Function: JobQueueDB::claimRandom Error: 1290 The MariaDB server is running with the --read-only option so it cannot execute this statement` (10.192.32.8) ``` ```name=channel:memcached Memcached error for key "WANCache:m:global:Wikimedia\Rdbms\LoadBalancer:server-read-only:db2037" on server "127.0.0.1:11213": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY ``` See https://logstash.wikimedia.org/goto/d53c9f8625b96c9107ba83dcc72c97ac