Maniphest T201082

labtestweb2001 is sending updates to a read-only db host: db2037
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	• mmodell
	Aug 2 2018, 7:28 PM

Description

I'm seeing quite a few read-only errors in fatalmonitor. I didn't hold up the train because the failures don't appear to be related to the deployment of 1.32.0-wmf.15 (and the error rate hasn't changed after deploying to group2)

These are apparently all coming from labtestweb2001

channel:exception

[f812f324c280f1d54cec7be6] [no req]   JobQueueError from line 828 of /srv/mediawiki/php-1.32.0-wmf.15/includes/jobqueue/JobQueueDB.php: Wikimedia\Rdbms\DBQueryError:
A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? `
Query: UPDATE  `job` SET job_token = 'f06fded40378d34ed3a7621fd30d58d0',job_token_timestamp = '20180802191202',job_attempts = job_attempts+1 WHERE job_cmd = 'cirrusSearchCheckerJob' AND job_id = '3487' AND job_token = ''`

Function: JobQueueDB::claimRandom

Error: 1290 The MariaDB server is running with the --read-only option so it cannot execute this statement` (10.192.32.8)

See https://logstash.wikimedia.org/goto/d53c9f8625b96c9107ba83dcc72c97ac

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved	PRODUCTION ERROR	Andrew	T201082 labtestweb2001 is sending updates to a read-only db host: db2037
Resolved		CDanis	T233236 Move labtestwikitech database to clouddb2001-dev
Resolved		Marostegui	T236010 Drop labtestwikitech database from m5
Resolved		Andrew	T236145 processEchoEmailBatch.php failing for labtestwiki
Resolved		aborrero	T237971 Cron <www-data@cloudweb2001-dev> /usr/local/bin/mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=labswiki --force > /dev/null

Event Timeline

• mmodell created this task.Aug 2 2018, 7:28 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 2 2018, 7:28 PM

db2037 is read-only because it is m5 codfw master, and as codfw is the passive DC, nothing should write to it.
Whatever is trying to write to it, instead of using m5-master (db1073) probably needs to be reviewed.
Maybe @Andrew can help with this?

I am going to guess that is labtestwiki, which was added to db2037, but cannot live there, as there can only be one active (read-write) master at the moment, and that is m5-master.eqiad.wmnet . This is something that cloud people were already told, and we offered to setup a separate instance somewhere else (can be even the same host, but not the same instance).

• mmodell renamed this task from db2037 is read only? to labtestweb2001 is sending updates to a read-only db host: db2037.Aug 3 2018, 6:44 PM

Krinkle moved this task from Untriaged to Mar 2021 on the Wikimedia-production-error board.Aug 14 2018, 2:16 AM

• mmodell merged a task: T202674: labtestwiki jobqueue error.Aug 28 2018, 8:16 PM

• mmodell added subscribers: thcipriani, Krenair.

dduvall mentioned this in T191065: 1.32.0-wmf.19 deployment blockers.Aug 29 2018, 7:47 PM

hashar added a project: Release-Engineering-Team.Sep 4 2018, 3:51 PM

jcrespo mentioned this in T203479: labtestweb2001: Memcached error for key on server "127.0.0.1:11213": SERVER HAS FAILED.Sep 4 2018, 4:08 PM

Krinkle updated the task description. (Show Details)Sep 5 2018, 2:19 AM

Krinkle updated the task description. (Show Details)

greg edited projects, added Release-Engineering-Team (Watching / External); removed Release-Engineering-Team.Sep 5 2018, 3:08 PM

Andrew mentioned this in T167973: Move database for wikitech (labswiki) to a main cluster section.Sep 5 2018, 5:34 PM

The context here is that a while ago I moved the local labtestwikitech database off of labtestweb2002 because Jaime asked me to 'productionize' it and I misunderstood his request. I could certainly just move it back there (although that would leave me with the puzzle of what 'productionize' means) or we could move labtestwikitech to db1073 (which would require some kind of tunnel encryption) or... I'm open to suggestions.

It is not currently a problem to have labtestwikitech be read-only, other than that the error messages seem to alarm people. At some point when I take back up the task of moving wikitech to an SUL wiki I'll probably want it read/write again.

• GTirloni edited projects, added cloud-services-team (Kanban); removed cloud-services-team.Mar 24 2019, 12:04 PM

• Phabricator_maintenance added a project: Release-Engineering-Team-TODO.Jun 12 2019, 11:43 PM

• Phabricator_maintenance moved this task from Should be empty (use Release-Engineering-Team) to Watching/External on the Release-Engineering-Team-TODO board.Jun 12 2019, 11:48 PM

• Phabricator_maintenance removed a project: Release-Engineering-Team (Watching / External).Jun 12 2019, 11:48 PM

greg added a project: Release-Engineering-Team.Jun 21 2019, 10:34 PM