Figure out why we can't get information about the causes of deadlocks
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Eileenmcnaughton
	Oct 27 2016, 12:35 AM

Description

When we hit mysql deadlocks the victim query is reported but not the causing query.

It should be possible to see the latest causing query using SHOW INNODB ENGINE STATUS but last Jeff checked nothing was reported. This is probably a mysql config issue

http://dev.mysql.com/doc/refman/5.7/en/innodb-deadlocks.html
http://dev.mysql.com/doc/refman/5.5/en/innodb-deadlocks.html

If we can get them to show we can get 'real information' using pt-logger which will track this info over time
https://www.percona.com/doc/percona-toolkit/2.1/pt-deadlock-logger.html

Event Timeline

Eileenmcnaughton created this task.Oct 27 2016, 12:35 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 27 2016, 12:35 AM

(Nb this is a @jeff_green type bug

• DStrine assigned this task to Jgreen.Nov 1 2016, 9:08 PM

• DStrine triaged this task as High priority.

• DStrine added a project: fundraising-tech-ops.

• DStrine moved this task from Triage to FR-Ops on the Fundraising-Backlog board.

http://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_print_all_deadlocks

The documentation isn't terribly clear, but apparently innodb_print_all_deadlocks may be the only (?) way to see the 'causing' query as well as the 'victim' query. I enabled this and set up logging to fundraisingdb_mysql_err on the central logger hosts.

OK - let's wait for the next evidence of a deadlock and you can try SHOW ENGINE INNODB status and see if the info is there

In T149275#2766204, @Eileenmcnaughton wrote:

OK - let's wait for the next evidence of a deadlock and you can try SHOW ENGINE INNODB status and see if the info is there

Unless I'm misreading the documentation, we won't see additional deadlock information in engine status, just the victim query. But the causing query should show up in the now-centralized mysql log.

@Eileenmcnaughton I think the ticked task is complete, if you agree can you close the ticket, otherwise send it back to me with comments and I'll take another look.

Closing because I think we've fixed this as best we can at this point.

Dwisehaupt moved this task from Triage to Done on the fundraising-tech-ops board.Feb 13 2020, 9:38 PM

Figure out why we can't get information about the causes of deadlocksClosed, ResolvedPublicActions

Description

Event Timeline

Figure out why we can't get information about the causes of deadlocks
Closed, ResolvedPublic
Actions