Page MenuHomePhabricator

dbstore1002 in bad shape
Closed, ResolvedPublic

Description

[8693793.015246] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
[8693795.023956] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
[8693796.860638] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
[8693798.869415] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
[8693800.834119] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
[8693802.810858] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
[8693804.807599] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
[8693806.724343] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
[8693808.733097] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
[8693810.581756] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)

Event Timeline

jcrespo created this task.Apr 4 2017, 11:27 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 4 2017, 11:27 PM

Mentioned in SAL (#wikimedia-operations) [2017-04-04T23:29:13Z] <jynus> unscheduled restart of dbstore1002 T162212

Probably excessive memory pressure due to heavy mysql usage... blah blah blah... restarted cleanly ... updated kernel... check new import script... check long running queries,... mysql error log is clean... tokudb should die, blah blah etc

Might be related to the work that has been done by some analysts with some SUPER heavy queries in the last few days...

As I mentioned here: T159430#3153285 I would like to convert a couple of enwiki tables to InnoDB+compression to see if it helps this: https://jira.mariadb.org/browse/MDEV-9027 which we are suffering with some analyst queries I believe.

jcrespo closed this task as Resolved.Apr 5 2017, 7:42 AM
jcrespo claimed this task.

Sure. For now I will close this as it seems healthy again.