[8693793.015246] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) [8693795.023956] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) [8693796.860638] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) [8693798.869415] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) [8693800.834119] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) [8693802.810858] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) [8693804.807599] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) [8693806.724343] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) [8693808.733097] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250) [8693810.581756] XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-04-04T23:29:13Z] <jynus> unscheduled restart of dbstore1002 T162212
Comment Actions
Probably excessive memory pressure due to heavy mysql usage... blah blah blah... restarted cleanly ... updated kernel... check new import script... check long running queries,... mysql error log is clean... tokudb should die, blah blah etc
Comment Actions
There is also more load than usual since the 29, that could have contributed to it: https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=dbstore1002&from=now-30d&to=now
Comment Actions
Might be related to the work that has been done by some analysts with some SUPER heavy queries in the last few days...
Comment Actions
As I mentioned here: T159430#3153285 I would like to convert a couple of enwiki tables to InnoDB+compression to see if it helps this: https://jira.mariadb.org/browse/MDEV-9027 which we are suffering with some analyst queries I believe.