Page MenuHomePhabricator

s51434 user with long running writes on labsdb1005
Closed, ResolvedPublic

Description

We have seen this user creating issues to itself mostly, with long running inserts which are locking themselves.

root@labsdb1005.eqiad.wmnet[(none)]> show full processlist;
| 124602858 | s51434          | 10.68.18.9:34751   | s51434__mixnmatch_p                   | Query       |   18588 | Waiting for table metadata lock                                       | TRUNCATE common_names_human                                                                                                                                                                                                                                           |    0.000 |
| 124613830 | s51434          | 10.68.23.154:47035 | s51434__mixnmatch_p                   | Query       |    1466 | Sending data                                                          | SELECT /*!40001 SQL_NO_CACHE */ * FROM `entry`                                                                                                                                                                                                                        |    0.000 |
| 124613884 | s51434          | 10.68.17.89:53344  | s51434__mixnmatch_p                   | Query       |   19516 | Waiting for table level lock                                          | INSERT IGNORE INTO overview (catalog) VALUES (1734)                                                                                                                                                                                                                   |    0.000 |
| 124614704 | s51434          | 10.68.18.14:39975  | s51434__mixnmatch_p                   | Query       |   16737 | Waiting for table level lock                                          | UPDATE entry SET q=56503087,user=4,`timestamp`='20180907045734' WHERE id=30067009 AND (user is null or user=0 or q=-1)                                                                                                                                                |    0.000 |
| 124616592 | s51434          | 10.68.17.89:54630  | s51434__mixnmatch_p                   | Query       |   19189 | Waiting for table level lock                                          | INSERT IGNORE user (name,last_block_check) VALUES ('redacted','1536293802')                                                                                                                                                                                           |    0.000 |
| 124616660 | s51434          | 10.68.17.89:54634  | s51434__mixnmatch_p                   | Query       |   19183 | Waiting for table level lock                                          | INSERT IGNORE user (name,last_block_check) VALUES ('redacted','1536293808')                                                                                                                                                                                           |    0.000 |
| 124616745 | s51434          | 10.68.17.89:54640  | s51434__mixnmatch_p                   | Query       |   19174 | Waiting for table level lock                                          | INSERT IGNORE user (name,last_block_check) VALUES ('redacted','1536293817')                                                                                                                                                                                           |    0.000 |
| 124619317 | s51434          | 10.68.19.159:56107 | s51434__mixnmatch_p                   | Query       |   18868 | Waiting for table level lock                                          | UPDATE autoscrape SET status='RUNNING' WHERE id=986                                                                                                                                                                                                                   |    0.000 |
| 124620447 | s51434          | 10.68.23.172:51658 | s51434__mixnmatch_p                   | Query       |   18731 | Waiting for table level lock                                          | UPDATE entry SET q=15945237,user=4,`timestamp`='20180907042420' WHERE id=24062596 AND (user is null or user=0 or q=-1)                                                                                                                                                |    0.000 |
| 124643471 | s51434          | 10.68.21.65:52337  | s51434__mixnmatch_p                   | Query       |   16852 | Waiting for table level lock                                          | UPDATE entry SET q=56452973,user=4,`timestamp`='20180907045539' WHERE id=57025604 AND (user is null or user=0 or q=-1)                                                                                                                                                |    0.000 |
| 124658872 | s51434          | 10.68.21.42:60753  | s51434__mixnmatch_p                   | Query       |   15267 | Waiting for table level lock                                          | UPDATE autoscrape SET status='RUNNING' WHERE id=986                                                                                                                                                                                                                   |    0.000 |
| 124684305 | s51434          | 10.68.23.154:59405 | s51434__mixnmatch_p                   | Query       |   13344 | Waiting for table level lock                                          | UPDATE entry SET q=56214481,user=4,`timestamp`='20180907055407' WHERE id=18288222 AND (user is null or user=0 or q=-1)                                                                                                                                                |    0.000 |
| 124701810 | s51434          | 10.68.22.236:35021 | s51434__mixnmatch_p                   | Query       |   11667 | Waiting for table level lock                                          | UPDATE autoscrape SET status='RUNNING' WHERE id=986                                                                                                                                                                                                                   |    0.000 |
| 124725885 | s51434          | 10.68.22.236:35383 | s51434__mixnmatch_p                   | Query       |    9913 | Waiting for table level lock                                          | UPDATE entry SET q=21406824 WHERE id=32973067 AND q=33681648                                                                                                                                                                                                          |    0.000 |
| 124727354 | s51434          | 10.68.21.65:52569  | s51434__mixnmatch_p                   | Query       |    9671 | Waiting for table metadata lock                                       | TRUNCATE common_names_taxon                                                                                                                                                                                                                                           |    0.000 |
| 124745072 | s51434          | 10.68.23.172:48834 | s51434__mixnmatch_p                   | Query       |    8068 | Waiting for table level lock                                          | UPDATE autoscrape SET status='RUNNING' WHERE id=986                                                                                                                                                                                                                   |    0.000 |
| 124790057 | s51434          | 10.68.19.24:57973  | s51434__mixnmatch_p                   | Query       |    4467 | Waiting for table level lock                                          | UPDATE autoscrape SET status='RUNNING' WHERE id=986                                                                                                                                                                                                                   |    0.000 |
| 124790064 | s51434          | 10.68.23.223:55194 | s51434__mixnmatch_p                   | Query       |    4467 | Waiting for table level lock                                          | update catalog set type='biography' where type='unknown' AND active=1 AND (SELECT count(*) from entry WHERE catalog=catalog.id)=(SELECT count(*) from entry WHERE catalog=catalog.id AND type='person')                                                               |    0.000 |
| 124821891 | s51434          | 10.68.18.137:45024 | s51434__mixnmatch_p                   | Query       |    2509 | Waiting for table level lock                                          | UPDATE entry SET q=29272560,user=4,`timestamp`='20180907085442' WHERE id=41487507 AND (user is null or user=0 or q=-1)                                                                                                                                                |    0.000 |
| 124843304 | s51434          | 10.68.17.239:50277 | s51434__mixnmatch_p                   | Query       |     867 | Waiting for table level lock                                          | UPDATE autoscrape SET status='RUNNING' WHERE id=986                                                                                                                                                                                                                   |    0.000 |
| 124855022 | s51434          | 10.68.17.202:36287 | s51434__mixnmatch_p                   | Query       |     143 | Waiting for table level lock                                          | UPDATE entry SET q=8841371,user=4,`timestamp`='20180907093408' WHERE id=53201367 AND (user is null or user=0 or q=-1)                                                                                                                                                 |    0.000 |
95 rows in set (0.00 sec)

This is mostly only creating issues to itself, however this could impact somehow other users as it is using more and more resources from the host.
If this keeps growing I will set a connection limit for this user to make sure it doesn't keep increasing

Event Timeline

I have seen this a few times, but I'm not exactly sure where it comes from.

The "table lock" could stem from the daily mysqldump I do (after an unfortunate mishap with the database that took me weeks to fix).

There are also some queries constructed from user input that can be long-running, but it is hard to determine which ones would run long in advance. I would use a query timeout limit, but AFAIK that's not supported on Labs current MariaDB.

To insist, this does not create any issues on the infrastructure, was filed just FYI (as it may create issues for your own tool users), so we believe you would like to know.

Upgrade to 10.1 (supporting query limit) is scheduled to happen as soon as we resolve some ongoing issues with new hardware for toolsdbs.

I am closing this as this is not happening anymore and the whole purpose of this task was to get it recorded somewhere, in case this becomes a problem or something.