Page MenuHomePhabricator
Create Task
Maniphest T188205

s52467__new_hashtags seems to be overloading the toolsdb host
Closed, ResolvedPublic
Actions

Description

It is creating hundreds of connections, apparently blocked by themselves:

root@labsdb1005[(none)]> pager grep s52467
PAGER set to 'grep s52467'
root@labsdb1005[(none)]> SHOW PROCESSLIST;
| 121381029 | s52467          | 10.68.18.53:33642  | s52467__new_hashtags               | Execute     |     422 | Copying to tmp table                                                  | SELECT *
| 121389046 | s52467          | 10.68.18.53:33719  | s52467__new_hashtags               | Execute     |     422 | Copying to tmp table                                                  | SELECT *
| 121399214 | s52467          | 10.68.18.53:33894  | s52467__new_hashtags               | Execute     |     415 | Sending data                                                          | SELECT COUNT(*) as revisions,
| 121403531 | s52467          | 10.68.18.53:33994  | s52467__new_hashtags               | Execute     |     427 | Copying to tmp table                                                  | SELECT *
| 121406342 | s52467          | 10.68.18.53:34025  | s52467__new_hashtags               | Execute     |     882 | Copying to tmp table                                                  | SELECT *
| 121413676 | s52467          | 10.68.18.53:34254  | s52467__new_hashtags               | Execute     |     421 | Copying to tmp table                                                  | SELECT *
| 121417695 | s52467          | 10.68.18.53:34320  | s52467__new_hashtags               | Execute     |     432 | Copying to tmp table                                                  | SELECT *
| 121419846 | s52467          | 10.68.18.53:34425  | s52467__new_hashtags               | Execute     |     412 | Waiting for table level lock                                          | SELECT COUNT(*) as revisions,
| 121425665 | s52467          | 10.68.18.53:34728  | s52467__new_hashtags               | Execute     |     414 | Copying to tmp table on disk                                          | SELECT COUNT(*) as revisions,
| 121436565 | s52467          | 10.68.18.53:35237  | s52467__new_hashtags               | Execute     |     412 | Waiting for table level lock                                          | SELECT *
| 121618868 | s52467          | 10.68.23.142:33417 | s52467__new_hashtags               | Execute     |    1917 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121628850 | s52467          | 10.68.23.166:43146 | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121648966 | s52467          | 10.68.18.17:50903  | s52467__new_hashtags               | Execute     |    5137 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121658430 | s52467          | 10.68.16.94:44973  | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 121673010 | s52467          | 10.68.20.54:50605  | s52467__new_hashtags               | Execute     |    7580 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121690451 | s52467          | 10.68.18.14:55023  | s52467__new_hashtags               | Execute     |    4098 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121699311 | s52467          | 10.68.19.159:37528 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121710365 | s52467          | 10.68.18.17:55564  | s52467__new_hashtags               | Execute     |    9093 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121722327 | s52467          | 10.68.16.126:34072 | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121777281 | s52467          | 10.68.23.142:38477 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 121785180 | s52467          | 10.68.18.208:43620 | s52467__new_hashtags               | Execute     |    8541 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121794828 | s52467          | 10.68.20.54:54932  | s52467__new_hashtags               | Execute     |    6989 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121803328 | s52467          | 10.68.19.24:42572  | s52467__new_hashtags               | Execute     |    8200 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121817764 | s52467          | 10.68.23.223:39178 | s52467__new_hashtags               | Execute     |    3900 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121827137 | s52467          | 10.68.19.24:43906  | s52467__new_hashtags               | Execute     |    3309 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121840786 | s52467          | 10.68.16.126:54900 | s52467__new_hashtags               | Execute     |    6989 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121860017 | s52467          | 10.68.23.178:53077 | s52467__new_hashtags               | Execute     |    4870 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121867967 | s52467          | 10.68.23.103:60427 | s52467__new_hashtags               | Execute     |    4098 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121882236 | s52467          | 10.68.23.172:44435 | s52467__new_hashtags               | Execute     |     413 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 121891572 | s52467          | 10.68.22.85:48992  | s52467__new_hashtags               | Execute     |    7580 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121899247 | s52467          | 10.68.23.178:57198 | s52467__new_hashtags               | Execute     |    4098 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121916762 | s52467          | 10.68.18.12:37522  | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121939724 | s52467          | 10.68.23.223:44449 | s52467__new_hashtags               | Execute     |    8200 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121949294 | s52467          | 10.68.22.236:40358 | s52467__new_hashtags               | Execute     |    6050 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121957047 | s52467          | 10.68.23.103:60679 | s52467__new_hashtags               | Execute     |    5137 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121961204 | s52467          | 10.68.23.142:60086 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 121964260 | s52467          | 10.68.21.65:36795  | s52467__new_hashtags               | Execute     |    6473 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121972336 | s52467          | 10.68.18.16:47010  | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121973886 | s52467          | 10.68.23.142:60615 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 121979930 | s52467          | 10.68.18.3:38318   | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | INSERT INTO hashtags
| 121992166 | s52467          | 10.68.18.208:56606 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 121996645 | s52467          | 10.68.17.202:45423 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | INSERT INTO hashtags
| 122007726 | s52467          | 10.68.18.9:56049   | s52467__new_hashtags               | Execute     |    8541 | Waiting for table level lock                                          | INSERT INTO hashtags
| 122007766 | s52467          | 10.68.21.65:41016  | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122007868 | s52467          | 10.68.23.154:33682 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122021500 | s52467          | 10.68.22.236:41006 | s52467__new_hashtags               | Execute     |    8200 | Waiting for table level lock                                          | INSERT INTO hashtags
| 122026074 | s52467          | 10.68.21.151:38514 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122029228 | s52467          | 10.68.23.14:50278  | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122037566 | s52467          | 10.68.17.205:52382 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122039100 | s52467          | 10.68.23.14:50613  | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122039107 | s52467          | 10.68.23.172:54957 | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122045359 | s52467          | 10.68.17.205:52648 | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122055267 | s52467          | 10.68.18.3:42913   | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122055281 | s52467          | 10.68.18.18:53739  | s52467__new_hashtags               | Execute     |    3650 | Waiting for table level lock                                          | UPDATE hashtags
| 122057322 | s52467          | 10.68.18.83:54480  | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122068230 | s52467          | 10.68.21.42:40468  | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | INSERT INTO hashtags
| 122068264 | s52467          | 10.68.17.202:49210 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122068271 | s52467          | 10.68.20.54:48479  | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122069254 | s52467          | 10.68.21.65:58611  | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122076518 | s52467          | 10.68.20.54:48845  | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122079984 | s52467          | 10.68.23.142:46849 | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122080612 | s52467          | 10.68.23.172:57493 | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122080630 | s52467          | 10.68.21.151:51464 | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122083807 | s52467          | 10.68.18.208:33802 | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122093449 | s52467          | 10.68.21.65:60035  | s52467__new_hashtags               | Execute     |    4870 | Waiting for table level lock                                          | UPDATE hashtags
| 122093451 | s52467          | 10.68.16.94:48556  | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122095591 | s52467          | 10.68.17.202:50788 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122101369 | s52467          | 10.68.18.137:38635 | s52467__new_hashtags               | Execute     |    4870 | Waiting for table level lock                                          | UPDATE hashtags
| 122113561 | s52467          | 10.68.23.223:55746 | s52467__new_hashtags               | Execute     |    4870 | Waiting for table level lock                                          | UPDATE hashtags
| 122124704 | s52467          | 10.68.23.172:58857 | s52467__new_hashtags               | Execute     |    9093 | Waiting for table level lock                                          | UPDATE hashtags
| 122131372 | s52467          | 10.68.21.151:36919 | s52467__new_hashtags               | Execute     |    4555 | Waiting for table level lock                                          | UPDATE hashtags
| 122138466 | s52467          | 10.68.18.13:40904  | s52467__new_hashtags               | Execute     |    6473 | Waiting for table level lock                                          | UPDATE hashtags
| 122146376 | s52467          | 10.68.16.151:56340 | s52467__new_hashtags               | Execute     |    4870 | Waiting for table level lock                                          | UPDATE hashtags
| 122146518 | s52467          | 10.68.23.178:44878 | s52467__new_hashtags               | Execute     |    6473 | Waiting for table level lock                                          | UPDATE hashtags
| 122153565 | s52467          | 10.68.17.202:55716 | s52467__new_hashtags               | Execute     |    4870 | Waiting for table level lock                                          | UPDATE hashtags
| 122156845 | s52467          | 10.68.21.65:44319  | s52467__new_hashtags               | Execute     |    4870 | Waiting for table level lock                                          | UPDATE hashtags
| 122167467 | s52467          | 10.68.21.65:44401  | s52467__new_hashtags               | Execute     |    5137 | Waiting for table level lock                                          | UPDATE hashtags
| 122175834 | s52467          | 10.68.21.151:46443 | s52467__new_hashtags               | Execute     |    4555 | Waiting for table level lock                                          | UPDATE hashtags
| 122175889 | s52467          | 10.68.20.251:50778 | s52467__new_hashtags               | Execute     |    4870 | Waiting for table level lock                                          | UPDATE hashtags
| 122175952 | s52467          | 10.68.16.94:34150  | s52467__new_hashtags               | Execute     |    4555 | Waiting for table level lock                                          | UPDATE hashtags
| 122182759 | s52467          | 10.68.16.126:59221 | s52467__new_hashtags               | Execute     |    4098 | Waiting for table level lock                                          | UPDATE hashtags
| 122185868 | s52467          | 10.68.18.137:43756 | s52467__new_hashtags               | Execute     |    3900 | Waiting for table level lock                                          | UPDATE hashtags
| 122186438 | s52467          | 10.68.18.13:46708  | s52467__new_hashtags               | Execute     |    3900 | Waiting for table level lock                                          | UPDATE hashtags
| 122189168 | s52467          | 10.68.23.172:48175 | s52467__new_hashtags               | Execute     |    4098 | Waiting for table level lock                                          | UPDATE hashtags
| 122199423 | s52467          | 10.68.23.172:53105 | s52467__new_hashtags               | Execute     |    4870 | Waiting for table level lock                                          | UPDATE hashtags
| 122199583 | s52467          | 10.68.18.13:47073  | s52467__new_hashtags               | Execute     |    3900 | Waiting for table level lock                                          | UPDATE hashtags
| 122203815 | s52467          | 10.68.21.151:47846 | s52467__new_hashtags               | Execute     |    4555 | Waiting for table level lock                                          | UPDATE hashtags
| 122213066 | s52467          | 10.68.22.85:58629  | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122216459 | s52467          | 10.68.16.126:36868 | s52467__new_hashtags               | Execute     |    1917 | Waiting for table level lock                                          | UPDATE hashtags
| 122222108 | s52467          | 10.68.21.65:45838  | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122224876 | s52467          | 10.68.23.103:60917 | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122239004 | s52467          | 10.68.18.14:58383  | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122239125 | s52467          | 10.68.17.239:56208 | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122239215 | s52467          | 10.68.18.3:56508   | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122248152 | s52467          | 10.68.18.3:56796   | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122253048 | s52467          | 10.68.22.236:51137 | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122254065 | s52467          | 10.68.18.137:46595 | s52467__new_hashtags               | Execute     |    1917 | Waiting for table level lock                                          | UPDATE hashtags
| 122258546 | s52467          | 10.68.21.151:51632 | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122266486 | s52467          | 10.68.21.151:51964 | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122266517 | s52467          | 10.68.23.14:33846  | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122273691 | s52467          | 10.68.23.172:33087 | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122277961 | s52467          | 10.68.16.94:41061  | s52467__new_hashtags               | Execute     |    9093 | Waiting for table level lock                                          | UPDATE hashtags
| 122284679 | s52467          | 10.68.17.239:56789 | s52467__new_hashtags               | Execute     |     413 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122294763 | s52467          | 10.68.23.142:59006 | s52467__new_hashtags               | Execute     |    8541 | Waiting for table level lock                                          | UPDATE hashtags
| 122294797 | s52467          | 10.68.18.14:59053  | s52467__new_hashtags               | Execute     |     413 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122294993 | s52467          | 10.68.23.154:38720 | s52467__new_hashtags               | Execute     |    8541 | Waiting for table level lock                                          | UPDATE hashtags
| 122295108 | s52467          | 10.68.18.3:60000   | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122301699 | s52467          | 10.68.21.65:54555  | s52467__new_hashtags               | Execute     |    7580 | Waiting for table level lock                                          | UPDATE hashtags
| 122306063 | s52467          | 10.68.18.16:35245  | s52467__new_hashtags               | Execute     |    6050 | Waiting for table level lock                                          | UPDATE hashtags
| 122306782 | s52467          | 10.68.17.202:37631 | s52467__new_hashtags               | Execute     |    8200 | Waiting for table level lock                                          | UPDATE hashtags
| 122310001 | s52467          | 10.68.19.24:59850  | s52467__new_hashtags               | Execute     |    7580 | Waiting for table level lock                                          | UPDATE hashtags
| 122310657 | s52467          | 10.68.16.94:52576  | s52467__new_hashtags               | Execute     |    7580 | Waiting for table level lock                                          | UPDATE hashtags
| 122318350 | s52467          | 10.68.19.160:53272 | s52467__new_hashtags               | Execute     |    7580 | Waiting for table level lock                                          | UPDATE hashtags
| 122318351 | s52467          | 10.68.18.17:34221  | s52467__new_hashtags               | Execute     |    5137 | Waiting for table level lock                                          | UPDATE hashtags
| 122320003 | s52467          | 10.68.21.151:33572 | s52467__new_hashtags               | Execute     |    6473 | Waiting for table level lock                                          | UPDATE hashtags
| 122326363 | s52467          | 10.68.18.17:34638  | s52467__new_hashtags               | Execute     |    6473 | Waiting for table level lock                                          | UPDATE hashtags
| 122326692 | s52467          | 10.68.21.65:59489  | s52467__new_hashtags               | Execute     |    5137 | Waiting for table level lock                                          | UPDATE hashtags
| 122329952 | s52467          | 10.68.17.239:57240 | s52467__new_hashtags               | Execute     |    6473 | Waiting for table level lock                                          | UPDATE hashtags
| 122339205 | s52467          | 10.68.21.65:33802  | s52467__new_hashtags               | Execute     |    5544 | Waiting for table level lock                                          | UPDATE hashtags
| 122349146 | s52467          | 10.68.23.103:37897 | s52467__new_hashtags               | Execute     |    5544 | Waiting for table level lock                                          | UPDATE hashtags
| 122349149 | s52467          | 10.68.18.17:35502  | s52467__new_hashtags               | Execute     |    5137 | Waiting for table level lock                                          | UPDATE hashtags
| 122357761 | s52467          | 10.68.16.94:56741  | s52467__new_hashtags               | Execute     |    4555 | Waiting for table level lock                                          | UPDATE hashtags
| 122361401 | s52467          | 10.68.23.172:40815 | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122361410 | s52467          | 10.68.21.151:36240 | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122361433 | s52467          | 10.68.21.151:36243 | s52467__new_hashtags               | Execute     |    3309 | Waiting for table level lock                                          | UPDATE hashtags
| 122365715 | s52467          | 10.68.22.85:42769  | s52467__new_hashtags               | Execute     |    3309 | Waiting for table level lock                                          | UPDATE hashtags
| 122365788 | s52467          | 10.68.16.126:52293 | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122376412 | s52467          | 10.68.16.151:36404 | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122376415 | s52467          | 10.68.18.208:59371 | s52467__new_hashtags               | Execute     |    1917 | Waiting for table level lock                                          | UPDATE hashtags
| 122376417 | s52467          | 10.68.22.85:43129  | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122376592 | s52467          | 10.68.21.65:39382  | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122379578 | s52467          | 10.68.18.205:58344 | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122389826 | s52467          | 10.68.20.54:43643  | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122395215 | s52467          | 10.68.17.209:34119 | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122395223 | s52467          | 10.68.23.223:45417 | s52467__new_hashtags               | Execute     |    1917 | Waiting for table level lock                                          | UPDATE hashtags
| 122395258 | s52467          | 10.68.18.205:58413 | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122403878 | s52467          | 10.68.18.13:38699  | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122407016 | s52467          | 10.68.23.142:42809 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | UPDATE hashtags
| 122424159 | s52467          | 10.68.23.178:34702 | s52467__new_hashtags               | Execute     |     883 | Waiting for table level lock                                          | UPDATE hashtags
| 122424522 | s52467          | 10.68.23.223:46173 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | UPDATE hashtags
| 122424650 | s52467          | 10.68.23.103:51389 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | INSERT INTO hashtags
| 122424837 | s52467          | 10.68.18.205:58478 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | UPDATE hashtags
| 122425056 | s52467          | 10.68.19.159:40579 | s52467__new_hashtags               | Execute     |     883 | Waiting for table level lock                                          | UPDATE hashtags
| 122443957 | s52467          | 10.68.22.85:50299  | s52467__new_hashtags               | Execute     |     413 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122447918 | s52467          | 10.68.18.13:55569  | s52467__new_hashtags               | Execute     |    8541 | Waiting for table level lock                                          | UPDATE hashtags
| 122447936 | s52467          | 10.68.18.13:55582  | s52467__new_hashtags               | Execute     |    8541 | Waiting for table level lock                                          | UPDATE hashtags
| 122447945 | s52467          | 10.68.18.208:59759 | s52467__new_hashtags               | Execute     |    8541 | Waiting for table level lock                                          | UPDATE hashtags
| 122447964 | s52467          | 10.68.18.83:41760  | s52467__new_hashtags               | Execute     |    8541 | Waiting for table level lock                                          | UPDATE hashtags
| 122448757 | s52467          | 10.68.19.160:60578 | s52467__new_hashtags               | Execute     |    8541 | Waiting for table level lock                                          | UPDATE hashtags
| 122448821 | s52467          | 10.68.20.54:46326  | s52467__new_hashtags               | Execute     |    8541 | Waiting for table level lock                                          | UPDATE hashtags
| 122452723 | s52467          | 10.68.18.13:58703  | s52467__new_hashtags               | Execute     |    8541 | Waiting for table level lock                                          | UPDATE hashtags
| 122461529 | s52467          | 10.68.18.205:58672 | s52467__new_hashtags               | Execute     |    7580 | Waiting for table level lock                                          | UPDATE hashtags
| 122461589 | s52467          | 10.68.18.137:55525 | s52467__new_hashtags               | Execute     |    7580 | Waiting for table level lock                                          | UPDATE hashtags
| 122461622 | s52467          | 10.68.16.126:56500 | s52467__new_hashtags               | Execute     |    7580 | Waiting for table level lock                                          | UPDATE hashtags
| 122461626 | s52467          | 10.68.18.18:34365  | s52467__new_hashtags               | Execute     |    7580 | Waiting for table level lock                                          | UPDATE hashtags
| 122463325 | s52467          | 10.68.17.205:43054 | s52467__new_hashtags               | Execute     |    7580 | Waiting for table level lock                                          | UPDATE hashtags
| 122463337 | s52467          | 10.68.23.103:40983 | s52467__new_hashtags               | Execute     |    7580 | Waiting for table level lock                                          | UPDATE hashtags
| 122463348 | s52467          | 10.68.23.223:51138 | s52467__new_hashtags               | Execute     |    7580 | Waiting for table level lock                                          | UPDATE hashtags
| 122464668 | s52467          | 10.68.18.13:37017  | s52467__new_hashtags               | Execute     |    7580 | Waiting for table level lock                                          | UPDATE hashtags
| 122470202 | s52467          | 10.68.18.3:53721   | s52467__new_hashtags               | Execute     |    6990 | Waiting for table level lock                                          | UPDATE hashtags
| 122471749 | s52467          | 10.68.23.172:52797 | s52467__new_hashtags               | Execute     |    6989 | Waiting for table level lock                                          | UPDATE hashtags
| 122474037 | s52467          | 10.68.23.14:33977  | s52467__new_hashtags               | Execute     |    6990 | Waiting for table level lock                                          | UPDATE hashtags
| 122474061 | s52467          | 10.68.19.160:33073 | s52467__new_hashtags               | Execute     |    6989 | Waiting for table level lock                                          | UPDATE hashtags
| 122474069 | s52467          | 10.68.23.166:57156 | s52467__new_hashtags               | Execute     |    6989 | Waiting for table level lock                                          | UPDATE hashtags
| 122480213 | s52467          | 10.68.16.126:57631 | s52467__new_hashtags               | Execute     |    6990 | Waiting for table level lock                                          | UPDATE hashtags
| 122480253 | s52467          | 10.68.18.83:49504  | s52467__new_hashtags               | Execute     |    6990 | Waiting for table level lock                                          | UPDATE hashtags
| 122480286 | s52467          | 10.68.23.178:44798 | s52467__new_hashtags               | Execute     |    6989 | Waiting for table level lock                                          | UPDATE hashtags
| 122483081 | s52467          | 10.68.23.223:52240 | s52467__new_hashtags               | Execute     |    6473 | Waiting for table level lock                                          | UPDATE hashtags
| 122497064 | s52467          | 10.68.19.24:54567  | s52467__new_hashtags               | Execute     |    6050 | Waiting for table level lock                                          | UPDATE hashtags
| 122497068 | s52467          | 10.68.23.223:52465 | s52467__new_hashtags               | Execute     |    6050 | Waiting for table level lock                                          | UPDATE hashtags
| 122497413 | s52467          | 10.68.23.142:53189 | s52467__new_hashtags               | Execute     |    6050 | Waiting for table level lock                                          | UPDATE hashtags
| 122497425 | s52467          | 10.68.19.24:54766  | s52467__new_hashtags               | Execute     |    6050 | Waiting for table level lock                                          | UPDATE hashtags
| 122497433 | s52467          | 10.68.18.3:58986   | s52467__new_hashtags               | Execute     |    6050 | Waiting for table level lock                                          | UPDATE hashtags
| 122497441 | s52467          | 10.68.23.223:52471 | s52467__new_hashtags               | Execute     |    6050 | Waiting for table level lock                                          | UPDATE hashtags
| 122497492 | s52467          | 10.68.18.83:51385  | s52467__new_hashtags               | Execute     |    6050 | Waiting for table level lock                                          | UPDATE hashtags
| 122497500 | s52467          | 10.68.22.236:57778 | s52467__new_hashtags               | Execute     |    6050 | Waiting for table level lock                                          | UPDATE hashtags
| 122497705 | s52467          | 10.68.22.85:59085  | s52467__new_hashtags               | Execute     |    6050 | Waiting for table level lock                                          | UPDATE hashtags
| 122505566 | s52467          | 10.68.19.160:33810 | s52467__new_hashtags               | Execute     |    5544 | Waiting for table level lock                                          | UPDATE hashtags
| 122510014 | s52467          | 10.68.23.172:60041 | s52467__new_hashtags               | Execute     |    5137 | Waiting for table level lock                                          | UPDATE hashtags
| 122510045 | s52467          | 10.68.20.54:52932  | s52467__new_hashtags               | Execute     |    5137 | Waiting for table level lock                                          | UPDATE hashtags
| 122510049 | s52467          | 10.68.23.14:41931  | s52467__new_hashtags               | Execute     |    5137 | Waiting for table level lock                                          | UPDATE hashtags
| 122510790 | s52467          | 10.68.22.236:58038 | s52467__new_hashtags               | Execute     |    5137 | Waiting for table level lock                                          | UPDATE hashtags
| 122510794 | s52467          | 10.68.22.85:60573  | s52467__new_hashtags               | Execute     |    5137 | Waiting for table level lock                                          | UPDATE hashtags
| 122510798 | s52467          | 10.68.16.151:37466 | s52467__new_hashtags               | Execute     |    5137 | Waiting for table level lock                                          | UPDATE hashtags
| 122510821 | s52467          | 10.68.16.94:34493  | s52467__new_hashtags               | Execute     |    5137 | Waiting for table level lock                                          | UPDATE hashtags
| 122511457 | s52467          | 10.68.23.14:42734  | s52467__new_hashtags               | Execute     |    5137 | Waiting for table level lock                                          | UPDATE hashtags
| 122515375 | s52467          | 10.68.23.172:60145 | s52467__new_hashtags               | Execute     |    4870 | Waiting for table level lock                                          | UPDATE hashtags
| 122524004 | s52467          | 10.68.19.24:39684  | s52467__new_hashtags               | Execute     |    4098 | Waiting for table level lock                                          | UPDATE hashtags
| 122524034 | s52467          | 10.68.21.65:53913  | s52467__new_hashtags               | Execute     |    4098 | Waiting for table level lock                                          | UPDATE hashtags
| 122524091 | s52467          | 10.68.19.24:39731  | s52467__new_hashtags               | Execute     |    4098 | Waiting for table level lock                                          | UPDATE hashtags
| 122524134 | s52467          | 10.68.23.178:58307 | s52467__new_hashtags               | Execute     |    4098 | Waiting for table level lock                                          | UPDATE hashtags
| 122525994 | s52467          | 10.68.18.3:34183   | s52467__new_hashtags               | Execute     |    4098 | Waiting for table level lock                                          | UPDATE hashtags
| 122526021 | s52467          | 10.68.19.159:42504 | s52467__new_hashtags               | Execute     |    4098 | Waiting for table level lock                                          | UPDATE hashtags
| 122526048 | s52467          | 10.68.23.14:43905  | s52467__new_hashtags               | Execute     |    4098 | Waiting for table level lock                                          | UPDATE hashtags
| 122526056 | s52467          | 10.68.16.126:33990 | s52467__new_hashtags               | Execute     |    4098 | Waiting for table level lock                                          | UPDATE hashtags
| 122533598 | s52467          | 10.68.23.178:58550 | s52467__new_hashtags               | Execute     |    3900 | Waiting for table level lock                                          | UPDATE hashtags
| 122535356 | s52467          | 10.68.23.103:45474 | s52467__new_hashtags               | Execute     |    3651 | Waiting for table level lock                                          | UPDATE hashtags
| 122538014 | s52467          | 10.68.17.239:35866 | s52467__new_hashtags               | Execute     |    3309 | Waiting for table level lock                                          | UPDATE hashtags
| 122538016 | s52467          | 10.68.18.137:57713 | s52467__new_hashtags               | Execute     |    3309 | Waiting for table level lock                                          | UPDATE hashtags
| 122538026 | s52467          | 10.68.23.166:36241 | s52467__new_hashtags               | Execute     |    3305 | Waiting for table level lock                                          | UPDATE hashtags
| 122545496 | s52467          | 10.68.18.3:35750   | s52467__new_hashtags               | Execute     |    3309 | Waiting for table level lock                                          | UPDATE hashtags
| 122545509 | s52467          | 10.68.19.24:49069  | s52467__new_hashtags               | Execute     |    3309 | Waiting for table level lock                                          | UPDATE hashtags
| 122545510 | s52467          | 10.68.16.126:34767 | s52467__new_hashtags               | Execute     |    3309 | Waiting for table level lock                                          | UPDATE hashtags
| 122545515 | s52467          | 10.68.21.65:54285  | s52467__new_hashtags               | Execute     |    3309 | Waiting for table level lock                                          | UPDATE hashtags
| 122548129 | s52467          | 10.68.23.142:57648 | s52467__new_hashtags               | Execute     |    2928 | Waiting for table level lock                                          | UPDATE hashtags
| 122560067 | s52467          | 10.68.22.236:58796 | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122560083 | s52467          | 10.68.18.205:60157 | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122560200 | s52467          | 10.68.18.13:33283  | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122560207 | s52467          | 10.68.21.151:39259 | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122560216 | s52467          | 10.68.16.94:37719  | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122560255 | s52467          | 10.68.23.172:60484 | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122560283 | s52467          | 10.68.18.83:36148  | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122560292 | s52467          | 10.68.23.172:60487 | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122560402 | s52467          | 10.68.18.3:36268   | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122560747 | s52467          | 10.68.17.202:43720 | s52467__new_hashtags               | Execute     |    2489 | Waiting for table level lock                                          | UPDATE hashtags
| 122573343 | s52467          | 10.68.18.83:38805  | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122573345 | s52467          | 10.68.18.3:36868   | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122573347 | s52467          | 10.68.17.209:43685 | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122573421 | s52467          | 10.68.16.126:36785 | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122574144 | s52467          | 10.68.17.239:36653 | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122574160 | s52467          | 10.68.23.14:50876  | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122574175 | s52467          | 10.68.23.154:34270 | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122577755 | s52467          | 10.68.17.209:43799 | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122578594 | s52467          | 10.68.17.239:36700 | s52467__new_hashtags               | Execute     |    1349 | Waiting for table level lock                                          | UPDATE hashtags
| 122587914 | s52467          | 10.68.23.14:51382  | s52467__new_hashtags               | Execute     |     883 | Waiting for table level lock                                          | UPDATE hashtags
| 122587993 | s52467          | 10.68.16.94:41638  | s52467__new_hashtags               | Execute     |     883 | Waiting for table level lock                                          | UPDATE hashtags
| 122588022 | s52467          | 10.68.18.208:36116 | s52467__new_hashtags               | Execute     |     883 | Waiting for table level lock                                          | UPDATE hashtags
| 122588040 | s52467          | 10.68.21.169:42565 | s52467__new_hashtags               | Execute     |     883 | Waiting for table level lock                                          | UPDATE hashtags
| 122588058 | s52467          | 10.68.18.17:50334  | s52467__new_hashtags               | Execute     |     883 | Waiting for table level lock                                          | UPDATE hashtags
| 122588072 | s52467          | 10.68.16.126:37566 | s52467__new_hashtags               | Execute     |     883 | Waiting for table level lock                                          | UPDATE hashtags
| 122590054 | s52467          | 10.68.17.205:47501 | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | UPDATE hashtags
| 122590060 | s52467          | 10.68.22.85:41193  | s52467__new_hashtags               | Execute     |     414 | Waiting for table level lock                                          | UPDATE hashtags
| 122598801 | s52467          | 10.68.23.223:55073 | s52467__new_hashtags               | Execute     |     305 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122604173 | s52467          | 10.68.17.205:47944 | s52467__new_hashtags               | Execute     |      33 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122604182 | s52467          | 10.68.16.94:44369  | s52467__new_hashtags               | Execute     |      32 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122604188 | s52467          | 10.68.18.83:44652  | s52467__new_hashtags               | Execute     |      32 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
| 122604200 | s52467          | 10.68.18.13:33307  | s52467__new_hashtags               | Execute     |      31 | Waiting for table level lock                                          | SELECT ht_id, ht_update_timestamp
350 rows in set (0.00 sec)

I am going to limit the user to 10 connections, as otherwise the whole db will start failing.

Related Objects
Search...

Event Timeline

jcrespo created this task.Feb 25 2018, 12:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 25 2018, 12:50 PM
jcrespo added a comment.Feb 25 2018, 3:21 PM

I lowered max connections to 5, and setup a query killer on a screen.

Here you can clearly see the overload:

https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=labsdb1005&var-port=9104&panelId=37&fullscreen&from=1519399218979&to=1519572018980

jcrespo added a parent task: T119601: Certain tools users create multiple long running queries that take all memory and/or CPU from labsdb hosts, slowing it down and potentially crashing (tracking).Feb 26 2018, 4:57 PM
bd808 assigned this task to mahmoud.Feb 27 2018, 12:55 AM
bd808 triaged this task as High priority.
bd808 edited projects, added Hashtags; removed Tools.
mahmoud added a comment.Feb 27 2018, 8:31 AM

Yeah, sorry about that. This was due to us attempting to re-enable the hashtag search on IFTTT. Turns out it's a very popular feature! Too popular unfortunately. We've wired that off again, so if we can go back to having a regular limit that would be nice.

While I was at it, I added a jitter feature to our regularly scheduled jobs to hopefully help smooth out some of the load. Also, looking at the process list above, it doesn't seem like it'll help, but I did switch some of our lang wiki db queries to go to the analytics host.

Thanks!

jcrespo closed this task as Resolved.Feb 27 2018, 9:27 AM

I have removed all limits.

Turns out it's a very popular feature! Too popular unfortunately.

Note I don't have any problem towards that feature, my only job here is to make sure all users have enough resources- I would recommend you to ask for dedicated resources to attend your needs for that volumne.

However, the queries seem to be blocked between them- you may want to check the alters, writes and reads on your application so they do not block each other.

jcrespo reopened this task as Open.EditedAug 8 2018, 1:34 PM

Banning use for heavy usage after users' complains on IRC.

I would recommend you to follow my initial recommendation, or doing self-limiting/connection pooling:

I would recommend you to ask for dedicated resources to attend your needs for that volume.

Multiple >12 300+ second queries with such impact causes cpu starvation, affecting other users. Please request separate resources if you want to continue doing such heavy querying.

jcrespo added a comment.Aug 8 2018, 1:35 PM
| 87611024 | s52467          | 10.68.18.52:39429  | s52467__new_hashtags               | Execute     |     389 | Copying to tmp table
        FROM recentchanges AS rc
        JOIN hashtag_recentchanges AS htrc
        ON htrc.htrc_id = rc.htrc_id
        JOIN hashtags AS ht
        ON ht.ht_id = htrc.ht_id
        WHERE ht.ht_text = ?
        AND rc.htrc_lang LIKE ?
        AND rc.rc_timestamp BETWEEN ? AND ?
        ORDER BY rc.rc_timestamp DESC
        LIMIT ?, ?   |    0.000 |
| 87611025 | s52467          | 10.68.18.52:39430  | s52467__new_hashtags               | Execute     |     327 | Copying to tmp table
        FROM recentchanges AS rc
        JOIN hashtag_recentchanges AS htrc
        ON htrc.htrc_id = rc.htrc_id
        JOIN hashtags AS ht
        ON ht.ht_id = htrc.ht_id
        WHERE ht.ht_text = ?
        AND rc.htrc_lang LIKE ?
        AND rc.rc_timestamp BETWEEN ? AND ?
        ORDER BY rc.rc_timestamp DESC
        LIMIT ?, ?   |    0.000 |
| 87611026 | s52467          | 10.68.18.52:39431  | s52467__new_hashtags               | Execute     |     389 | Copying to tmp table
        FROM recentchanges AS rc
        JOIN hashtag_recentchanges AS htrc
        ON htrc.htrc_id = rc.htrc_id
        JOIN hashtags AS ht
        ON ht.ht_id = htrc.ht_id
        WHERE ht.ht_text = ?
        AND rc.htrc_lang LIKE ?
        AND rc.rc_timestamp BETWEEN ? AND ?
        ORDER BY rc.rc_timestamp DESC
        LIMIT ?, ?   |    0.000 |
| 87611027 | s52467          | 10.68.18.52:39432  | s52467__new_hashtags               | Execute     |     389 | Copying to tmp table
        FROM recentchanges AS rc
        JOIN hashtag_recentchanges AS htrc
        ON htrc.htrc_id = rc.htrc_id
        JOIN hashtags AS ht
        ON ht.ht_id = htrc.ht_id
        WHERE ht.ht_text = ?
        AND rc.htrc_lang LIKE ?
        AND rc.rc_timestamp BETWEEN ? AND ?
        ORDER BY rc.rc_timestamp DESC
        LIMIT ?, ?   |    0.000 |
| 87611028 | s52467          | 10.68.18.52:39433  | s52467__new_hashtags               | Execute     |     377 | Copying to tmp table
        FROM recentchanges AS rc
        JOIN hashtag_recentchanges AS htrc
        ON htrc.htrc_id = rc.htrc_id
        JOIN hashtags AS ht
        ON ht.ht_id = htrc.ht_id
        WHERE ht.ht_text = ?
        AND rc.htrc_lang LIKE ?
        AND rc.rc_timestamp BETWEEN ? AND ?
        ORDER BY rc.rc_timestamp DESC
        LIMIT ?, ?   |    0.000 |
| 87611029 | s52467          | 10.68.18.52:39434  | s52467__new_hashtags               | Execute     |     383 | Copying to tmp table
        FROM recentchanges AS rc
        JOIN hashtag_recentchanges AS htrc
        ON htrc.htrc_id = rc.htrc_id
        JOIN hashtags AS ht
        ON ht.ht_id = htrc.ht_id
        WHERE ht.ht_text = ?
        AND rc.htrc_lang LIKE ?
        AND rc.rc_timestamp BETWEEN ? AND ?
        ORDER BY rc.rc_timestamp DESC
        LIMIT ?, ?   |    0.000 |
| 87611030 | s52467          | 10.68.18.52:39435  | s52467__new_hashtags               | Execute     |     389 | Copying to tmp table
        FROM recentchanges AS rc
        JOIN hashtag_recentchanges AS htrc
        ON htrc.htrc_id = rc.htrc_id
        JOIN hashtags AS ht
        ON ht.ht_id = htrc.ht_id
        WHERE ht.ht_text = ?
        AND rc.htrc_lang LIKE ?
        AND rc.rc_timestamp BETWEEN ? AND ?
        ORDER BY rc.rc_timestamp DESC
        LIMIT ?, ?   |    0.000 |
| 87611031 | s52467          | 10.68.18.52:39436  | s52467__new_hashtags               | Execute     |     328 | Copying to tmp table
        FROM recentchanges AS rc
        JOIN hashtag_recentchanges AS htrc
        ON htrc.htrc_id = rc.htrc_id
        JOIN hashtags AS ht
        ON ht.ht_id = htrc.ht_id
        WHERE ht.ht_text = ?
        AND rc.htrc_lang LIKE ?
        AND rc.rc_timestamp BETWEEN ? AND ?
        ORDER BY rc.rc_timestamp DESC
        LIMIT ?, ?   |    0.000 |
| 87611032 | s52467          | 10.68.18.52:39437  | s52467__new_hashtags               | Execute     |     389 | Copying to tmp table
        FROM recentchanges AS rc
        JOIN hashtag_recentchanges AS htrc
        ON htrc.htrc_id = rc.htrc_id
        JOIN hashtags AS ht
        ON ht.ht_id = htrc.ht_id
        WHERE ht.ht_text = ?
        AND rc.htrc_lang LIKE ?
        AND rc.rc_timestamp BETWEEN ? AND ?
        ORDER BY rc.rc_timestamp DESC
        LIMIT ?, ?   |    0.000 |
| 87611033 | s52467          | 10.68.18.52:39438  | s52467__new_hashtags               | Execute     |     389 | Copying to tmp table
        FROM recentchanges AS rc
        JOIN hashtag_recentchanges AS htrc
        ON htrc.htrc_id = rc.htrc_id
        JOIN hashtags AS ht
        ON ht.ht_id = htrc.ht_id
        WHERE ht.ht_text = ?
        AND rc.htrc_lang LIKE ?
        AND rc.rc_timestamp BETWEEN ? AND ?
        ORDER BY rc.rc_timestamp DESC
        LIMIT ?, ?   |    0.000 |

.... (continues)
Samwalton9-WMF mentioned this in T198707: Hashtag search tools is down.Aug 28 2018, 10:10 AM
Samwalton9-WMF added a comment.Sep 11 2018, 2:05 PM

Per the above, I've taken the tool webservice down (not that it was able to do anything without db access anyway), and removed all scheduled db tasks from the crontab.

@jcrespo - Could you reinstate db access for the tool? I'm working on a replacement (which won't be hosted on Toolforge) and would like to import the old database. The tool shouldn't be making any connections to it anymore.

jcrespo added a comment.Sep 11 2018, 2:08 PM

@Samwalton9 thank you, I had to ban the account temporarily as the overload was affecting other users quite badly- I hope you understand. Let me try to reenable it ASAP.

Samwalton9-WMF mentioned this in T204059: Request creation of hashtags VPS project.Sep 11 2018, 2:13 PM
jcrespo added a comment.Sep 11 2018, 2:13 PM
root@labsdb1005[(none)]> UPDATE mysql.user SET password=REVERSE(password) WHERE user='s5****';
Query OK, 1 row affected (0.03 sec)
Rows matched: 1  Changed: 1  Warnings: 0

root@labsdb1005[(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.09 sec)

You should have access to toolddb, please check and resolve if positive. Also ask for tips to prevent this if you want (e.g. I can setup a connection limit for you or other tips, I can help with that.

Samwalton9-WMF added a comment.Sep 11 2018, 2:14 PM

Thank you! As part of the rewrite I planned to move the tool over to a VPS project (T204059) rather than keeping it on Toolforge, to avoid future issues :)

jcrespo added a comment.Sep 11 2018, 2:17 PM

I planned to move the tool over to a VPS project

That is great!- toolsdb is a highly crowded db, and not suitable for high IOPS projects- a dedicated db will give you more resources and flexibility and you will get affected less by load and vice-versa, thank you!

jcrespo closed this task as Resolved.Sep 11 2018, 2:23 PM

The DB isn't burning in flames right now, so I would consider this fixed.

jcrespo reassigned this task from mahmoud to Samwalton9-WMF.Sep 11 2018, 2:24 PM
jcrespo added a subscriber: mahmoud.
bd808 mentioned this in T329461: Drop `s52467__new_hashtags` database from toolsdb if unused.Feb 12 2023, 7:34 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL