Page MenuHomePhabricator

[toolsdb] Transaction History Length growing too much
Open, In Progress, LowPublic

Assigned To
Authored By
fnegri
Jun 4 2026, 10:04 AM
Referenced Files
F96644067: Screenshot 2026-07-31 at 14.26.21.png
Fri, Jul 31, 12:32 PM
F96644058: Screenshot 2026-07-31 at 14.27.18.png
Fri, Jul 31, 12:32 PM
F96528830: Screenshot 2026-07-30 at 17.14.33.png
Thu, Jul 30, 3:15 PM
F96301434: Screenshot 2026-07-28 at 17.38.32.png
Tue, Jul 28, 3:40 PM
F96115202: Screenshot 2026-07-27 at 14.28.14.png
Mon, Jul 27, 12:31 PM
F95587182: Screenshot 2026-07-24 at 11.52.24.png
Fri, Jul 24, 9:53 AM
F95419195: Screenshot 2026-07-23 at 17.11.50.png
Thu, Jul 23, 3:12 PM
F94263820: Screenshot 2026-07-17 at 18.29.54.png
Fri, Jul 17, 4:35 PM

Description

While debugging T427187: ToolsDB disk space usage growing too fast I noticed the MariaDB "Transaction History Length" is growing too much:

Screenshot 2026-06-04 at 11.39.36.png (1,270×614 px, 89 KB)

This looks like a repeat of T409716: [toolsdb] ibdata1 growing on primary, the size of ibdata1 is slowly increasing:

  • 2026-05-25: 142G
  • 2026-06-04: 163G

SHOW ENGINE INNODB STATUS\G is showing some stuck queries for user s52323 (dimastbkbot):

---TRANSACTION (0x7f02997bd700), ACTIVE 3340 sec
mysql tables in use 9, locked 0
0 lock struct(s), heap size 1128, 0 row lock(s)
MariaDB thread id 599777751, OS thread handle 139614768051904, query id 25837671024 172.16.4.204 s52323 Sending data
SELECT anon_1.template_id, anon_1.page_id, anon_1.id, anon_1.value, anon_1.value_1, anon_1.namespace_id, anon_1.title, template_1.redirect_id, template_1.title AS title_1, template_1.id AS id_1, namespace_1.number, namespace_1.name, namespace_1.id AS id_2, page_1.wiki_id, page_1.namespace_id AS namespace_id_1, page_1.title AS title_2, page_1.id AS id_3, param_1.page_template_id, param_1.name AS name_1, param_1.value AS value_2, param_1.id AS id_4
FROM (SELECT page_template.template_id AS template_id, page_template.page_id AS page_id, page_template.id AS id, param_2.value AS value, param_3.value AS value_1, page.namespace_id AS namespace_id, page.title AS title
FROM page_template INNER JOIN page ON page.id = page_template.page_id INNER JOIN namespace ON namespace.id = page.namespace_id LEFT OUTER JOIN param AS param_2 ON param_2.page_template_id = page_template.id AND param_2.name = 'GTG,boxstyle_ АБС' LEFT OUTER JOIN param AS param_3 ON param_3.page_template_id = page_template.id AND param_3.name = 'Тереза'
WHERE page_template.template_id IN (1615) ORDER BY param_2.value ASC, param_3.value ASC, page.namespace_id, page.title
 LIMIT 0, 50) AS anon_1 LEFT OUTER JOIN template AS template_1 ON template_1.id = anon_1.template_id LEFT OUTER JOIN page AS page_1 ON page_1.id = anon_1.page_id LEFT OUTER JOIN namespace AS namespace_1 ON namespace_1.id = page_1.namespace_id LEFT OUTER JOIN param AS param_1 ON anon_1.id = param_1.page_template_id ORDER BY anon_1.value ASC, anon_1.value_1 ASC, anon_1.namespace_id, anon_1.title
Trx read view will not see trx with id >= 167751592329, sees < 167751068720

I tried killing these queries, but the tool is automatically spawning up new ones:

MariaDB [(none)]> SELECT id, time, command, state, host, LEFT(info,50) AS info FROM information_schema.processlist WHERE user = 's52323';
+-----------+------+---------+--------------+--------------------+----------------------------------------------------+
| id        | time | command | state        | host               | info                                               |
+-----------+------+---------+--------------+--------------------+----------------------------------------------------+
| 600018621 |    3 | Query   | Sending data | 172.16.4.204:35544 | SELECT anon_1.template_id, anon_1.page_id, anon_1. |
| 600017551 |   18 | Query   | Sending data | 172.16.4.204:52875 | SELECT anon_1.template_id, anon_1.page_id, anon_1. |
| 600008528 |  166 | Query   | Sending data | 172.16.4.204:27493 | SELECT anon_1.template_id, anon_1.page_id, anon_1. |
| 599965058 |  800 | Query   | Sending data | 172.16.4.204:18217 | SELECT anon_1.template_id, anon_1.page_id, anon_1. |
| 599929684 | 1337 | Query   | Sending data | 172.16.4.204:62641 | SELECT anon_1.template_id, anon_1.page_id, anon_1. |
| 599863260 | 2281 | Query   | Sending data | 172.16.4.204:5139  | SELECT anon_1.template_id, anon_1.page_id, anon_1. |
| 599845801 | 2546 | Query   | Sending data | 172.16.4.204:19631 | SELECT anon_1.template_id, anon_1.page_id, anon_1. |
| 599807315 | 3167 | Query   | Sending data | 172.16.4.204:18977 | SELECT anon_1.template_id, anon_1.page_id, anon_1. |
| 599793774 | 3407 | Query   | Sending data | 172.16.4.204:3757  | SELECT anon_1.template_id, anon_1.page_id, anon_1. |
+-----------+------+---------+--------------+--------------------+----------------------------------------------------+
9 rows in set (0.004 sec)

The queries are getting killed when they reach max_statement_time (1 hour), and are logged in /srv/labsdb/data/tools-db-6-slow.log. I can see similar queries in the slow query log since last year (2025-11-18) so I'm not sure why the "Transaction History Length" has only started increasing now.

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Add new alert ToolsDBHistoryLengthGrowingrepos/cloud/toolforge/alerts!64fnegriT428139main
Customize query in GitLab

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

There is something that is compounding the effect of queries from s52323 (dimasitkbot), and it's queries from s51138 (heritage):

---TRANSACTION (0x7f0299762b00), ACTIVE 4642 sec
0 lock struct(s), heap size 1128, 0 row lock(s)
MariaDB thread id 601366423, OS thread handle 139614785337024, query id 25947274120 172.16.1.96 s51138
Trx read view will not see trx with id >= 167770627708, sees < 167770624002

It's keeping transactions open and idle for hours, which is similarly preventing MariaDB from purging the undo log, leading to History Length growing.

I'm setting SET GLOBAL idle_transaction_timeout=300; as discussed in T409857: [toolsdb] Automatically terminate long transactions to see if it helps.

I can see similar queries in the slow query log since last year (2025-11-18) so I'm not sure why the "Transaction History Length" has only started increasing now.

I counted how many times the queries from s52323 appear in the slow query log, the trend is growing in the last few days, but it could also depend on the general load on the database: when the host is under more load, slow queries will get even slower.

Screenshot 2026-06-04 at 19.13.19.png (3,712×2,308 px, 548 KB)

I have temporarily stopped the dimastbkbot tool. I sent an email to the maintainer and also posted a message to their user page linking to this Phabricator task.

History length is starting to decrease, but it will take a while:

Screenshot 2026-06-04 at 22.22.41.png (1,974×628 px, 103 KB)

Mentioned in SAL (#wikimedia-cloud) [2026-06-04T20:23:49Z] <dhinus> stopped the tool to investigate db issues T428139

fnegri changed the task status from Open to In Progress.Jun 4 2026, 8:27 PM
fnegri claimed this task.
fnegri triaged this task as High priority.
fnegri added a project: tools-platform-team.

I also lowered idle_transaction_timeout to 60:

SET GLOBAL idle_transaction_timeout=60;

But that's not enough to avoid the long transactions from user s51138 (heritage):

---TRANSACTION (0x7f029976a400), ACTIVE 754 sec
0 lock struct(s), heap size 1128, 0 row lock(s)
MariaDB thread id 602485920, OS thread handle 139614370244288, query id 25994983783 172.16.1.96 s51138
Trx read view will not see trx with id >= 167781842301, sees < 167781823070

@Lokal_Profil @Multichill could you please check if heritage is connecting to ToolsDB with autocommit=False? That would explain it holding idle transactions for a long time. Adding autocommit=True to pymysql.connect() might be enough to fix this.

I suspect there is also an underlying issue that I haven't discovered yet, and the stuck queries from heritage and dimastbkbot might be a symptom rather than the root cause.

The history length started to increase around 2026-05-28, which coincides with this increase in InnoDB I/O:

Screenshot 2026-06-04 at 23.08.46.png (1,930×544 px, 168 KB)

The situation after a few hours has improved:

Screenshot 2026-06-05 at 10.58.33.png (1,264×622 px, 63 KB)

I'm investigating the reason for the increase in InnoDB I/O Data Reads.

The history length started to increase around 2026-05-28, which coincides with this increase in InnoDB I/O:

@magnusmanske I wonder if this could be the mixnmatch "cache warmer" you mentioned in T427187: ToolsDB disk space usage growing too fast. Is it still running?

I think we need to prioritize T428087: [toolsdb] Add db-level and user-level monitoring because with so many tools accessing the database it's very hard to find which one caused the increase in data reads.

I tried two approaches to understand which user/db is driving the high rate of "Data Reads".

  1. Checking innodb_buffer_page:
MariaDB [(none)]> SELECT table_name, COUNT(*) AS pages,
    ->        ROUND(COUNT(*)*16/1024,1) AS mb
    -> FROM information_schema.innodb_buffer_page
    -> WHERE table_name IS NOT NULL
    -> GROUP BY table_name ORDER BY pages DESC LIMIT 20;

+------------------------------------------------------+--------+--------+
| table_name                                           | pages  | mb     |
+------------------------------------------------------+--------+--------+
| `s51434__mixnmatch_p`.`entry`                        | 612438 | 9569.3 |
| `s52421__commonsdelinquent_p`.`event`                | 252082 | 3938.8 |
| `s53952__freebase_p`.`keys`                          | 205261 | 3207.2 |
| `s51412__data`.`de_td_values`                        | 144221 | 2253.5 |
| `s56344__wiki_talents`.`en_delete_revisions`         | 124445 | 1944.5 |

[...]
  1. Logging all queries for 5 minutes using SET GLOBAL long_query_time = 0; and analyizing the results with pt-query-digest:
sudo pt-query-digest --order-by Rows_examined:sum --since '2026-06-05 12:10:00' --until '2026-06-05 12:15:00' --group-by db /srv/labsdb/data/tools-db-6-slow.log

# Profile
# Rank Query ID Response time              Calls  R/Call         V/M   Ite
# ==== ======== ========================== ====== ============== ===== ===
#    1 0x                   195.7199  0.0%   4444         0.0440  3.30 s56344__wiki_talents
#    2 0x                   277.3827  0.0% 161993         0.0017 10... s51434__mixnmatch_p
#    3 0x                   237.4376  0.0%   1821         0.1304  0.84 s52532__listeria_bot
#    4 0x                   230.0238  0.0%    176         1.3070  6.63 s51449__sbllog_p
#    5 0x                    98.3230  0.0%    477         0.2061  1.74 s52481__stats_global

OK, s51434__mixnmatch_p and s52421__commonsdelinquent_p are mine.

I have just stopped the mixnmatch "cache warmer", I think it's sufficient now, see if that makes a difference.

commonsdelinquent is a bot that has been running for ... decades really. Not originally mine but I and a few others adopted it. I don't think anything has been changed there in the last - years, really.

The anon_1 thing is not mine, AFAICT

I have just stopped the mixnmatch "cache warmer", I think it's sufficient now, see if that makes a difference.

I think that helped, thanks!

Screenshot 2026-06-05 at 16.38.12.png (1,260×542 px, 138 KB)

Screenshot 2026-06-05 at 16.38.22.png (1,682×540 px, 270 KB)

The anon_1 thing is not mine, AFAICT

Yep that one is from the dimastbkbot tool, I completely stopped that tool last night but db activity was still high, so I looked for other tools that could explain it.

Transaction History Length is back in check, though it has never reached zero:

Screenshot 2026-06-08 at 11.05.11.png (960×630 px, 58 KB)

InnoDB reads have also returned to their previous level:

Screenshot 2026-06-08 at 15.16.28.png (1,936×538 px, 204 KB)

It looks like all or most of the increase in reads was driven by the mixnmatch cache warmer.

I think we should prioritize a plan to give mixnmatch a dedicated db, a possible migration strategy was discussed a couple years ago in T350862: [toolsdb] Migrate mixnmatch db to Trove.

Mentioned in SAL (#wikimedia-cloud) [2026-06-08T13:24:00Z] <dhinus> webservice python3.9 start (T428139)

fnegri lowered the priority of this task from High to Low.Jun 8 2026, 1:50 PM

I have restarted the tool dimastbkbot, as I think it was not the main cause of the History Length growth, but more of a symptom of general db overload.

The tool maintainer replied to me on their talk page and said they will "remove" this tool this week.

Keeping this task open and low priority until I hear back from the user.

After restarting dimastbkbot, it's again showing up very frequently in the slow query log:

fnegri@tools-db-6:~$ sudo grep -B1 'Host.*s52323' /srv/labsdb/data/tools-db-6-slow.log |grep Time: |awk '{print $3}' |uniq -c
[...]
    178 260601
    159 260602
    258 260603
    176 260604
    105 260608
    350 260609
    273 260610
    187 260611

I notified the maintainer and will stop the tool again tomorrow if they don't intervene.

Mentioned in SAL (#wikimedia-cloud) [2026-06-12T13:29:32Z] <dhinus> stopped the tool again, as it's causing issues on toolsdb T428139

I stopped the dimastbkbot tool again. Before doing that, I noticed the pod is frequently crashing and restarting:

tools.dimastbkbot@tools-bastion-15:~$ kubectl get all
NAME                               READY   STATUS    RESTARTS          AGE
pod/dimastbkbot-68b58c9bd8-j92s5   1/1     Running   601 (9m56s ago)   4d

I filed a dedicated bug report in T429027: dimastbkbot tool is making very long db queries and crashing repeatedly, advising to only restart the tool after these issues are fixed.

I will keep monitoring the ToolsDB history length and resolve this task if it goes back to acceptable levels.

After stopping dimastbkbot, the next issue preventing history log purge was a long transaction from user s56581:

MariaDB [(none)]> SELECT trx_id, trx_started,        TIMESTAMPDIFF(SECOND, trx_started, NOW()) AS age_sec,        trx_state, trx_mysql_thread_id FROM information_schema.innodb_trx ORDER BY trx_started ASC;
+--------+---------------------+---------+-----------+---------------------+
| trx_id | trx_started         | age_sec | trx_state | trx_mysql_thread_id |
+--------+---------------------+---------+-----------+---------------------+
|      0 | 2026-06-12 12:00:05 |    7483 | RUNNING   |           633662162 |

[...]

MariaDB [(none)]> SELECT id, user, time, command, state, host, LEFT(info,50) AS info FROM information_schema.processlist WHERE id = 633662162;
+-----------+--------+------+---------+-------+-------------------+------+
| id        | user   | time | command | state | host              | info |
+-----------+--------+------+---------+-------+-------------------+------+
| 633662162 | s56581 |    0 | Sleep   |       | 172.16.1.83:7589  | NULL |
+-----------+--------+------+---------+-------+-------------------+------+
1 row in set (0.002 sec)

s56581 is the tool commonsdb-registry. This is another tool holding transactions open for a very long time, but not reaching the 60 seconds "idle transaction" timeout. I killed that thread manually but the tool will likely do it again. We can ask the maintainers to fix the tool, but the better solution would be T409857: [toolsdb] Automatically terminate long transactions.

The longest transactions are now for user s53685 (editgroups):

MariaDB [(none)]> SELECT trx_id, trx_started,        TIMESTAMPDIFF(SECOND, trx_started, NOW()) AS age_sec,        trx_state, trx_mysql_thread_id FROM information_schema.innodb_trx ORDER BY trx_started ASC;
+--------------+---------------------+---------+-----------+---------------------+
| trx_id       | trx_started         | age_sec | trx_state | trx_mysql_thread_id |
+--------------+---------------------+---------+-----------+---------------------+
|            0 | 2026-06-15 09:36:38 |     851 | RUNNING   |           675665986 |
|            0 | 2026-06-15 09:36:53 |     836 | RUNNING   |           675667320 |
|            0 | 2026-06-15 09:37:15 |     814 | RUNNING   |           675668876 |

MariaDB [(none)]> SELECT id, user, time, command, state, host, LEFT(info,50) AS info FROM information_schema.processlist WHERE id IN (675665986, 675667320, 675668876);
+-----------+--------+------+---------+--------------+--------------------+----------------------------------------------------+
| id        | user   | time | command | state        | host               | info                                               |
+-----------+--------+------+---------+--------------+--------------------+----------------------------------------------------+
| 675668876 | s53685 | 1065 | Query   | Sending data | 172.16.0.103:32584 | SELECT `store_edit`.`id`, `store_edit`.`oldrevid`, |
| 675667320 | s53685 | 1087 | Query   | Sending data | 172.16.0.103:64143 | SELECT `store_edit`.`id`, `store_edit`.`oldrevid`, |
| 675665986 | s53685 | 1101 | Query   | Sending data | 172.16.0.103:5770  | SELECT `store_edit`.`id`, `store_edit`.`oldrevid`, |
+-----------+--------+------+---------+--------------+--------------------+----------------------------------------------------+
3 rows in set (0.003 sec)

The history length is back to more reasonable values, but I'll keep monitoring it for another couple of days before resolving this task:

Screenshot 2026-06-15 at 11.56.17.png (1,118×610 px, 67 KB)

Unfortunately it has not stabilized and it's growing again:

Screenshot 2026-06-17 at 20.45.34.png (1,966×624 px, 90 KB)

Current long transactions:

MariaDB [(none)]> SELECT trx_id, trx_started, TIMESTAMPDIFF(SECOND, trx_started, NOW()) AS age_sec, trx_state, trx_mysql_thread_id, user, command, state FROM information_schema.innodb_trx JOIN information_schema.processlist p ON trx_mysql_thread_id = p.id HAVING age_sec > 60 ORDER BY trx_started ASC;
+--------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
| trx_id | trx_started         | age_sec | trx_state | trx_mysql_thread_id | user   | command | state        |
+--------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
|      0 | 2026-06-17 18:29:10 |     959 | RUNNING   |           681954993 | s56581 | Sleep   |              |
|      0 | 2026-06-17 18:32:39 |     750 | RUNNING   |           690793888 | s53685 | Query   | Sending data |
|      0 | 2026-06-17 18:42:27 |     162 | RUNNING   |           690824533 | s51434 | Execute | Sending data |
+--------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
3 rows in set (0.005 sec)

Still growing:

Screenshot 2026-06-18 at 16.44.51.png (1,958×630 px, 89 KB)

Current long transactions:

MariaDB [(none)]> SELECT trx_id, trx_started, TIMESTAMPDIFF(SECOND, trx_started, NOW()) AS age_sec, trx_state, trx_mysql_thread_id, user, command, state FROM information_schema.innodb_trx JOIN information_schema.processlist p ON trx_mysql_thread_id = p.id HAVING age_sec > 60 ORDER BY trx_started ASC;
+--------------+---------------------+---------+-----------+---------------------+--------+---------+---------------------+
| trx_id       | trx_started         | age_sec | trx_state | trx_mysql_thread_id | user   | command | state               |
+--------------+---------------------+---------+-----------+---------------------+--------+---------+---------------------+
|            0 | 2026-06-18 14:01:59 |     461 | RUNNING   |           695915477 | s51412 | Query   | Sending data        |
|            0 | 2026-06-18 14:03:40 |     360 | RUNNING   |           695925824 | s51138 | Sleep   |                     |
|            0 | 2026-06-18 14:03:47 |     353 | RUNNING   |           681954993 | s56581 | Sleep   |                     |
| 168475764570 | 2026-06-18 14:05:20 |     260 | RUNNING   |           695915345 | s55175 | Query   | Creating sort index |
|            0 | 2026-06-18 14:06:05 |     215 | RUNNING   |           695936535 | s54874 | Query   | Sending data        |
|            0 | 2026-06-18 14:06:42 |     178 | RUNNING   |           695938689 | s51434 | Execute | Sending data        |
+--------------+---------------------+---------+-----------+---------------------+--------+---------+---------------------+
6 rows in set (0.004 sec)

It did stabilize a little:

Screenshot 2026-06-29 at 11.39.11.png (1,012×626 px, 75 KB)

In https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/64, I added an alert to be notified when it goes above 50 millions (arbitrary threshold).

I also created a corresponding runbook: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing

Reopening as the transaction history length is again over 50mil, and it's been high in the past 2 weeks:

Screenshot 2026-07-15 at 15.15.40.png (1,082×608 px, 81 KB)

The alert ToolsDBHistoryLengthGrowing has fired today as expected when it reached 50mil.

The long transactions are from the same dbs as last month:

  • s53685 (tools.editgroups)
  • s51138 (tools.heritage)
  • s51434 (tools.mix-n-match)
  • s53685 (tools.editgroups)
  • s55175 (tools.ganfilter)
MariaDB [(none)]> SELECT trx_id, trx_started, TIMESTAMPDIFF(SECOND, trx_started, NOW()) AS age_sec, trx_state, trx_mysql_thread_id, user, command, state FROM information_schema.innodb_trx JOIN information_schema.processlist p ON trx_mysql_thread_id = p.id HAVING age_sec > 60 ORDER BY trx_started ASC;
+--------------+---------------------+---------+-----------+---------------------+--------+---------+---------------------+
| trx_id       | trx_started         | age_sec | trx_state | trx_mysql_thread_id | user   | command | state               |
+--------------+---------------------+---------+-----------+---------------------+--------+---------+---------------------+
|            0 | 2026-07-15 12:43:14 |    1614 | RUNNING   |           863943096 | s53685 | Query   | Sending data        |
| 169799715906 | 2026-07-15 12:45:28 |    1480 | RUNNING   |           863955242 | s51138 | Sleep   |                     |
|            0 | 2026-07-15 12:58:14 |     714 | RUNNING   |           864035485 | s51434 | Execute | Sending data        |
|            0 | 2026-07-15 12:58:18 |     710 | RUNNING   |           863858256 | s51434 | Execute | Sending data        |
|            0 | 2026-07-15 12:59:05 |     663 | RUNNING   |           864040350 | s53685 | Query   | Sending data        |
|            0 | 2026-07-15 13:04:50 |     318 | RUNNING   |           864074270 | s53685 | Query   | Sending data        |
| 169800421450 | 2026-07-15 13:08:47 |      81 | RUNNING   |           864046930 | s55175 | Query   | Creating sort index |
|            0 | 2026-07-15 13:08:55 |      73 | RUNNING   |           864098175 | s53685 | Query   | Sending data        |
+--------------+---------------------+---------+-----------+---------------------+--------+---------+---------------------+
8 rows in set (0.005 sec)
aputhin changed the task status from Open to In Progress.Wed, Jul 15, 4:03 PM
aputhin moved this task from Done to In progress on the tools-platform-team board.
fnegri moved this task from In progress to Done on the tools-platform-team board.

This went down on its own, and the history length is back to around 20 mil (which is still high, but at least is not growing out of control).

Screenshot 2026-07-17 at 18.29.54.png (1,288×622 px, 67 KB)

Resolving this task again, as I don't think there's anything to do right now, and the long-term plan is tracked in T291782: Migrate largest ToolsDB users to Trove.

fnegri moved this task from Done to In progress on the tools-platform-team board.

History length is still growing:

Screenshot 2026-07-24 at 11.52.24.png (1,106×626 px, 95 KB)

Current long transactions, most of them are from s51434 (mix-n-match) (cc @magnusmanske):

MariaDB [(none)]> SELECT trx_id, trx_started, TIMESTAMPDIFF(SECOND, trx_started, NOW()) AS age_sec, trx_state, trx_mysql_thread_id, user, command, state FROM information_schema.innodb_trx JOIN information_schema.processlist p ON trx_mysql_thread_id = p.id HAVING age_sec > 60 ORDER BY trx_started ASC;
+--------------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
| trx_id       | trx_started         | age_sec | trx_state | trx_mysql_thread_id | user   | command | state        |
+--------------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
| 170270385938 | 2026-07-24 05:50:48 |   14400 | RUNNING   |           921662454 | s56734 | Sleep   |              |
|            0 | 2026-07-24 09:26:31 |    1457 | RUNNING   |           922898818 | s51434 | Execute | Sending data |
|            0 | 2026-07-24 09:26:32 |    1456 | RUNNING   |           922896632 | s51434 | Execute | Sending data |
|            0 | 2026-07-24 09:26:32 |    1456 | RUNNING   |           922825005 | s51434 | Execute | Sending data |
|            0 | 2026-07-24 09:26:36 |    1452 | RUNNING   |           922819756 | s51434 | Execute | Sending data |
|            0 | 2026-07-24 09:27:21 |    1407 | RUNNING   |           922906613 | s51434 | Execute | Sending data |
| 170276202233 | 2026-07-24 09:30:08 |    1240 | RUNNING   |           922923396 | s51138 | Sleep   |              |
|            0 | 2026-07-24 09:44:04 |     404 | RUNNING   |           922995380 | s57719 | Query   | Sending data |
|            0 | 2026-07-24 09:46:34 |     254 | RUNNING   |           922999758 | s51434 | Execute | Sending data |
|            0 | 2026-07-24 09:46:34 |     254 | RUNNING   |           922962167 | s51434 | Execute | Sending data |
|            0 | 2026-07-24 09:46:35 |     253 | RUNNING   |           922945851 | s51434 | Execute | Sending data |
|            0 | 2026-07-24 09:46:41 |     247 | RUNNING   |           922945823 | s51434 | Execute | Sending data |
|            0 | 2026-07-24 09:47:24 |     204 | RUNNING   |           923011540 | s51434 | Execute | Sending data |
|            0 | 2026-07-24 09:47:35 |     193 | RUNNING   |           922945844 | s51434 | Execute | Sending data |
+--------------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
14 rows in set (0.007 sec)

History length is continuing to increase:

Screenshot 2026-07-27 at 14.28.14.png (1,082×626 px, 87 KB)

@magnusmanske there are too many long queries coming from mixnmatch (see list below), some of them are reaching the maximum duration of 1 hour and getting killed. Can you please check what's happening with mixnmatch and reduce the load on ToolsDB?

MariaDB [(none)]> SELECT trx_id, trx_started, TIMESTAMPDIFF(SECOND, trx_started, NOW()) AS age_sec, trx_state, trx_mysql_thread_id, user, command, state FROM information_schema.innodb_trx JOIN information_schema.processlist p ON trx_mysql_thread_id = p.id HAVING age_sec > 60 ORDER BY trx_started ASC;
+--------------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
| trx_id       | trx_started         | age_sec | trx_state | trx_mysql_thread_id | user   | command | state        |
+--------------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
|            0 | 2026-07-27 12:20:33 |     328 | RUNNING   |           944441747 | s51434 | Execute | Sending data |
|            0 | 2026-07-27 12:20:40 |     321 | RUNNING   |           944441696 | s51434 | Execute | Sending data |
|            0 | 2026-07-27 12:20:42 |     319 | RUNNING   |           944441698 | s51434 | Execute | Sending data |
|            0 | 2026-07-27 12:20:42 |     319 | RUNNING   |           944441722 | s51434 | Execute | Sending data |
|            0 | 2026-07-27 12:20:46 |     315 | RUNNING   |           944441715 | s51434 | Execute | Sending data |
|            0 | 2026-07-27 12:20:48 |     313 | RUNNING   |           944441748 | s51434 | Execute | Sending data |
|            0 | 2026-07-27 12:21:04 |     297 | RUNNING   |           944441735 | s51434 | Execute | Sending data |
|            0 | 2026-07-27 12:21:06 |     295 | RUNNING   |           944441752 | s51434 | Execute | Sending data |
|            0 | 2026-07-27 12:21:40 |     261 | RUNNING   |           944441703 | s51434 | Execute | Sending data |
|            0 | 2026-07-27 12:22:51 |     190 | RUNNING   |           944460668 | s54113 | Query   | Sending data |
|            0 | 2026-07-27 12:23:58 |     123 | RUNNING   |           944457859 | s55175 | Sleep   |              |
| 170454096586 | 2026-07-27 12:24:23 |      98 | RUNNING   |           944477213 | s51434 | Execute | Sending data |
|            0 | 2026-07-27 12:24:23 |      98 | RUNNING   |           944477262 | s51434 | Execute | Sending data |
|            0 | 2026-07-27 12:24:23 |      98 | RUNNING   |           944477273 | s51434 | Execute | Sending data |
|            0 | 2026-07-27 12:24:52 |      69 | RUNNING   |           944477236 | s51434 | Execute | Sending data |
|            0 | 2026-07-27 12:24:56 |      65 | RUNNING   |           944477260 | s51434 | Execute | Sending data |
|            0 | 2026-07-27 12:24:58 |      63 | RUNNING   |           944477259 | s51434 | Execute | Sending data |
+--------------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
17 rows in set (0.018 sec)

Improved a little but still very high:

Screenshot 2026-07-28 at 17.38.32.png (920×618 px, 79 KB)

Long transactions are still mostly from mixnmatch:

MariaDB [(none)]> SELECT trx_id, trx_started, TIMESTAMPDIFF(SECOND, trx_started, NOW()) AS age_sec, trx_state, trx_mysql_thread_id, user, command, state FROM information_schema.innodb_trx JOIN information_schema.processlist p ON trx_mysql_thread_id = p.id HAVING age_sec > 60 ORDER BY trx_started ASC;
+--------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
| trx_id | trx_started         | age_sec | trx_state | trx_mysql_thread_id | user   | command | state        |
+--------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
|      0 | 2026-07-28 15:31:27 |     461 | RUNNING   |           951528895 | s51434 | Execute | Sending data |
|      0 | 2026-07-28 15:36:19 |     169 | RUNNING   |           951567656 | s51434 | Execute | Sending data |
|      0 | 2026-07-28 15:36:28 |     160 | RUNNING   |           951567694 | s51434 | Execute | Sending data |
|      0 | 2026-07-28 15:36:28 |     160 | RUNNING   |           951567671 | s51434 | Execute | Sending data |
|      0 | 2026-07-28 15:36:31 |     157 | RUNNING   |           951567665 | s51434 | Execute | Sending data |
|      0 | 2026-07-28 15:36:33 |     155 | RUNNING   |           951567666 | s51434 | Execute | Sending data |
|      0 | 2026-07-28 15:36:34 |     154 | RUNNING   |           951567664 | s51434 | Execute | Sending data |
|      0 | 2026-07-28 15:36:38 |     150 | RUNNING   |           951567689 | s51434 | Execute | Sending data |
|      0 | 2026-07-28 15:36:45 |     143 | RUNNING   |           951567691 | s51434 | Execute | Sending data |
|      0 | 2026-07-28 15:36:45 |     143 | RUNNING   |           951567655 | s51434 | Execute | Sending data |
|      0 | 2026-07-28 15:37:29 |      99 | RUNNING   |           951567693 | s51434 | Execute | Sending data |
|      0 | 2026-07-28 15:38:02 |      66 | RUNNING   |           951570838 | s54113 | Query   | Sending data |
+--------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
12 rows in set (0.003 sec)

Update:

Screenshot 2026-07-30 at 17.14.33.png (1,100×616 px, 90 KB)

MariaDB [(none)]> SELECT trx_id, trx_started, TIMESTAMPDIFF(SECOND, trx_started, NOW()) AS age_sec, trx_state, trx_mysql_thread_id, user, command, state FROM information_schema.innodb_trx JOIN information_schema.processlist p ON trx_mysql_thread_id = p.id HAVING age_sec > 60 ORDER BY trx_started ASC;
+--------------+---------------------+---------+-----------+---------------------+--------+---------+---------------------+
| trx_id       | trx_started         | age_sec | trx_state | trx_mysql_thread_id | user   | command | state               |
+--------------+---------------------+---------+-----------+---------------------+--------+---------+---------------------+
|            0 | 2026-07-30 12:45:38 |    8967 | RUNNING   |           965046012 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 12:55:38 |    8367 | RUNNING   |           965101388 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 13:01:55 |    7990 | RUNNING   |           965137605 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 13:08:26 |    7599 | RUNNING   |           965173710 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 13:15:14 |    7191 | RUNNING   |           965210395 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 13:23:10 |    6715 | RUNNING   |           965251883 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 13:32:29 |    6156 | RUNNING   |           965301841 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 13:42:34 |    5551 | RUNNING   |           965357007 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 13:48:46 |    5179 | RUNNING   |           965389889 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 13:55:30 |    4775 | RUNNING   |           965426824 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 14:03:13 |    4312 | RUNNING   |           965466031 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 14:10:57 |    3848 | RUNNING   |           965503519 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 14:20:25 |    3280 | RUNNING   |           965550127 | s51434 | Execute | Sending data        |
| 170640139443 | 2026-07-30 14:21:24 |    3221 | RUNNING   |           965239869 | s53220 | Execute | Updating            |
|            0 | 2026-07-30 14:30:29 |    2676 | RUNNING   |           965602077 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 14:32:33 |    2552 | RUNNING   |           965602076 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 14:36:33 |    2312 | RUNNING   |           965632796 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 14:43:03 |    1922 | RUNNING   |           965665442 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 14:47:22 |    1663 | RUNNING   |           965685798 | s51412 | Query   | Sending data        |
|            0 | 2026-07-30 14:49:32 |    1533 | RUNNING   |           965695809 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 14:49:32 |    1533 | RUNNING   |           965695870 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 14:57:10 |    1075 | RUNNING   |           965736887 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 14:57:11 |    1074 | RUNNING   |           965736888 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 15:00:46 |     859 | RUNNING   |           965755339 | s51412 | Query   | Sending data        |
|            0 | 2026-07-30 15:04:02 |     663 | RUNNING   |           965753953 | s51412 | Query   | Sending data        |
|            0 | 2026-07-30 15:05:45 |     560 | RUNNING   |           965782535 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 15:05:45 |     560 | RUNNING   |           965782475 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 15:05:46 |     559 | RUNNING   |           965782528 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 15:08:24 |     401 | RUNNING   |           965796013 | s51138 | Sleep   |                     |
|            0 | 2026-07-30 15:09:45 |     320 | RUNNING   |           965803545 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 15:09:45 |     320 | RUNNING   |           965782607 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 15:10:06 |     299 | RUNNING   |           965782510 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 15:10:09 |     296 | RUNNING   |           965804641 | s51434 | Execute | Sending data        |
|            0 | 2026-07-30 15:12:25 |     160 | RUNNING   |           965816965 | s53952 | Query   | Sending data        |
|            0 | 2026-07-30 15:13:09 |     116 | RUNNING   |           965820812 | s53952 | Query   | Sending data        |
|            0 | 2026-07-30 15:13:59 |      66 | RUNNING   |           965825447 | s51138 | Query   | Creating sort index |
|            0 | 2026-07-30 15:14:00 |      65 | RUNNING   |           965825480 | s51138 | Query   | Creating sort index |
+--------------+---------------------+---------+-----------+---------------------+--------+---------+---------------------+
37 rows in set (0.082 sec)

Things are not getting better, history length is now over 100 mil. There are currently 52 queries from the mixnmatch user (s51434), which should not even be possible as we generally limit tools to max 10 connections:

MariaDB [(none)]> SELECT trx_id, trx_started, TIMESTAMPDIFF(SECOND, trx_started, NOW()) AS age_sec, trx_state, trx_mysql_thread_id, user, command, state FROM information_schema.innodb_trx JOIN information_schema.processlist p ON trx_mysql_thread_id = p.id HAVING age_sec > 60 ORDER BY trx_started ASC;
+--------------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
| trx_id       | trx_started         | age_sec | trx_state | trx_mysql_thread_id | user   | command | state        |
+--------------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
|            0 | 2026-07-31 06:18:21 |   10152 | RUNNING   |           971368105 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 06:28:33 |    9540 | RUNNING   |           971441535 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 06:34:12 |    9201 | RUNNING   |           971472910 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 06:40:12 |    8841 | RUNNING   |           971505217 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 06:46:34 |    8459 | RUNNING   |           971541377 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 06:53:25 |    8048 | RUNNING   |           971577984 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 07:01:49 |    7544 | RUNNING   |           971623500 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 07:11:51 |    6942 | RUNNING   |           971675444 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 07:17:29 |    6604 | RUNNING   |           971703325 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 07:23:33 |    6240 | RUNNING   |           971734792 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 07:30:21 |    5832 | RUNNING   |           971769270 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 07:37:46 |    5387 | RUNNING   |           971807771 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 07:46:17 |    4876 | RUNNING   |           971851586 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 07:56:24 |    4269 | RUNNING   |           971901703 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 08:02:10 |    3923 | RUNNING   |           971930371 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 08:08:34 |    3539 | RUNNING   |           971960205 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 08:13:09 |    3264 | RUNNING   |           971983950 | s56066 | Query   | Sending data |
|            0 | 2026-07-31 08:13:09 |    3264 | RUNNING   |           971983951 | s56066 | Query   | Sending data |
|            0 | 2026-07-31 08:14:56 |    3157 | RUNNING   |           971993237 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 08:17:27 |    3006 | RUNNING   |           972005950 | s56066 | Query   | Sending data |
|            0 | 2026-07-31 08:17:27 |    3006 | RUNNING   |           972005949 | s56066 | Query   | Sending data |
|            0 | 2026-07-31 08:21:58 |    2735 | RUNNING   |           972028770 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 08:23:34 |    2639 | RUNNING   |           972028791 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 08:29:47 |    2266 | RUNNING   |           972071088 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 08:39:50 |    1663 | RUNNING   |           972125473 | s51434 | Execute | Sending data |
| 170687932961 | 2026-07-31 08:39:57 |    1656 | RUNNING   |           972125903 | s51138 | Sleep   |              |
|            0 | 2026-07-31 08:44:33 |    1380 | RUNNING   |           972150142 | s53685 | Query   | Sending data |
|            0 | 2026-07-31 08:44:39 |    1374 | RUNNING   |           972150712 | s53685 | Query   | Sending data |
|            0 | 2026-07-31 08:45:09 |    1344 | RUNNING   |           972153407 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 08:51:22 |     971 | RUNNING   |           972184537 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 08:51:23 |     970 | RUNNING   |           972184529 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 08:57:17 |     616 | RUNNING   |           972212674 | s53685 | Query   | Sending data |
|            0 | 2026-07-31 08:57:35 |     598 | RUNNING   |           972213939 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 08:57:36 |     597 | RUNNING   |           972213979 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 08:57:53 |     580 | RUNNING   |           972213957 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 08:58:18 |     555 | RUNNING   |           972217103 | s53952 | Query   | Sending data |
|            0 | 2026-07-31 08:58:59 |     514 | RUNNING   |           972220674 | s53952 | Query   | Sending data |
|            0 | 2026-07-31 08:59:51 |     462 | RUNNING   |           972224927 | s53952 | Query   | Sending data |
|            0 | 2026-07-31 09:00:00 |     453 | RUNNING   |           972225853 | s53952 | Query   | Sending data |
|            0 | 2026-07-31 09:01:36 |     357 | RUNNING   |           972231428 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 09:01:36 |     357 | RUNNING   |           972213993 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 09:01:37 |     356 | RUNNING   |           972213995 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 09:01:37 |     356 | RUNNING   |           972230911 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 09:04:30 |     183 | RUNNING   |           972247562 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 09:04:30 |     183 | RUNNING   |           972247582 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 09:04:30 |     183 | RUNNING   |           972247529 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 09:04:31 |     182 | RUNNING   |           972247573 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 09:04:31 |     182 | RUNNING   |           972247555 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 09:04:43 |     170 | RUNNING   |           972229060 | s51412 | Query   | Sending data |
|            0 | 2026-07-31 09:05:00 |     153 | RUNNING   |           972249965 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 09:05:32 |     121 | RUNNING   |           972252540 | s51434 | Execute | Sending data |
|            0 | 2026-07-31 09:06:30 |      63 | RUNNING   |           972254588 | s55928 | Sleep   |              |
|            0 | 2026-07-31 09:06:32 |      61 | RUNNING   |           972254771 | s55928 | Sleep   |              |
+--------------+---------------------+---------+-----------+---------------------+--------+---------+--------------+
53 rows in set (0.013 sec)

MariaDB [(none)]> SELECT id, time, command, state, host, LEFT(info,50) AS info FROM information_schema.processlist WHERE user = 's51434';
+-----------+-------+---------+---------------------------------+---------------------+----------------------------------------------------+
| id        | time  | command | state                           | host                | info                                               |
+-----------+-------+---------+---------------------------------+---------------------+----------------------------------------------------+
| 972264467 |     0 | Sleep   |                                 | 172.16.3.187:7310   | NULL                                               |
| 972264463 |     0 | Sleep   |                                 | 172.16.19.228:4189  | NULL                                               |
| 972264462 |     0 | Sleep   |                                 | 172.16.19.228:43939 | NULL                                               |
| 972263312 |     1 | Sleep   |                                 | 172.16.3.187:50515  | NULL                                               |
| 972263310 |     1 | Sleep   |                                 | 172.16.3.187:44333  | NULL                                               |
| 972262672 |     0 | Sleep   |                                 | 172.16.19.228:50800 | NULL                                               |
| 972260642 |    48 | Execute | Fulltext initialization         | 172.16.3.187:63880  | SELECT entry.id,entry.catalog,entry.ext_id,entry.e |
| 972259501 |    58 | Execute | Fulltext initialization         | 172.16.3.187:55047  | SELECT entry.id,entry.catalog,entry.ext_id,entry.e |
| 972257614 |     0 | Sleep   |                                 | 172.16.19.228:47955 | NULL                                               |
| 972257540 |     0 | Sleep   |                                 | 172.16.19.228:19116 | NULL                                               |
| 972257538 |     0 | Execute | Update                          | 172.16.19.228:7047  | INSERT IGNORE INTO `entry` (`catalog`,`ext_id`,`ex |
| 972252540 |   143 | Execute | Sending data                    | 172.16.19.228:38759 | (SELECT /* automatch_by_search_fetch_page_unmatche |
| 972249965 |   175 | Execute | Sending data                    | 172.16.19.228:26668 | (SELECT /* automatch_by_search_fetch_page_unmatche |
| 972247582 |     2 | Sleep   |                                 | 172.16.19.228:62313 | NULL                                               |
| 972247573 |   204 | Execute | Sending data                    | 172.16.19.228:44169 | (SELECT /* automatch_by_search_fetch_page_unmatche |
| 972247562 |   205 | Execute | Sending data                    | 172.16.19.228:45153 | CREATE TEMPORARY TABLE tmp_cnd AS
                 |
| 972247559 |   205 | Execute | Waiting for table metadata lock | 172.16.19.228:54842 | DROP TABLE IF EXISTS tmp_taxa                      |
| 972247555 |   205 | Execute | Sending data                    | 172.16.19.228:49309 | SELECT `e`.`id` AS entry_id, `e`.`ext_name`, `pd`. |
| 972247529 |   205 | Execute | Sending data                    | 172.16.19.228:65349 | SELECT /* maintenance_match_people_via_name_and_fu |
| 972247516 |   205 | Execute | Waiting for table metadata lock | 172.16.19.228:48352 | TRUNCATE common_names_birth_year_tmp               |
| 972231428 |   379 | Execute | Sending data                    | 172.16.19.228:35187 | SET STATEMENT max_statement_time=480 FOR (SELECT / |
| 972230911 |   378 | Execute | Sending data                    | 172.16.19.228:3998  | SET STATEMENT max_statement_time=480 FOR (SELECT / |
| 972213995 |   379 | Execute | Sending data                    | 172.16.19.228:33310 | SET STATEMENT max_statement_time=480 FOR SELECT `e |
| 972213993 |   378 | Execute | Sending data                    | 172.16.19.228:42560 | SET STATEMENT max_statement_time=480 FOR (SELECT / |
| 972213979 |   619 | Execute | Sending data                    | 172.16.19.228:44559 | CREATE TEMPORARY TABLE tmp_cnd AS
                 |
| 972213957 |   603 | Execute | Sending data                    | 172.16.19.228:5648  | INSERT /* maintenance_common_names_birth_year */ I |
| 972213939 |   619 | Execute | Sending data                    | 172.16.19.228:33860 | SELECT /* maintenance_match_people_via_name_and_fu |
| 972184537 |   993 | Execute | Sending data                    | 172.16.19.228:51446 | SELECT /* maintenance_match_people_via_name_and_fu |
| 972184529 |   992 | Execute | Sending data                    | 172.16.19.228:14348 | CREATE TEMPORARY TABLE tmp_cnd AS
                 |
| 972153407 |  1366 | Execute | Sending data                    | 172.16.19.228:51345 | SELECT /* maintenance_match_people_via_name_and_fu |
| 972125473 |  1684 | Execute | Sending data                    | 172.16.19.228:53749 | SELECT /* maintenance_match_people_via_name_and_fu |
| 972071088 |  2288 | Execute | Sending data                    | 172.16.19.228:57655 | SELECT /* maintenance_match_people_via_name_and_fu |
| 972028791 |  2661 | Execute | Sending data                    | 172.16.19.228:1324  | CREATE TABLE tmp_taxa AS
		        SELECT /* maint |
| 972028770 |  2757 | Execute | Sending data                    | 172.16.19.228:45674 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971993237 |  3178 | Execute | Sending data                    | 172.16.19.228:1964  | SELECT /* maintenance_match_people_via_name_and_fu |
| 971960205 |  3561 | Execute | Sending data                    | 172.16.19.228:41019 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971930371 |  3944 | Execute | Sending data                    | 172.16.19.228:33612 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971901703 |  4291 | Execute | Sending data                    | 172.16.19.228:8267  | SELECT /* maintenance_match_people_via_name_and_fu |
| 971851586 |  4898 | Execute | Sending data                    | 172.16.19.228:15720 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971809759 |    35 | Sleep   |                                 | 172.16.16.240:45890 | NULL                                               |
| 971807771 |  5409 | Execute | Sending data                    | 172.16.19.228:20194 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971769270 |  5854 | Execute | Sending data                    | 172.16.19.228:51204 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971734792 |  6261 | Execute | Sending data                    | 172.16.19.228:61482 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971703325 |  6625 | Execute | Sending data                    | 172.16.19.228:53696 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971675444 |  6964 | Execute | Sending data                    | 172.16.19.228:35661 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971623500 |  7565 | Execute | Sending data                    | 172.16.19.228:13852 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971577984 |  8070 | Execute | Sending data                    | 172.16.19.228:20777 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971541377 |  8481 | Execute | Sending data                    | 172.16.19.228:9587  | SELECT /* maintenance_match_people_via_name_and_fu |
| 971505217 |  8863 | Execute | Sending data                    | 172.16.19.228:29928 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971472910 |  9222 | Execute | Sending data                    | 172.16.19.228:61831 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971441535 |  9562 | Execute | Sending data                    | 172.16.19.228:63273 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971368105 | 10174 | Execute | Sending data                    | 172.16.19.228:44354 | SELECT /* maintenance_match_people_via_name_and_fu |
+-----------+-------+---------+---------------------------------+---------------------+----------------------------------------------------+
52 rows in set (0.002 sec)

I didn't get any reply from @magnusmanske either on this task or via email, I will now stop some of the mixnmatch processes, starting from rustbot which is in CrashLoopBackOff:

tools.mix-n-match@tools-bastion-15:~$ kubectl get all
NAME                               READY   STATUS             RESTARTS          AGE
pod/mix-n-match-5d8c746b87-v4j84   1/1     Running            3 (37h ago)       47h
pod/rustbot-687f45dbcd-vb9gw       0/1     CrashLoopBackOff   383 (2m44s ago)   47h

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/mix-n-match   ClusterIP   10.101.208.88   <none>        8000/TCP   79d

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mix-n-match   1/1     1            1           79d
deployment.apps/rustbot       0/1     1            0           47h

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/mix-n-match-57cd46697d   0         0         0       10d
replicaset.apps/mix-n-match-5d8c746b87   1         1         1       47h
replicaset.apps/mix-n-match-5f59bcd8f7   0         0         0       32d
replicaset.apps/mix-n-match-65df98c764   0         0         0       3d2h
replicaset.apps/mix-n-match-65f6946b5b   0         0         0       32d
replicaset.apps/mix-n-match-67fd476f7d   0         0         0       47h
replicaset.apps/mix-n-match-6f66dccf6    0         0         0       13d
replicaset.apps/mix-n-match-84786d5cbd   0         0         0       31d
replicaset.apps/mix-n-match-869c8dfb74   0         0         0       2d20h
replicaset.apps/mix-n-match-c4dd7b678    0         0         0       36d
replicaset.apps/mix-n-match-cb5f94f79    0         0         0       11d
replicaset.apps/rustbot-687f45dbcd       1         1         0       47h
tools.mix-n-match@tools-bastion-15:~$ kubectl scale --replicas=0 deployment.apps/rustbot

tools.mix-n-match@tools-bastion-15:~$ kubectl get all
NAME                               READY   STATUS    RESTARTS      AGE
pod/mix-n-match-5d8c746b87-v4j84   1/1     Running   3 (37h ago)   47h

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/mix-n-match   ClusterIP   10.101.208.88   <none>        8000/TCP   79d

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mix-n-match   1/1     1            1           79d
deployment.apps/rustbot       0/0     0            0           47h

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/mix-n-match-57cd46697d   0         0         0       10d
replicaset.apps/mix-n-match-5d8c746b87   1         1         1       47h
replicaset.apps/mix-n-match-5f59bcd8f7   0         0         0       32d
replicaset.apps/mix-n-match-65df98c764   0         0         0       3d2h
replicaset.apps/mix-n-match-65f6946b5b   0         0         0       32d
replicaset.apps/mix-n-match-67fd476f7d   0         0         0       47h
replicaset.apps/mix-n-match-6f66dccf6    0         0         0       13d
replicaset.apps/mix-n-match-84786d5cbd   0         0         0       31d
replicaset.apps/mix-n-match-869c8dfb74   0         0         0       2d20h
replicaset.apps/mix-n-match-c4dd7b678    0         0         0       36d
replicaset.apps/mix-n-match-cb5f94f79    0         0         0       11d
replicaset.apps/rustbot-687f45dbcd       0         0         0       47h

There are still 41 active queries from user s51434, I will kill all the queries with time > 3600 (1 hour). We should also investigate why they were not killed by MariaDB as we have max_statement_time = 3600 in the config.

I killed all queries with time > 3600. We're down to 27 threads for s51434, most of them in Sleep. The remaining maintenance_match_people_... queries are not likely to complete, so I will kill them as well.

MariaDB [(none)]> SELECT id, time, command, state, host, LEFT(info,50) AS info FROM information_schema.processlist WHERE user = 's51434';
+-----------+------+---------+---------------------+---------------------+----------------------------------------------------+
| id        | time | command | state               | host                | info                                               |
+-----------+------+---------+---------------------+---------------------+----------------------------------------------------+
| 972361932 |    0 | Sleep   |                     | 172.16.3.187:56584  | NULL                                               |
| 972361931 |    0 | Sleep   |                     | 172.16.3.187:7187   | NULL                                               |
| 972360874 |    5 | Sleep   |                     | 172.16.3.187:25696  | NULL                                               |
| 972360873 |    5 | Sleep   |                     | 172.16.3.187:54979  | NULL                                               |
| 972360871 |    0 | Sleep   |                     | 172.16.3.187:59648  | NULL                                               |
| 972360870 |    5 | Sleep   |                     | 172.16.3.187:14189  | NULL                                               |
| 972360869 |    5 | Sleep   |                     | 172.16.3.187:17965  | NULL                                               |
| 972360868 |    5 | Sleep   |                     | 172.16.3.187:2099   | NULL                                               |
| 972360827 |    5 | Sleep   |                     | 172.16.3.187:20093  | NULL                                               |
| 972360826 |    5 | Sleep   |                     | 172.16.3.187:17116  | NULL                                               |
| 972360713 |    5 | Sleep   |                     | 172.16.3.187:48039  | NULL                                               |
| 972360292 |   24 | Sleep   |                     | 172.16.3.187:38789  | NULL                                               |
| 972360291 |   24 | Sleep   |                     | 172.16.3.187:33705  | NULL                                               |
| 972360290 |   24 | Sleep   |                     | 172.16.3.187:45199  | NULL                                               |
| 972360288 |   24 | Sleep   |                     | 172.16.3.187:28939  | NULL                                               |
| 972360279 |    5 | Sleep   |                     | 172.16.3.187:55349  | NULL                                               |
| 972360278 |   24 | Sleep   |                     | 172.16.3.187:58266  | NULL                                               |
| 972289910 |  847 | Execute | Sending data        | 172.16.19.228:14621 | CREATE TEMPORARY TABLE tmp_cnd AS
                 |
| 972289837 |  848 | Execute | Sending data        | 172.16.19.228:14346 | SELECT /* maintenance_match_people_via_name_and_fu |
| 972247562 | 1347 | Execute | Creating sort index | 172.16.19.228:45153 | CREATE TEMPORARY TABLE tmp_cnd AS
                 |
| 972247529 | 1348 | Execute | Sending data        | 172.16.19.228:65349 | SELECT /* maintenance_match_people_via_name_and_fu |
| 972213939 | 1762 | Execute | Sending data        | 172.16.19.228:33860 | SELECT /* maintenance_match_people_via_name_and_fu |
| 971809759 |   38 | Sleep   |                     | 172.16.16.240:45890 | NULL                                               |
| 972184537 | 2135 | Execute | Sending data        | 172.16.19.228:51446 | SELECT /* maintenance_match_people_via_name_and_fu |
| 972153407 | 2509 | Execute | Sending data        | 172.16.19.228:51345 | SELECT /* maintenance_match_people_via_name_and_fu |
| 972125473 | 2827 | Execute | Sending data        | 172.16.19.228:53749 | SELECT /* maintenance_match_people_via_name_and_fu |
| 972071088 | 3431 | Execute | Sending data        | 172.16.19.228:57655 | SELECT /* maintenance_match_people_via_name_and_fu |
+-----------+------+---------+---------------------+---------------------+----------------------------------------------------+
27 rows in set (0.002 sec)

This is the current situation, I will let it run for a while and check back later:

MariaDB [(none)]> SELECT id, time, command, state, host, LEFT(info,50) AS info FROM information_schema.processlist WHERE user = 's51434';
+-----------+------+---------+---------------------+---------------------+----------------------------------------------------+
| id        | time | command | state               | host                | info                                               |
+-----------+------+---------+---------------------+---------------------+----------------------------------------------------+
| 972382569 |   21 | Sleep   |                     | 172.16.3.187:31726  | NULL                                               |
| 972382568 |    0 | Sleep   |                     | 172.16.3.187:43857  | NULL                                               |
| 972382567 |   21 | Sleep   |                     | 172.16.3.187:48612  | NULL                                               |
| 972382566 |    2 | Sleep   |                     | 172.16.3.187:23476  | NULL                                               |
| 972382565 |    2 | Sleep   |                     | 172.16.3.187:40096  | NULL                                               |
| 972382564 |   21 | Sleep   |                     | 172.16.3.187:52481  | NULL                                               |
| 972382563 |   21 | Sleep   |                     | 172.16.3.187:8925   | NULL                                               |
| 972382562 |   21 | Sleep   |                     | 172.16.3.187:64744  | NULL                                               |
| 972382541 |   21 | Sleep   |                     | 172.16.3.187:62600  | NULL                                               |
| 972381670 |   21 | Sleep   |                     | 172.16.3.187:28401  | NULL                                               |
| 972289910 | 1143 | Execute | Creating sort index | 172.16.19.228:14621 | CREATE TEMPORARY TABLE tmp_cnd AS
                 |
| 971809759 |   33 | Sleep   |                     | 172.16.16.240:45890 | NULL                                               |
+-----------+------+---------+---------------------+---------------------+----------------------------------------------------+
12 rows in set (0.004 sec)

History length has started going down:

Screenshot 2026-07-31 at 14.27.18.png (1,114×624 px, 67 KB)

Disk metrics are also much healthier now:

Screenshot 2026-07-31 at 14.26.21.png (2,220×536 px, 211 KB)