Change Details

The extra writes could apparently come from the new jobqueue- this is not yet proven- but the deploy and the load increase timestamps match almost perfectly: https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1052&var-port=9104 The extra writes by themselves are not a huge problem (even at 2-3 times, they are relatively not huge for a datababase server), but it is causing disk slowdowns, plus there are extra load in connections/s, which make reads queue and in some cases fail to check replication lag. With 100-200 connections per server, instead of the usual <100, queries get queued for longer, affecting overall performance of all queries.

The extra writes come from the new jobqueue- the refreshlinks job. The deploy and the load increase timestamps match almost perfectly: https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1052&var-port=9104 This was later proven by lowering the concurrency, which immediately affected the databases. The extra writes by themselves are not a huge problem (even at 2-3 times, they are relatively not huge for a datababase server), but it is causing disk slowdowns, plus there are extra load in connections/s, which make reads queue and in some cases fail to check replication lag. With 100-200 connections per server, instead of the usual <100, queries get queued for longer, affecting overall performance of all queries.

The extra writes could apparently come from the new jobqueue- this is not yet proven- but thethe refreshlinks job. The deploy and the load increase timestamps match almost perfectly: https://grafana.wikimedia.org/dashboard/db/mysql?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1052&var-port=9104 This was later proven by lowering the concurrency, which immediately affected the databases. The extra writes by themselves are not a huge problem (even at 2-3 times, they are relatively not huge for a datababase server), but it is causing disk slowdowns, plus there are extra load in connections/s, which make reads queue and in some cases fail to check replication lag. With 100-200 connections per server, instead of the usual <100, queries get queued for longer, affecting overall performance of all queries.