Page MenuHomePhabricator

x1 huge increase on database binlogs
Closed, ResolvedPublic

Assigned To
Authored By
Marostegui
Nov 23 2023, 10:11 AM
Referenced Files
F41648359: grafik.png
Jan 2 2024, 4:32 AM
F41614212: Captura de pantalla 2023-12-20 a las 7.51.48.png
Dec 20 2023, 6:52 AM
F41543604: grafik.png
Nov 28 2023, 3:39 PM
F41531921: grafik.png
Nov 23 2023, 8:05 PM

Description

There's been a huge increase on binlogs usage on x1, more than 2000 per host.
This was db2096: https://grafana.wikimedia.org/goto/C9z6JgNIk?orgId=1

After purging them it went back to normal disk values.
This seems the pattern across x1, we could reduce the expiration from 30 days to something lower, but we need to first investigate why this increase on writes (or transaction size)

Event Timeline

Marostegui renamed this task from x1 huge increase on binlogs to x1 huge increase on database binlogs.Nov 23 2023, 10:12 AM
Marostegui triaged this task as High priority.
Marostegui moved this task from Triage to In progress on the DBA board.

FWIW, I looked at backup sizes in the past year and didn't find any increase of table sizes per se:

grafik.png (2×4 px, 346 KB)

From what I saw it was just the amount of binlog generated which could be:

  1. more writes
  2. bigger transactions making the binlogs reach their 1G size faster and thus generating more withing the purge time (30 days)

Once we hit the 30 days (first binlog is 27th Nov), it should start to get stable.

Yep, this seems fine

/dev/mapper/tank-data xfs       3.6T  1.2T  2.5T  33% /srv

Thanks everyone!