| • Marostegui | |
| Nov 23 2023, 10:11 AM |
| F41648359: grafik.png | |
| Jan 2 2024, 4:32 AM |
| F41614212: Captura de pantalla 2023-12-20 a las 7.51.48.png | |
| Dec 20 2023, 6:52 AM |
| F41543604: grafik.png | |
| Nov 28 2023, 3:39 PM |
| F41531921: grafik.png | |
| Nov 23 2023, 8:05 PM |
There's been a huge increase on binlogs usage on x1, more than 2000 per host.
This was db2096: https://grafana.wikimedia.org/goto/C9z6JgNIk?orgId=1
After purging them it went back to normal disk values.
This seems the pattern across x1, we could reduce the expiration from 30 days to something lower, but we need to first investigate why this increase on writes (or transaction size)
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Ladsgroup | T351871 x1 huge increase on database binlogs | |||
| Resolved | Urbanecm_WMF | T351898 Reduce size of growthexperiments_user_impact.geui_data json blobs | |||
| Open | None | T352013 Revise UserImpactFormatter logic for noPageViewDataYet |
FWIW, I looked at backup sizes in the past year and didn't find any increase of table sizes per se:
From what I saw it was just the amount of binlog generated which could be:
Yeah, Looking at the binlogs the only thing that stands out heavily is T351898: Reduce size of growthexperiments_user_impact.geui_data json blobs
This still looks decent https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=db2096&var-datasource=thanos&var-cluster=mysql&viewPanel=28&from=now-100d&to=now but let's wait for 20 more days until the binlog expiration time (30 days) is back so we can see how stable it is.
https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=db2096&var-datasource=thanos&var-cluster=mysql&viewPanel=28&from=1695279043258&to=1703055043258 this is looking a lot better even though the growth has definitely changed compared to the last 3 months.
It has been stabilized: https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=db2096&var-datasource=thanos&var-cluster=mysql&viewPanel=28&from=1700713903397&to=1704169903397
Shall we close it or you think further clean ups are needed?
Yep, this seems fine
/dev/mapper/tank-data xfs 3.6T 1.2T 2.5T 33% /srv
Thanks everyone!
