Page MenuHomePhabricator

Monitor the growth of CheckUser tables at large wikis
Closed, ResolvedPublic

Description

T253802 was deployed to the majority of our wikis now (list of excluded wikis is at T253802#6536344). This will increase database writes to cu_changes a little bit.

I would like to ask the DBA team to monitor the database growth, to make sure the change won't cause an issue when deployed to the rest of the wikis. We would like to deploy it to the rest of the wikis (ie. wikis listed at at T253802#6536344) in two weeks.

The monitoring will be done at the following wikis:

  • eswiki
  • metawiki
  • ruwiki

For easier access to the data, this table will be updated every time new data is retrieved and shared by the DBA team. Sizes are presented: "Compressed / Uncompressed"

wiki2020-10-062020-10-132020-10-202020-10-272020-11-03
eswiki143M / 863M145M / 878M147M / 892M151M / 921M153M / 941M
metawiki65M / 357M65M / 354M65M / 354M65M / 356M66M / 360M
ruwiki140M / 920M143M / 944M145M / 962M148M / 978M150M / 988M

Event Timeline

Urbanecm renamed this task from Monitor the growth of CheckUser databases at large wikis to Monitor the growth of CheckUser tables at large wikis.Oct 13 2020, 11:23 AM
Marostegui triaged this task as Medium priority.
Marostegui added a project: Data-Persistence.
Marostegui subscribed.

Thanks for creating this task!
I will do it for the following wikis:

  • eswiki
  • metawiki
  • ruwiki
Marostegui moved this task from Inbox to Epic on the Data-Persistence board.

@Marostegui quick ping that an update as of Oct 20th would be in order.

@Huji thanks for the ping. I have a calendar alert for this, but yesterday I was super busy and I couldn't do it, but it is on my radar.

@Marostegui let's say you pull data tomorrow, and you see the current trends continue; by current trends, I mean the uncompressed size increasing by 14MB a week for eswiki, and 20MB for ruwiki. What will be the next step? Do we extend monitoring for 4 more weeks and hope that a plateau a reached (ultimately, a plateau is expected to be reached in 90 days since the change). Do we just generalize it for the 90-day period and estimate what that plateau would be (which, right now, is estimate at 1.1GB for eswiki and 1.2 GB for ruwiki, uncompressed)? Or even try to generalize that to all other wikis?

What is too much growth? Do we have some external limits (for example in terms of disk size) that can govern how much growth we would allow?

@Huji 20MB for ruwiki means around 1GB per year at current growth (assuming it keeps growing the same rate). That is perfectly acceptable.
However, we do need to do this same exercise for the big wikis excluded at T253802#6536344 once it is enabled there.

There was a jump on eswiki, which grew 30MB in a week, but that's not too worrying I think. Even if it did grow like that it would be 1.5G a year.
Let's do one more monitoring week before closing this

I am going to close this as fixed per T265344#6583817

Thanks @Marostegui! So, I guess we can now enable that at the wikis excluded at T253802#6536344, and create a third monitoring task?

Yes, let's do that, I want to make sure enwiki and friends are monitored.
Thanks!