Page MenuHomePhabricator

Add a high-throughput counter system to MediaWiki
Open, Needs TriagePublicFeature

Description

A common problem is that you want to keep track of the number of certain things; and while creating or deleting things generally modifies different database rows for each thing, all the actions try to modify the same counter, so you get peformance problems, deadlocks and timeouts, sometimes even outages.

Examples include edit counts, category membership counts (T365303), change tag counts.

One solution is to move these changes to a job, but that introduces random delays (and occasionally the jobs getting lost) which can be a mild annoyance for editors who sometimes rely on these counts (e.g. edit counts can determine permissions; non-empty maintenance categories are sometimes used as calls-to-action for admins). It might be worth coming up with a generic mechanism that's easy to add to new individual use cases.

Event Timeline

A strawman proposal: add a counter table, with id / type / target / count columns; the id is auto-incremented, the type is an enum-ish, the target is an ID whose meaning depends on the type (could be a user ID, page ID etc), the count is how much the counter should change. Inserting a row into the counter table doesn't block other threads inserting rows. A job periodically aggregates these rows into wherever the actual counter lives, and deletes them. We can show accurate counts by aggregating on the fly (a query with a small result set, and easy to cache).

The best solution on having a scalable counter is sharing. We have done this with site_stats (T306589: Add sharding to site_stats table) and it did wonders to lock stats of large wikis. The next important one that needs tackling is user_editcount which is one of the top lock times. I thought of having a dedicated user edit count table (which wouldn't have any rows for majority of users since they have zero edits, and then multiple rows for users with many edits, it could have even a randomizer that increases the sharding the more the user edits, maxed at ten for example). This would help on our db locks a lot and it could even save some space on user table.

The next important one that needs tackling is user_editcount which is one of the top lock times.

That's T237043: REST API - page history counts - Keep edit counts in separate database table and update on edit I guess?

I still think it would be nice to come up with a mechanism that doesn't require separate schema changes for every counter type. Maybe a more generic stats table, with a configurable number of shards per stat type / target?

The next important one that needs tackling is user_editcount which is one of the top lock times.

That's T237043: REST API - page history counts - Keep edit counts in separate database table and update on edit I guess?

I still think it would be nice to come up with a mechanism that doesn't require separate schema changes for every counter type. Maybe a more generic stats table, with a configurable number of shards per stat type / target?

See: T352823: Split user table to multiple tables

Aklapper changed the subtype of this task from "Task" to "Feature Request".Aug 26 2025, 6:40 AM

I proposed a redis-based counter system in T402952#11124299, specific to user edit counts but can be generalized. I am a bit disappointed that we're rigid in what backends we use and are trying to shoehorn everything - even the delta counts - into MariaDb which seems like a poor fit for it.

I think that If MariaDB with partitioned counters (or with delta + rollup) would give enough performance then Redis would not be needed but using it would add complexity. Ie. on how things are backed up, replicated, howto keep data in-sync ...

Change #1185984 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] kubernetes: Set default Envoy version to 1.26.8

https://gerrit.wikimedia.org/r/1185984

Change #1185984 merged by RLazarus:

[operations/puppet@production] kubernetes: Set default Envoy version to 1.26.8

https://gerrit.wikimedia.org/r/1185984

Hi! We need this for MediaWiki-extensions-CommunityRequests which will power the Community Wishlist. Our use case is essentially the same WikiBase with auto-generating IDs that are used in page titles

… I thought of having a dedicated user edit count table … This would help on our db locks a lot and it could even save some space on user table.

So are you saying we don't need to take into account the user edit count use case for this task?


Going off of @Tgr's proposed schema and the need for sharding, I'm proposing:

CREATE TABLE /*_*/counters (
  counter_id INT UNSIGNED AUTO_INCREMENT NOT NULL,
  counter_type VARCHAR(255) NOT NULL,
  counter_target INT UNSIGNED DEFAULT NULL,
  counter_count INT UNSIGNED DEFAULT 0 NOT NULL,
  counter_shard SMALLINT UNSIGNED NOT NULL,
  PRIMARY KEY(counter_id)
)

That would satisfy the Community Wishlist / WIkibase use-case, and various other kinds of counters too.

For counter_type, instead of using an ENUM which would require schema changes when we need new counters, would be a normal VARBINARY but with some level of enforcement on what is allowed on the PHP side. Maybe we have the core counters hard-coded somewhere as constants, and provide a hook for extensions to add new kinds of counters, and check against both before writing to the DB.

This "Counter" service could provide a method like ::generateNewId() that works just like Wikibase's, either creating a new ID for the given type and target, or increasing the existing one. With the sharding, I'm assuming we don't need to do a MySQL-specific method that uses LAST_INSERT_ID (T194299), right?

How does this sound? I believe Community-Tech would like to prioritize this as we are trying to get the new wishlist extension deployed ASAP. If the above looks OK, I'd love to take a stab at it.

Okay, scratch the above with respect to CommunityRequests. Amir has said we can move forward with our own ID generator and table. For whoever may work on this task, it'd be great to keep the Wikibase-esque use case in mind :)

Note: Even with sharding, unless we need to guarantee no ID is wasted (i.e. skipped) or consistency of counts, it may be a bad idea to have ID generation in the main transaction since a transaction may be long. If a counter is used frequently we should consider using a separate DB connection for it (prior art: T194299: Lock wait timeout exceeded in SqlIdGenerator::generateNewId).

Also we need to consider how this table should be replicated to cloud. Some counts may be considered private.

The proposed counter table can have two usages, one is for counting number of some sort of things (potentially replacing site_stats table, which can not be extended; note if we use it to store number of total edits, counter_count must be a bigint), another is to be used as a serial ID generator (sharded or unsharded), where we can replace the wb_id_counters table. Therefore, different rows will have very different writing and reading pattern (e.g. ss_total_edits will increase on every edit).

We should also consider we can have the ability to call a counter from another wiki (e.g. a central wiki like metawiki) or from a central database (x1 or CentralAuth database). One use case is potentially replacing user_autocreate_serial with the new counter system. To migrate a existing system to the new system, we need a way to sync (i.e. write both) them.

We can have an index on counter_count with given counter_type (ignoring counter_target), so the we can find the highest count of a given type (e.g. T344782: New special page to list users by highest edit count). If we want to support use cases like this, sharding for that specific counter_type is a bad idea.

Change #1191203 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] wikifeeds: Remove envoy image_version override

https://gerrit.wikimedia.org/r/1191203