Different parts of Wikimedia production code write items to MainStash, the (DB-backed SqlBagOStuff running on x2). As a common resource shared by many teams, there's a risk of individual teams/products being irresponsible in how we use the common service. However, there's no built-in tooling to see what is actually resident for tenant teams (or SRE Data Persistence themselves), AFAIAA.
I suggest (if SRE Data Persistence agree!) adding some general-purpose resident-inventory measurement tooling to MediaWiki core to answer some questions: how many keys and how many bytes each keygroup (collection) occupies, the total and average footprint, and (by extracting this regularly) how any of this grows over time. I'm hoping to get agreement and input on shape and safety; if we're OK with this, I'm happy to lead the implementation.
Background
The idea is mainly about giving tenant teams (initially us in Abstract Wikipedia, but others too) the means to self-audit their own responsible use, as well as general operator visibility. The object-cache layer already emits bagostuff_* StatsFactory counters labelled by keygroup, but these are flow metrics (bytes read/written through the cache) — they never decrement on expiry or eviction, so they cannot answer "what is resident right now, and is it growing?". The BagOStuff interface exposes no enumeration, count, or size method; the only state-touching operation is deleteObjectsExpiringBefore(). To answer the proposed inventory questions will need aggregating the objectcache back-end directly, which no available tool does today.
Technical notes
Proposed shape, offered as a starting point for discussion rather than a fixed design:
- Add a read-only public method on the concrete SqlBagOStuff class (e.g. getKeyGroupStats()), structurally a read-only twin of deleteObjectsExpiringBefore(), re-using its existing per-(server shard × table shard) iteration rather than re-deriving the layout. This method will return per-keygroup resident key count and byte totals. Grouping is normalised in PHP to match determinekeyGroupForStats() exactly, so buckets line up with the existing bagostuff_* counter labels in Grafana.
- Query: Will this answer the questions we should be asking? Or are there other things we should measure too?
- Add a maintenance script with two modes: default human-readable audit to stdout (one-off "are we storing too much?"), and a --report-to-stats emitting resident StatsFactory gauges (mainstash_resident_keys, mainstash_resident_bytes, labelled keygroup) for a timer-driven dashboard showing evolution over time, set up as a mwcron job running every 24 hrs at production load low point.
- Query: Does this make sense? Also, would welcome bikeshedding on what we call the new StatsFactory counters here.
- The reads would always target a replica (new read-index parameter), never the write primary, as a SUM(OCTET_LENGTH(value)) full scan per table shard must stay off the x2 primary.
- Query: Ts it acceptable to run on a replica off-peak? Do we need shard-targeting/rate controls to spread the scan? (This is the main specific point on which SRE Data Persistence input is wanted.)
- Reported bytes could be one or the other (or both) of the on-disk, gzdeflate-compressed footprint (what actually stresses storage, replication, and backups), or the logical payload size.
- Query: Another check here that this sounds reasonable.
Acceptance criteria
- SRE Data Persistence agree the approach (replica full-scan, cadence, any shard-targeting/rate limits) is safe to run against x2, or steer it to a safer shape.
- Core gains a documented, maintenance/admin-only way to report resident MainStash inventory by keygroup (count and compressed bytes), consistent with existing keygroup metric labels.
- We run this as an mwcron and see the metrication flow in.
- AWT and other tenant teams are encouraged to review and track their inventory in Grafana from the StatsFactory gauges to monitor growth over time.