@Danny_Benjafield_WMDE - it was mentioned by the WMF to contact the 'cloud announce mailing list'

@SuzanneWood-WMDE Thanks Suzie. Meeting with Lydia later today to discuss details and hopefully mail soon after that.

I have a tool that produces data relied upon by MSynAbot, which handles site-wide automatic page protections on Wikidata. The source can be found on the Toolforge NFS mount at /data/project/kvasir1/entityusage/entityusage.py and you can find the data it produces and a readme at https://tools-static.wmflabs.org/kvasir1/. It does read from the wbc_entity_usage table for every single foundation wiki having the Wikibase Client extension installed and enabled, but doesn't perform any joins. It also has a connection open to the Toolforge database to which it writes aggregated data to. It's a long running task which IIRC takes something like 3 hours to complete. It runs weekly.

@Ladsgroup mentioned in T424133 that the new tables might not be filled all at once but have data flowing into them over time. This will mean that MSynAbot will have to be taken offline for this period in order not to perform protections/unprotections based on faulty counts. For this reason we'd appreciate being told for which time-period the data in these tables are to be considered incomplete. @MisterSynergy

This was sent out to cloud-announce without a review from us first. Cloud-announce is meant to be a low-volume, high-signal list so we are somewhat picky about its contents. The main bit of confusion is that as far as I can read, the mail implying the data can be queried from x1 in the future, but x1 is not available on the replicas (T395881) and might not be for a while?

I think the idea was to gauge how much it's used and based on the usage and criticality of those usages decide to either:

• Just never replicate to wikireplicas anymore
• Move to x1 and let is be broken until wikireplicas are set up there
• Wait until wikireplicas are set up in x1 and then move

Hello @taavi , apologies for sending without a review (first-time poster, didn't see your handy advice page), would it be better to send this to the higher-volume cloud@ list? and may I email you directly for a review before sending?

Ladsgroup is correct, we'd like to get an idea of who / how much the table is being used, to factor into migration planning and mitigation.

We could add that using the MediaWiki API or recent changes API could act as a stopgap until a/if no new table is set up? Als that all queries to wbc_entity_usage will be affected, not just JOINs/cross-cluster.

I think cloud-announce@ is fine, just please see the guidelines I linked in my previous comment. Yes, feel free to reach out directly with a draft - experience has shown that an etherpad or a gdoc that allows editing is much more helpful than just an email with the contents that can't be edited inline.

I have sent that to you now, thanks so much

Danny will get in touch with Taavi