Page MenuHomePhabricator

Create a table to store calculated properties for SuggestedInvestigations
Closed, ResolvedPublic

Description

Data generated from the users in cases needed for filtering in T433840: Implement case filtering on "edits on shared pages" for SuggestedInvestigations and subsequent similar filters will need to be persistently stored and queried against.

This table needs to store:

  • the case id this property belongs do
  • a key-value pair denoting a property being tracked (eg. "number of edits on shared pages")
    • key is an int, correlating to a string key defined in PHP
    • value is float to account for the possibility of decimal values in the future. All properties we're interested in so far are numeric.
  • a tinyint flag that denotes whether or not the value is considered stale

The primary key is a combination of the case id and the property id and no index should be needed as it should only ever be queried on primary key columns.


Should this table be replicated to wiki replicas (does it not contain private data)?

Probably not but this data itself is not private. Suggested Investigations is a private feature but the data we collect here is not derived from private values. It probably will not need to be in the near future, if ever.

Will you be doing cross-joins with the wiki metadata?

No.

Size of the table (number of rows expected).

As all cases are expected to have all properties, cusi_case * (property count + 1 stale indicator). As of writing, this works out to up to low 100k rows but is expected to grow as cases are generated.

Expected growth per year (number of rows).

It grows in direct proportion with cusi_case

Expected amount of queries, both writes and reads (per minute, per hour...per day, any of those are ok).

We're still workshopping this. The most ideal case would be to have the most up to date data, refreshing stale data as users take actions. There aren't many of them, proportional to the number of users/edits, but we'd like to instrument and step down fidelity as necessary based on what we see.

Examples of queries that will be using the table.

// Pull properties for display
SELECT * FROM cusi_case_property
  WHERE sicp_sic_id IN (1,2, 3);

// Update stale property
UPDATE cusi_case_property
SET sicp_stale = 0,
  sicp_value = 2
WHERE sicp_sic_id = 1
  AND sicp_property = 1;

The release plan for the feature (are there specific wikis you'd like to test first etc).

Suggested Investigations is live on enwiki and we'd want to implement there but we can start by only adding properties onto new cases with activity.

Event Timeline

Change #1320783 had a related patch set uploaded (by STran; author: STran):

[mediawiki/extensions/CheckUser@master] Add cusi_case_property table

https://gerrit.wikimedia.org/r/1320783

STran updated the task description. (Show Details)

Reviewing this:

  • sicp_property can be smallint instead of int I think, it'd save two bytes per row. Not a big deal though.
  • Please catalog the table (https://github.com/wikimedia/operations-puppet/blob/production/modules/mediawiki/files/mariadb/tables-catalog.yaml). I assume it'll be private which means some work before we can allow this to be deployed.
  • Please make sure you don't create the table on all or majority of wikis, I assume suggested investigations is only enabled in small set of wikis. Even if you enable it on any wiki that has at least one CU (or two per policy), it won't be more than 40 wikis (last I counted) meaning only 4% of wikis. Having one table on all wikis is causing pain on maintenance of s3.
  • Suggestion, the idea here implies that it grows without bound. Maybe set a system to purge the values after X years (similarly we drop echo notification after 5 years). Of course, it doesn't need to happen now.

// Pull properties for display
SELECT * FROM cusi_case_property
WHERE sicp_stale = 1
AND sicp_sic_id IN (1,2, 3);

Would this not be WHERE sicp_stale = 0 if we searching for properties to display (I presume we only show unstale properties for display)

Change #1324266 had a related patch set uploaded (by STran; author: STran):

[operations/puppet@production] Document cusi_case_property table

https://gerrit.wikimedia.org/r/1324266

sicp_property can be smallint instead of int I think, it'd save two bytes per row. Not a big deal though.

Updated

Please catalog the table (https://github.com/wikimedia/operations-puppet/blob/production/modules/mediawiki/files/mariadb/tables-catalog.yaml). I assume it'll be private which means some work before we can allow this to be deployed.

Done - Document cusi_case_property table

Please make sure you don't create the table on all or majority of wikis, I assume suggested investigations is only enabled in small set of wikis.

Yep, it'll use the SI dblist: https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/18e922e7946902c1e51ed38e5298985d98d9ca1c/dblists/checkuser-suggested-investigations.dblist

Suggestion, the idea here implies that it grows without bound. Maybe set a system to purge the values after X years (similarly we drop echo notification after 5 years). Of course, it doesn't need to happen now.

I think this is reasonable but should probably happen in a broader discussion about stale SI cases.

Would this not be WHERE sicp_stale = 0 if we searching for properties to display (I presume we only show unstale properties for display)

Copy pasta. I think display doesn't actually care as depending on how performant updating data is (or isn't), we might have to tolerate slightly stale data.

If there is a ticket for the getting rid of stale data. It has my sign off.

Change #1320783 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] Add cusi_case_property table

https://gerrit.wikimedia.org/r/1320783

Change #1324266 merged by Ladsgroup:

[operations/puppet@production] Document cusi_case_property table

https://gerrit.wikimedia.org/r/1324266