Page MenuHomePhabricator

Install globaljsonlinks* tables on X1 for use with commons commons for Charts deployment
Closed, ResolvedPublic1 Estimated Story Points

Description

To provide usage tracking off tabular data and chart definitions used in the Chart extension (T374746), JsonConfig has been extended (T374747) with an optional cross-wiki usage tracking table set: globaljsonlinks, globaljsonlinks_target, and globaljsonlinks_wiki.

For the production environment these will live on x1 cluster.

(A test deployment [T378127] occurred to avoid polluting tables with pre-production data, and has a temporary installation of these tables on testcommonswiki)

  • Should this table be replicated to wiki replicas (does it not contain private data)?
    • ok to replicate (public derived data)
  • Will you be doing cross-joins with the wiki metadata?
    • no cross-joins, is meant to live standalone. references page_namespace/page_title indirectly on each side.
  • Size of the table (number of rows expected).
    • 1 row in globaljsonlinks per chart invocation, this is expected to be a few tens of thousands of rows
    • smaller number of rows in globaljsonlinks_wiki and globaljsonlinks_target which consolidate strings for source wikis (a few hundred) and target data page references (smaller than the total number of invocations)
  • Expected growth per year (number of rows).
    • relatively small in future, though it will grow more intensely at first as we migrate old Graphs invocations and create new Charts invocations. on the order of thousands of rows/year expected.
  • Expected amount of queries, both writes and reads (per minute, per hour...per day, any of those are ok).
    • makes at least one read query per LinksUpdate in job queue
    • makes inserts/deletes during LinksUpdate in job queue
    • updating a data page on Commons will trigger a read to distribute cache invalidation jobs
  • Examples of queries that will be using the table.
    • Mostly SELECTs joining the three globaljsonlinks tables in-place and issuance of fresh INSERTs when recording new links; these either return all matching links for a given outgoing source page or given incoming target page, or return id numbers of existing records.
    • No joins against remote tables, only between the tree globaljsonlinks* tables.
  • The release plan for the feature (are there specific wikis you'd like to test first etc).
    • we are running this setup on testcommonswiki as a temporary master for a test rollout
    • This production version will live on x1 in a standalone database to serve as the store for changes made on commons. Testing of the globallinks functionality with commons will occur on Test2 wiki

Event Timeline

Seddon triaged this task as High priority.Nov 18 2024, 7:46 PM
Seddon moved this task from Backlog to Up Next on the Charts board.
CCiufo-WMF set the point value for this task to 1.Nov 18 2024, 7:56 PM
CCiufo-WMF moved this task from Up Next to Sprint 11 on the Charts board.
CCiufo-WMF edited projects, added: Charts (Sprint 11); removed: Charts.
Seddon added a subscriber: CDanis.

testwiki + testcommons seems to have gone without a hitch so far. We'd like to proceed with testing the infra on commons.

@CDanis is happy to run the deployment of the tables. Just want to go through the formality of getting DBA approval.

You got the approval already. No need for second time!

Mentioned in SAL (#wikimedia-operations) [2024-11-20T14:29:46Z] <cdanis> T380226 💙cdanis@mwmaint2002.codfw.wmnet ~ 🕤☕ mwscript sql.php --wiki=commonswiki --cluster=extension1 /srv/mediawiki/php-1.44.0-wmf.4/extensions/JsonConfig/sql/mysql/tables-generated.sql

CDanis claimed this task.
💙cdanis@mwmaint2002.codfw.wmnet ~ 🕤☕ mwscript sql.php --wiki=commonswiki  --cluster=extension1  --query 'show tables like "%globaljsonlink%"'
stdClass Object
(
    [Tables_in_commonswiki (%globaljsonlink%)] => globaljsonlinks
)
stdClass Object
(
    [Tables_in_commonswiki (%globaljsonlink%)] => globaljsonlinks_target
)
stdClass Object
(
    [Tables_in_commonswiki (%globaljsonlink%)] => globaljsonlinks_wiki
)