For the "other tools + striker" sync meeting that has no task, the notes are:

Apr 24, 2024 | Misc tools + Striker
Attendees: Arturo Borrero c_7b9ad6d28760abb302f0909412d1ed85b8d1db6ade03cbf2242fededb17164f1@group.calendar.google.com David Caro Raymond Ndibe Raymond Olisaemeka Taavi Väänänen

Notes

• Striker:
  • Written in django, current tool management interface
    • Has a small database with toolinfo data and some caching, membership requests
    • Some of the data there (ex. Wiki SUL account) should be moved to LDAP so we can share with bitu
  • Hosted in cloudweb production hosts, running on docker, moving to blubber
  • Authentication directly to ldap (directly write rights), eventually should change to idp/sso
  • Talks with gitlab/phabricator to create repos and projects
  • Has a docker-compose setup to run locally
  • Code -> https://gerrit.wikimedia.org/r/plugins/gitiles/labs/striker/
  • workflows need write ldap rights
    • Account creations (might move to bitu)
    • Account management (adding admin, disabling, …)
  • Ldap container image might need work soon as it does not seem maintained (https://phabricator.wikimedia.org/T340958)
  • Striker code needs some updating (ex. old version of django)
  • if Taavi is not around, Bryan can help with patch review & maintenance. He should be familiar with the codebase.
• Misc tools
  • https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Infrastructure_tools
  • https://toolhub.wikimedia.org/lists/192
  • Some that Taavi touched recently are:
    • K8s-status
    • Openstack-browser
    • Fourohfour
    • Replag
    • Db-names

Action items
*

For the "wikireplicas" meeting, the notes are:

May 7, 2024 | Wiki replicas
Attendees: c_7b9ad6d28760abb302f0909412d1ed85b8d1db6ade03cbf2242fededb17164f1@group.calendar.google.com David Caro Francesco Negri Joanna Borun Taavi Väänänen

Notes

• Data persistence takes care of getting the data already redacted in the dbs
• Mainly maintaining the scripts that keep the views layer to hide some of the tables/columns on top of the above
  • We do maintain the script and run it
  • The decision of what should be public and what's not is tricky and the process does not go very well (ex. https://phabricator.wikimedia.org/T344108)
  • Config example: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/templates/wmcs/db/wikireplicas/maintain-views.yaml
    • ^source of truth of where the views are defined
    • Applied by the script: https://gerrit.wikimedia.org/g/operations/puppet/+/refs/heads/production/modules/profile/files/wmcs/db/wikireplicas/views/maintain-views.py
      • There's a cookbook to apply it on all the hosts
      • Issues:
        • If there's a long-running query it will not be able to update the views as it needs a lock, fixed with automatic depooling is not very reliably (https://phabricator.wikimedia.org/T300427)
          • Otherwise you have to kill the queries that are stuck
        • Avoid unnecessary table recreations / diff of what needs to be created (https://phabricator.wikimedia.org/T351637)
        • Alert if the script has not been run in a while (probably using the dry-run/diff change above to check if changes are needed)
• Workboard: https://phabricator.wikimedia.org/project/board/2874/
  • Current priority task (has been stuck for a bit https://phabricator.wikimedia.org/T344108)
• Some common errors
  • Sometimes an index get removed/changed in mediawiki or in the views, and queries that were fast start getting slow
    • Maybe able to detect by tracking query times
• Need to check any tables that are exposed in the wikireplicas, though probably data persistence might already be doing so (verify)
  • Sync with the new team (adam baso) on slack/irc
• There's another script, maintain_replica_indices that adds replica-specific indices
  • Very similar to the maintain_views script
  • Changes are less frequent
• Maintain-meta_p script adds metadata about the databases that are replicated, it's run by the cookbook to add a new wiki
• https://replag.toolforge.org/
  • Keeps track of the lag (most of the time expected)
• Useful tool https://orchestrator.wikimedia.org/web/cluster/alias/s1 (non-public)
• Wiki docs https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replicas

Action items

• Try to figure out some process to ingest tasks/diffs/changes needed to the views
• Add alerts to notice when the views are breaking/unapplied (people will notice very fast though, and sql errors are common so it's hard to automate)