Page MenuHomePhabricator
Create Task
Maniphest T432721

Scripts and libraries repository consolidation
Open, Needs TriagePublic
Actions

Description

Based on past discussions we want to consolidate scripts and libraries that currently span across multiple repositories.
The main benefit are:

  • atomic deployment/checkout reducing the risk of out-of-sync dependencies
  • making changes spanning multiple libraries or scripts in a single commit or code review
  • running unit tests and mypy in CI across the whole codebase
  • easier refactoring to reduce duplication

In the cookbooks repository we are incrementally moving shared functions into __init__, and for repositories on GitLab we can:

Event Timeline

FCeratto-WMF created this task.Tue, Jul 21, 12:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Jul 21, 12:25 PM
FCeratto-WMF updated the task description. (Show Details)Tue, Jul 21, 2:37 PM
FCeratto-WMF added subscribers: CWilliams-WMF, Marostegui.Thu, Jul 23, 12:59 PM

@Marostegui @CWilliams-WMF it would be good this now that we don't have many MR open (so that it takes a short time to move them to the consolidated repo). Any thoughs?

CWilliams-WMF added a comment.Tue, Jul 28, 8:27 AM

@FCeratto-WMF where are you suggesting that these are rehomed to?

FCeratto-WMF added a comment.Tue, Jul 28, 8:41 AM

@CWilliams-WMF if we want to use one of the 4 repos perhaps "scripts" because it has a more generic name, otherwise we would need to create a new one

CWilliams-WMF added a comment.Tue, Jul 28, 9:28 AM

I would suggest a new repo:

  • tracking migration is easy, the import / script path will be different
  • old repos can be made read-only
FCeratto-WMF added a comment.Tue, Jul 28, 10:16 AM

It should be under the https://gitlab.wikimedia.org/repos/data_persistence/ maybe https://gitlab.wikimedia.org/repos/data_persistence/dbtools , ideally with a name that does not create confusion with scripts/wmfdb/wmfmariadbpy ... maybe just dbtools? mariadb-ops?

CWilliams-WMF added a comment.Tue, Jul 28, 11:54 AM

I would suggest that you name the repo so that the Python module path matches, which rules out mariadb-ops due to the hypen.

FCeratto-WMF added a comment.Tue, Jul 28, 1:00 PM

It's not a Python module, there are going to be different directories with tools and libraries in them, anyhow shall we just call it dbtools then?

Marostegui moved this task from Triage to Refine on the DBA board.Wed, Jul 29, 6:44 AM
CWilliams-WMF added a comment.Wed, Jul 29, 8:14 AM

It's not a Python module, there are going to be different directories with tools and libraries in them

Why not? Isn't that the point, to simplify and have one path to import from?

anyhow shall we just call it dbtools then?

Do you mean https://gitlab.wikimedia.org/repos/data_persistence/dbtools/dbtools ? That seems odd naming

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits