Page MenuHomePhabricator
Create Task
Maniphest T395266

[wikireplicas] Refactor maintenance scripts to allow local testing
Closed, ResolvedPublic
Actions

Description

As far as I know, maintain-views.py, maintain-meta_p.py and maintain_replica_indexes.py were never tested locally but always directly against a live wiki replica database.

This task is to track the required refactor to make it possible to run the scripts against a local database.

We should also add a basic Tox configuration to run automated tests.

Running the automated tests on the CI for operations/puppet is probably going to be more difficult, because they will require a MariaDB
container they can connect to, and I don't know how to run that in Jenkins/Zuul. We could consider migrating these scripts to a GitLab repo.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
sre.wikireplicas.add-wiki: update script pathsoperations/cookbooksmaster+3 -3
wikireplicas: install scripts from deb packageoperations/puppetproduction+16 -1 K
aptrepo: import wikireplicas-utils from apt-stagingoperations/puppetproduction+10 -0
wikireplicas scripts: setup pytest, add first testoperations/puppetproduction+1 K -10
wikireplicas: split db config into separate fileoperations/puppetproduction+1 K -21
wikireplicas: split db config into separate fileoperations/puppetproduction+64 -46
wikireplicas: remove dashes from script namesoperations/puppetproduction+3 -3
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
CI: use trusted runner for build_ci_debrepos/cloud/wikireplicas-utils!3fnegritrusted-runnermain
Add repos/cloud/wikireplicas-utilsrepos/releng/gitlab-trusted-runner!132fnegripatch-1main
Create .deb packagerepos/cloud/wikireplicas-utils!1fnegrideb-packagemain
Customize query in GitLab

Related Objects
Search...

Event Timeline

fnegri created this task.May 26 2025, 4:16 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMay 26 2025, 4:16 PM
fnegri triaged this task as Medium priority.May 26 2025, 4:17 PM
fnegri mentioned this in T351637: [wikireplicas] add proper dry-run/diff mode to maintain-views.
gerritbot added a comment.May 26 2025, 4:20 PM

Change #1148358 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] wikireplicas: remove dashes from script names

https://gerrit.wikimedia.org/r/1148358

gerritbot added a project: Patch-For-Review.May 26 2025, 4:20 PM

Change #1150715 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] wikireplicas: split db config from maintain-views

https://gerrit.wikimedia.org/r/1150715

gerritbot added a comment.May 26 2025, 4:20 PM

Change #1148394 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] wikireplicas scripts: setup pytest, add first test

https://gerrit.wikimedia.org/r/1148394

fnegri changed the task status from Open to In Progress.May 26 2025, 4:24 PM
fnegri moved this task from Backlog to Wiki replicas on the Data-Services board.
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2024/2025-Q3-Q4) board.
gerritbot added a comment.May 28 2025, 11:07 AM

Change #1148358 merged by FNegri:

[operations/puppet@production] wikireplicas: remove dashes from script names

https://gerrit.wikimedia.org/r/1148358

gerritbot added a comment.May 28 2025, 11:25 AM

Change #1151656 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] wikireplicas: split db config into separate file

https://gerrit.wikimedia.org/r/1151656

gerritbot added a comment.May 28 2025, 11:28 AM

Change #1150715 abandoned by FNegri:

[operations/puppet@production] wikireplicas: split db config into separate file

Reason:

Replaced by I59ab4faf8481d912dcb4bd9fa715df7893a78873

https://gerrit.wikimedia.org/r/1150715

fnegri changed the status of subtask T395432: [wikireplicas] Remove maintainviews and maintainindexes users from Open to In Progress.May 28 2025, 11:36 AM
fnegri closed subtask T395432: [wikireplicas] Remove maintainviews and maintainindexes users as Declined.Jun 5 2025, 1:41 PM
fnegri mentioned this in T395432: [wikireplicas] Remove maintainviews and maintainindexes users.
fnegri edited projects, added cloud-services-team (FY2025/2026-Q1-Q2); removed cloud-services-team (FY2024/2025-Q3-Q4).Jul 15 2025, 1:45 PM
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2025/2026-Q1-Q2) board.
gerritbot added a comment.Jul 31 2025, 9:52 AM

Change #1151656 abandoned by FNegri:

[operations/puppet@production] wikireplicas: split db config into separate file

Reason:

The db config used to be the only templated part, but now we also template the "fullviews" definition, so I need a different approach.

https://gerrit.wikimedia.org/r/1151656

gerritbot added a comment.Aug 1 2025, 10:58 AM

Change #1148394 abandoned by FNegri:

[operations/puppet@production] wikireplicas scripts: setup pytest, add first test

Reason:

I created https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils and I'm migrating the changes from this patch to there.

https://gerrit.wikimedia.org/r/1148394

CodeReviewBot added a comment.Aug 1 2025, 6:36 PM

fnegri opened https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1

Create .deb package

CodeReviewBot added a comment.Aug 12 2025, 2:57 PM

fnegri merged https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1

Create .deb package

CodeReviewBot added a comment.Aug 18 2025, 2:27 PM

fnegri opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/132

Add repos/cloud/wikireplicas-utils

CodeReviewBot added a comment.Aug 18 2025, 2:48 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/132

Add repos/cloud/wikireplicas-utils

gerritbot added a comment.Aug 18 2025, 3:54 PM

Change #1179728 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] aptrepo: import wikireplicas-utils from apt-staging

https://gerrit.wikimedia.org/r/1179728

CodeReviewBot added a comment.Aug 18 2025, 4:19 PM

fnegri opened https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/3

CI: use trusted runner for build_ci_deb

gerritbot added a comment.Aug 19 2025, 8:54 AM

Change #1179728 merged by FNegri:

[operations/puppet@production] aptrepo: import wikireplicas-utils from apt-staging

https://gerrit.wikimedia.org/r/1179728

gerritbot added a comment.Aug 19 2025, 11:45 AM

Change #1180121 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] wikireplicas: install scripts from deb package

https://gerrit.wikimedia.org/r/1180121

gerritbot added a comment.Aug 19 2025, 2:27 PM

Change #1180121 merged by FNegri:

[operations/puppet@production] wikireplicas: install scripts from deb package

https://gerrit.wikimedia.org/r/1180121

gerritbot added a comment.Aug 19 2025, 2:59 PM

Change #1180157 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/cookbooks@master] sre.wikireplicas.add-wiki: update script paths

https://gerrit.wikimedia.org/r/1180157

gerritbot added a comment.Aug 19 2025, 3:21 PM

Change #1180157 merged by jenkins-bot:

[operations/cookbooks@master] sre.wikireplicas.add-wiki: update script paths

https://gerrit.wikimedia.org/r/1180157

CodeReviewBot added a comment.Aug 20 2025, 2:49 PM

fnegri merged https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/3

CI: use trusted runner for build_ci_deb

fnegri closed this task as Resolved.Aug 28 2025, 12:49 PM
fnegri moved this task from In progress to Done on the cloud-services-team (FY2025/2026-Q1-Q2) board.

The scripts were migrated to the new https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils repo and they now have a basic testing setup including CI.
This task can be resolved, follow-up work will be tracked in separate tickets, e.g. T351637: [wikireplicas] add proper dry-run/diff mode to maintain-views.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits