Page MenuHomePhabricator

Rename xLab to test kitchen
Closed, ResolvedPublic5 Estimated Story Points

Description

xLab: https://gitlab.wikimedia.org/repos/data-engineering/mpic

  • Rename repo (The name of the docker image depends on the repository name)
    • Update the repository name in GitLab
    • Update the repository name in the GitLab runner repository (merged+deployed)
    • Update references in the project itself (package.json and some comment) (merged)
    • Update devserver references in the TestKitchen extension (merged)
    • Update image name in kubernetes configuration for staging environment
    • Update image name in kubernetes configuration for production environment
    • Remove old images from docker registry
  • Update Makefiles (merged+deployed)
  • Update README + package.json (merged+deployed)
  • Update Docker compose files (merged+deployed)
    • Goals
    • Services
    • Images
  • Address Domain changes:
    • Kubernetes
    • SSO
  • Rename mpic-next kubernetes chart to test-kitchen-next (deployment-charts repository)
  • Update mpic kubernetes chart to test-kitchen (deployment-charts repository)
  • Consult with SREs for implementation details
  • Update database references (in the new test-kitchen chart/helmfiles)
  • Update codebase (a couple of variables use mpic as prefix: mpicAPI for example and some comments/configuration files say xLab or MPIC somewhere) (merged)
  • Update configuration files
    • Service name (merged+deployed)
    • Database configuration in staging
    • Database configuration in production
    • IDP configuration
  • Update OpenSearch dashboards
    • Staging
    • Production
  • Update Grafana dashboards
    • Staging
    • Production
  • Revise documentation
  • Revise Design Document

Details

Other Assignee
brouberol
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Santiago Facimediawiki/extensions/MetricsPlatformmaster+1 -1
Santiago Facimediawiki/extensions/TestKitchenmaster+7 -7
Brouberoloperations/deployment-chartsmaster+0 -14
Brouberoloperations/puppetproduction+12 -12
Brouberoloperations/puppetproduction+3 -3
Brouberoloperations/puppetproduction+0 -24
Brouberoloperations/puppetproduction+16 -16
Brouberoloperations/puppetproduction+0 -4
Brouberoloperations/puppetproduction+0 -8
Brouberoloperations/deployment-chartsmaster+0 -2 K
Brouberoloperations/deployment-chartsmaster+0 -189
Brouberoloperations/deployment-chartsmaster+87 -0
Brouberoloperations/deployment-chartsmaster+5 -0
Brouberoloperations/puppetproduction+0 -3
Brouberoloperations/deployment-chartsmaster+0 -1
Brouberoloperations/deployment-chartsmaster+99 -0
Brouberoloperations/deployment-chartsmaster+5 -0
Brouberoloperations/deployment-chartsmaster+12 -12
Brouberoloperations/deployment-chartsmaster+2 K -0
Brouberoloperations/deployment-chartsmaster+5 -5
Brouberoloperations/deployment-chartsmaster+5 -5
Brouberoloperations/deployment-chartsmaster+1 -1
Brouberoloperations/deployment-chartsmaster+0 -4
Brouberoloperations/deployment-chartsmaster+0 -2
Brouberoloperations/deployment-chartsmaster+2 -2
Santiago Facioperations/deployment-chartsmaster+1 -1
Santiago Facioperations/deployment-chartsmaster+1 -1
Clare Mingoperations/deployment-chartsmaster+1 -1
Clare Mingoperations/deployment-chartsmaster+1 -1
Brouberoloperations/deployment-chartsmaster+1 -1
Brouberoloperations/deployment-chartsmaster+4 -0
Brouberoloperations/deployment-chartsmaster+6 -0
Show related patches Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Renamed mpic repository to test-kitchenrepos/releng/gitlab-trusted-runner!139sfaciT407808-rename-xlabmain
Change the published image name to repos/data-engineering/test-kitchenrepos/data-engineering/test-kitchen!287brouberolT407808main
Renamed mpic => test-kitchenrepos/data-engineering/test-kitchen!286sfaciT407808-rename-mpic-test-kitchenmain
Updated some occurrences of mpic, MPIC and xLab in the codebaserepos/data-engineering/test-kitchen!282sfaciT407808-rename-mpicAPImain
Renamed local dev environment + READMErepos/data-engineering/test-kitchen!276sfaciT407808-rename-local-test-envmain
Customize query in GitLab

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Sfaci updated the task description. (Show Details)
Sfaci updated the task description. (Show Details)

Change #1226276 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Test Kitchen UI: Deploying v1.1.5 release to staging

https://gerrit.wikimedia.org/r/1226276

Change #1226276 merged by jenkins-bot:

[operations/deployment-charts@master] Test Kitchen UI: Deploying v1.1.5 release to staging

https://gerrit.wikimedia.org/r/1226276

Change #1226281 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Test Kitchen UI: Deploying v1.1.5 release to staging

https://gerrit.wikimedia.org/r/1226281

Change #1226281 merged by jenkins-bot:

[operations/deployment-charts@master] Test Kitchen UI: Deploying v1.1.5 release to production

https://gerrit.wikimedia.org/r/1226281

Change #1212426 abandoned by Brouberol:

[operations/deployment-charts@master] testkitchen: rename the OIDC services

Reason:

I'm starting from scratch, too many things have diverged since december.

https://gerrit.wikimedia.org/r/1212426

Change #1212425 abandoned by Brouberol:

[operations/deployment-charts@master] testkitchen: drop the mpic.w.o domains from the ingress gateways

Reason:

I'm starting from scratch, too many things have diverged since december.

https://gerrit.wikimedia.org/r/1212425

Change #1212424 abandoned by Brouberol:

[operations/deployment-charts@master] test-kitchen: drop the mpic.w.o SANs from the certificate

Reason:

I'm starting from scratch, too many things have diverged since december.

https://gerrit.wikimedia.org/r/1212424

Change #1212421 abandoned by Brouberol:

[operations/deployment-charts@master] test-kitchen: set the OIDC callback URL domain to test-kitchen.w.o

Reason:

I'm starting from scratch, too many things have diverged since december.

https://gerrit.wikimedia.org/r/1212421

Change #1212423 abandoned by Brouberol:

[operations/deployment-charts@master] Rename mpic service to test-kitchen

Reason:

I'm starting from scratch, too many things have diverged since december.

https://gerrit.wikimedia.org/r/1212423

Change #1212422 abandoned by Brouberol:

[operations/deployment-charts@master] Rename mpic-next service to test-kitchen-next

Reason:

I'm starting from scratch, too many things have diverged since december.

https://gerrit.wikimedia.org/r/1212422

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/mpic/-/merge_requests/287

Change the published image name to repos/data-engineering/test-kitchen

Change #1228521 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Definition of the test-kitchen chart

https://gerrit.wikimedia.org/r/1228521

Change #1228522 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Rename any reference of mpic into test-kitchen

https://gerrit.wikimedia.org/r/1228522

Change #1228528 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Define the test-kitchen-next namespace

https://gerrit.wikimedia.org/r/1228528

Change #1228529 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Define the test-kitchen-next service

https://gerrit.wikimedia.org/r/1228529

Change #1228530 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Define the test-kitchen namespace

https://gerrit.wikimedia.org/r/1228530

Change #1228531 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Define the test-kitchen service

https://gerrit.wikimedia.org/r/1228531

Change #1228521 merged by jenkins-bot:

[operations/deployment-charts@master] Definition of the test-kitchen chart

https://gerrit.wikimedia.org/r/1228521

Change #1228522 merged by jenkins-bot:

[operations/deployment-charts@master] Rename any reference of mpic into test-kitchen

https://gerrit.wikimedia.org/r/1228522

Change #1228528 merged by jenkins-bot:

[operations/deployment-charts@master] Define the test-kitchen-next namespace

https://gerrit.wikimedia.org/r/1228528

Change #1228529 merged by Brouberol:

[operations/deployment-charts@master] Define the test-kitchen-next service

https://gerrit.wikimedia.org/r/1228529

Change #1228542 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Fix duplicate discovery domain in ingress FQDNs

https://gerrit.wikimedia.org/r/1228542

Change #1228542 merged by Brouberol:

[operations/deployment-charts@master] Fix duplicate discovery domain in ingress FQDNs

https://gerrit.wikimedia.org/r/1228542

Change #1229078 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[mediawiki/extensions/TestKitchen@master] devserver: Updated according to the renaming

https://gerrit.wikimedia.org/r/1229078

Sfaci updated the task description. (Show Details)
Sfaci set the point value for this task to 5.

Change #1229087 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] trafficserver: remove mpic.w.o from the list of targets

https://gerrit.wikimedia.org/r/1229087

Change #1229087 merged by Brouberol:

[operations/puppet@production] trafficserver: remove mpic.w.o from the list of targets

https://gerrit.wikimedia.org/r/1229087

Change #1228530 merged by jenkins-bot:

[operations/deployment-charts@master] Define the test-kitchen namespace

https://gerrit.wikimedia.org/r/1228530

Change #1228531 merged by jenkins-bot:

[operations/deployment-charts@master] Define the test-kitchen service

https://gerrit.wikimedia.org/r/1228531

Change #1229096 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] dse-k8s-eqiad: remove mpic helmfiles

https://gerrit.wikimedia.org/r/1229096

Change #1229097 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Delete the mpic chart, that was replaced by test-kitchen

https://gerrit.wikimedia.org/r/1229097

Change #1229098 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] dse-k8s-eqiad: remove mpic namespaces

https://gerrit.wikimedia.org/r/1229098

Change #1229099 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] deployment_server: remove the mpic kubeconfigs

https://gerrit.wikimedia.org/r/1229099

Change #1229100 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] cache: remove caching rules related to the mpic domains

https://gerrit.wikimedia.org/r/1229100

Change #1229101 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] mariadb: replace mpic database names by their test_kitchen counterpart

https://gerrit.wikimedia.org/r/1229101

Change #1229102 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] service: drop the mpic services from hieradata

https://gerrit.wikimedia.org/r/1229102

Change #1229103 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] service_proxy: rename the mpic listener to test-kitchen

https://gerrit.wikimedia.org/r/1229103

Change #1229104 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] data_plaform/slos: rename mpic to test-kitchen

https://gerrit.wikimedia.org/r/1229104

Change #1229096 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s-eqiad: remove mpic helmfiles

https://gerrit.wikimedia.org/r/1229096

Change #1229097 merged by jenkins-bot:

[operations/deployment-charts@master] Delete the mpic chart, that was replaced by test-kitchen

https://gerrit.wikimedia.org/r/1229097

Change #1229099 merged by Brouberol:

[operations/puppet@production] deployment_server: remove the mpic kubeconfigs

https://gerrit.wikimedia.org/r/1229099

Change #1229100 merged by Brouberol:

[operations/puppet@production] cache: remove caching rules related to the mpic domains

https://gerrit.wikimedia.org/r/1229100

Change #1229101 merged by Brouberol:

[operations/puppet@production] mariadb: replace mpic database names by their test_kitchen counterpart

https://gerrit.wikimedia.org/r/1229101

Change #1229102 merged by Brouberol:

[operations/puppet@production] service: drop the mpic services from hieradata

https://gerrit.wikimedia.org/r/1229102

Change #1229103 merged by Brouberol:

[operations/puppet@production] service_proxy: rename the mpic listener to test-kitchen

https://gerrit.wikimedia.org/r/1229103

Change #1229104 merged by Brouberol:

[operations/puppet@production] data_plaform/slos: rename mpic to test-kitchen

https://gerrit.wikimedia.org/r/1229104

Change #1229098 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s-eqiad: remove mpic namespaces

https://gerrit.wikimedia.org/r/1229098

Here's a dump of my personal notes, now that this is all done.

MPIC -> test-kitchen migration

  • mpic-next -> test-kitchen redirection in place, let's keep it
  • mpic:
    • we currently have tk.w.o working EXCEPT for OIDC
    • enable OIDC login for tk.w.o instead of mpic (rename client id and update URL)
    • soft redirection for mpic.w.o -> tk.w.o
  • cleanup:
    • duplication the puppet secrets
    • ~[x] rename docker image from mpic to test-kitchen by adding PUBLISH_IMAGE_NAME: repos/data-engineering/test-kitchen and trigger a pipeline on main~
    • rename the repos/data-engineering/mpic repo to repos/data-engineering/test-kitchen and update the list repos trusted by the trusted runners
    • create the test-kitchen namespaces
      • staging
      • production
    • Create a new test-kitchen chart from the mpic chart
    • Stop the application and rename the database from mpic_{staging,production} to test_kitchen_{staging,production} (same with the associated DB user
      • staging
      • production
    • Create new test-kitchen services using the test-kitchen chart, with the test_kitchen database/user
      • staging
      • production
    • ~[ ] Update monitors~
    • update dashboards
    • update log streams
    • Update SLOs
    • Update service.yaml and service proxy
    • delete the mpic services
    • delete the mpic chart from deployment-charts
    • remove mpic services from service.yaml
    • remove mpic namespaces
    • Remove mpic kubeconfigs
    • Remove mpic* puppet secrets
    • Drop mpic databases
      • staging
      • production
    • ~[ ] Delete the repos/data-engineering/mpic docker image~
    • Delete the mpic chart from chartmuseum

⚠ we keep all mpic public DNS records for now

Database migration

staging

CREATE DATABASE test_kitchen_staging;

SELECT CONCAT(
  'RENAME TABLE mpic_staging.', table_name,
  ' TO test_kitchen_staging.', table_name, ';'
)
FROM information_schema.tables
WHERE table_schema = 'mpic_staging';

RENAME USER 'mpic_staging'@'%' TO 'test_kitchen_staging'@'%';

GRANT ALL PRIVILEGES ON test_kitchen_staging.* TO 'test_kitchen_staging'@'%';
FLUSH PRIVILEGES;

SHOW GRANTS FOR 'test_kitchen_staging'@'%';
SHOW DATABASES;

production

CREATE DATABASE test_kitchen_production;

SELECT CONCAT(
  'RENAME TABLE mpic_production.', table_name,
  ' TO test_kitchen_production.', table_name, ';'
)
FROM information_schema.tables
WHERE table_schema = 'mpic_production';

RENAME USER 'mpic_production'@'%' TO 'test_kitchen_production'@'%';

GRANT ALL PRIVILEGES ON test_kitchen_production.* TO 'test_kitchen_production'@'%';
FLUSH PRIVILEGES;

SHOW GRANTS FOR 'test_kitchen_production'@'%';
SHOW DATABASES;

Change #1229078 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@master] devserver: Updated according to the renaming

https://gerrit.wikimedia.org/r/1229078

Change #1233661 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[mediawiki/extensions/MetricsPlatform@master] Replaced mpic image with test-kitchen

https://gerrit.wikimedia.org/r/1233661

Change #1233661 merged by jenkins-bot:

[mediawiki/extensions/MetricsPlatform@master] Replaced mpic image with test-kitchen

https://gerrit.wikimedia.org/r/1233661