Page MenuHomePhabricator

Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab
Open, In Progress, MediumPublic

Description

After T281538 is done and created the new kubernetes service miscweb, this ticket is about removing the (remaining) micro sites from "miscweb on Ganeti VMs" (miscweb1/2* machines) to the kubernetes service with the name. With the long-term goal to completely remove the miscweb* VMs.

Also part of this ticket is removing static-bugzilla code and other sites that are moved completely from the puppet repo.

The first site that has already moved was https://static-bugzilla.wikimedia.org/, it's still hosted on Gerrit so still needs to be migrated to GitLab.

One medium-term goal will be moving 3 additional sites since this is an OKR in Betterworks for this quarter.

Sites currently on the ganeti VMs are:

On Kubernetes but needs migrating from Gerrit to GitLab

This is a mix of static sites and apps though. That is because we merged "webserver misc static" and "webserver misc apps" into a single role in the past.

For the kubernetes service miscweb only the static sites are expected to move there, not the applications.

Details

SubjectRepoBranchLines +/-
operations/deployment-chartsmaster+3 -0
operations/deployment-chartsmaster+4 -0
operations/deployment-chartsmaster+2 -4
operations/deployment-chartsmaster+4 -4
operations/puppetproduction+0 -163
operations/deployment-chartsmaster+2 -2
operations/deployment-chartsmaster+23 -1
operations/puppetproduction+17 -16
operations/deployment-chartsmaster+11 -0
operations/puppetproduction+11 -55
operations/puppetproduction+0 -10
operations/puppetproduction+10 -9
operations/puppetproduction+1 -1
operations/puppetproduction+29 -1
operations/puppetproduction+0 -74
operations/puppetproduction+31 -2
operations/puppetproduction+14 -0
operations/puppetproduction+12 -0
operations/puppetproduction+14 -0
operations/puppetproduction+29 -29
operations/puppetproduction+0 -30
operations/puppetproduction+1 -1
operations/puppetproduction+8 -8
operations/puppetproduction+1 -0
operations/deployment-chartsmaster+3 -0
operations/deployment-chartsmaster+2 -2
operations/deployment-chartsmaster+1 -1
operations/deployment-chartsmaster+3 -0
operations/deployment-chartsmaster+46 -30
operations/container/miscwebmaster+8 -3
operations/container/miscwebmaster+1 -1
operations/container/miscwebmaster+3 -16
operations/container/miscwebmaster+61 -6
operations/container/miscwebmaster+3 -0
operations/container/miscwebmaster+34 -0
operations/container/miscwebmaster+27 -0
operations/container/miscwebmaster+31 K -0
operations/puppetproduction+0 -3
operations/puppetproduction+10 -8
operations/puppetproduction+0 -125
Show related patches Customize query in gerrit
ReferenceSource BranchDest BranchAuthorTitle
repos/sre/miscweb/research-landing-page!15replace-jelto-readmemasterjeltoreplace reviewer jelto with wmf-sre-collab group
repos/sre/miscweb/bugzilla!1add-cimainjeltoadd .gitlab-ci.yml
repos/releng/gitlab-trusted-runner!42add-bugzillamainjeltoadd /repos/sre/miscweb/bugzilla to Trusted Runners
repos/sre/miscweb/annualreport!5use-timestamp-tagsmasterjeltouse timestamp in image tag
repos/sre/miscweb/bienvenida!2use-timestamp-tagsmasterjeltouse timestamp in image tag
repos/sre/miscweb/transparencyreport!2use-timestamp-tagsmasterjeltouse timestamp in image tag
repos/releng/gitlab-trusted-runner!22add-miscweb-annualreportmainjeltoallow annualreport on trusted runners
repos/sre/miscweb/annualreport!2ci-image-buildmasterjeltoAdd Ci image build and publish
repos/sre/miscweb/annualreport!1blubber-buildmasterjeltoadd blubber image build for annual report sites
Customize query in GitLab

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 927998 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] admin: add all miscweb domains as extra SANs

https://gerrit.wikimedia.org/r/927998

updated the list of sites on miscweb* in this ticket to the current situation.

all classes under modules/profile/manifests/microsites/ are now covered here with one exception, peopleweb* which is used on people* VMs and not miscweb VMs despite being called a microsite

Dzahn updated the task description. (Show Details)

Change 930886 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/deployment-charts@master] miscweb: add statictendril release to miscweb staging

https://gerrit.wikimedia.org/r/930886

Change 930887 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/deployment-charts@master] miscweb: add release statictendril to eqiad and codfw

https://gerrit.wikimedia.org/r/930887

Change 927998 merged by jenkins-bot:

[operations/deployment-charts@master] admin: add all miscweb domains as extra SANs

https://gerrit.wikimedia.org/r/927998

Dzahn changed the task status from Open to In Progress.Jun 21 2023, 8:56 PM

Change 932337 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: remove static_tendril classes and files

https://gerrit.wikimedia.org/r/932337

Change 932338 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: move tests for static_tendril to k8s tests file

https://gerrit.wikimedia.org/r/932338

Jelto updated the task description. (Show Details)

Thanks! I opened T340182 to have a tracking task similar to the other services. I'll update the Bug in existing changes.

Dzahn removed Dzahn as the assignee of this task.Jun 23 2023, 3:02 PM

ok, thanks! Let me give this tracking / parent ticket back to the group

Change 932338 merged by Dzahn:

[operations/puppet@production] miscweb: move tests for static_tendril to k8s tests file

https://gerrit.wikimedia.org/r/932338

LSobanski renamed this task from move micro sites from ganeti to kubernetes to Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab.Jun 29 2023, 1:57 PM
LSobanski updated the task description. (Show Details)

Change 930886 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: add statictendril release to miscweb staging

https://gerrit.wikimedia.org/r/930886

Change 930887 abandoned by Dzahn:

[operations/deployment-charts@master] miscweb: add release statictendril to eqiad and codfw

Reason:

duplicate of https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/936775

https://gerrit.wikimedia.org/r/930887

https://dbtree.wikimedia.org and https://tendril.wikimedia.org succesfully moved to k8s today. (this is "statictendril")

Change 932337 merged by Dzahn:

[operations/puppet@production] miscweb: remove static_tendril classes and files

https://gerrit.wikimedia.org/r/932337

Change 937938 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb: use timestamp in image tags

https://gerrit.wikimedia.org/r/937938

Change 937938 merged by Dzahn:

[operations/deployment-charts@master] miscweb: use timestamp in image tags

https://gerrit.wikimedia.org/r/937938

Thanks for merging the above change @Dzahn . I deployed the change to all wikikube clusters. So we are using the same release workflow and tags for all services now (this issue was raised in T340182#8992829). One exception is bugzilla, but that service is still on Gerrit. Once we migrated this to GitLab, it should be similar as well.

kubectl get deployment -o=jsonpath="{range .items[*]}{'\n'}{.metadata.name}{':\t'}{range .spec.template.spec.containers[0]}{.image}{', '}{end}{end}"

miscweb-annualreport:   docker-registry.discovery.wmnet/repos/sre/miscweb/annualreport:2023-07-13-114826, 
miscweb-bienvenida:     docker-registry.discovery.wmnet/repos/sre/miscweb/bienvenida:2023-07-13-113046, 
miscweb-statictendril:  docker-registry.discovery.wmnet/repos/sre/miscweb/statictendril:2023-07-13-115511, 
miscweb-transparencyreport:     docker-registry.discovery.wmnet/repos/sre/miscweb/transparencyreport:2023-07-13-111133

I'll claim the task for now.

Yep! So regarding static-bugzilla on Gerrit. That content repo is _quite large_ compared to any other microsite. Because there are thousands and thousands of HTML pages in there, though they are gzipped. I remember splitting the original upload into several patches because it was too much to have 50k files in a single Gerrit patch. So we should be careful when trying to import that into gitlab.

Yep! So regarding static-bugzilla on Gerrit. That content repo is _quite large_ compared to any other microsite. Because there are thousands and thousands of HTML pages in there, though they are gzipped. I remember splitting the original upload into several patches because it was too much to have 50k files in a single Gerrit patch. So we should be careful when trying to import that into gitlab.

I refactored the repo a bit and zipped all html files into a single gz file. This gz file is around 440Mb in size and stored in GitLabs LFS.
During image build this gz archive gets extracted and added to the image (note: I had to use multi stage builds to make sure the archive does not end up in some of the docker layers).

See also T343914 for some more reasoning for the refactoring.

Change 952228 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb: migrate bugzilla image to GitLab

https://gerrit.wikimedia.org/r/952228

Change 952228 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: migrate bugzilla image to GitLab

https://gerrit.wikimedia.org/r/952228

Change 988495 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb: set requests and limit for bugzilla staging

https://gerrit.wikimedia.org/r/988495

Change 988495 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: set requests and limit for bugzilla staging

https://gerrit.wikimedia.org/r/988495

Change 988499 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb: also set requests for bugzilla staging

https://gerrit.wikimedia.org/r/988499

Change 988499 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: also set requests for bugzilla staging

https://gerrit.wikimedia.org/r/988499