Page MenuHomePhabricator
Create Task
Maniphest T300171

Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab
Open, In Progress, MediumPublic
Actions

Description

After T281538 is done and created the new kubernetes service miscweb, this ticket is about removing the (remaining) micro sites from "miscweb on Ganeti VMs" (miscweb1/2* machines) to the kubernetes service with the name. With the long-term goal to completely remove the miscweb* VMs.

Also part of this ticket is removing static-bugzilla code and other sites that are moved completely from the puppet repo.

The first site that has already moved was https://static-bugzilla.wikimedia.org/, it's still hosted on Gerrit so still needs to be migrated to GitLab.

One medium-term goal will be moving 3 additional sites since this is an OKR in Betterworks for this quarter.

Sites currently on the ganeti VMs are:

On Kubernetes but needs migrating from Gerrit to GitLab

This is a mix of static sites and apps though. That is because we merged "webserver misc static" and "webserver misc apps" into a single role in the past.

For the kubernetes service miscweb only the static sites are expected to move there, not the applications.

Details

SubjectRepoBranchLines +/-
miscweb: also set requests for bugzilla stagingoperations/deployment-chartsmaster+3 -0
miscweb: set requests and limit for bugzilla stagingoperations/deployment-chartsmaster+4 -0
miscweb: migrate bugzilla image to GitLaboperations/deployment-chartsmaster+2 -4
miscweb: use timestamp in image tagsoperations/deployment-chartsmaster+4 -4
miscweb: remove static_tendril classes and filesoperations/puppetproduction+0 -163
miscweb: add release statictendril to eqiad and codfwoperations/deployment-chartsmaster+2 -2
miscweb: add statictendril release to miscweb stagingoperations/deployment-chartsmaster+23 -1
miscweb: move tests for static_tendril to k8s tests fileoperations/puppetproduction+17 -16
admin: add all miscweb domains as extra SANsoperations/deployment-chartsmaster+11 -0
microsites: remove bienvenida.wikimedia.org, migrated to k8soperations/puppetproduction+11 -55
microsites: remove http blackbox monitor for 15.wikipedia.orgoperations/puppetproduction+0 -10
httpbb: move tests for bienvenida.wikimedia.org to miscweb-k8soperations/puppetproduction+10 -9
trafficserver: switch bienvenida.wikimedia.org backendoperations/puppetproduction+1 -1
microsites: move blackbox checks to dedicated monitoring profileoperations/puppetproduction+29 -1
microsites: remove annualreport, migrated to k8soperations/puppetproduction+0 -74
service::catalog add miscweb 15 and annual to service catalogoperations/puppetproduction+31 -2
miscweb: set ipv4 and port for 15 and annual blackbox checkoperations/puppetproduction+14 -0
Revert "Revert "miscweb: set ipv4 and ipv6 for 15 and annual blackbox check""operations/puppetproduction+12 -0
miscweb: set ipv4 and ipv6 for 15 and annual blackbox checkoperations/puppetproduction+14 -0
trafficserver: switch annual.wikimedia.org backendoperations/puppetproduction+29 -29
microsites: remove 15.wikipedia.orgoperations/puppetproduction+0 -30
trafficserver: switch 15.wikipedia.org backendoperations/puppetproduction+1 -1
httpbb: move tests for 15.wikipedia.orgoperations/puppetproduction+8 -8
service: add host to miscweb http probeoperations/puppetproduction+1 -0
add 15.wikipedia to cert and gateway hosts for miscweb behind istio ingressoperations/deployment-chartsmaster+3 -0
miscweb: add annualreport release to eqiad and codfwoperations/deployment-chartsmaster+2 -2
miscweb: fix image name for annual report micrositeoperations/deployment-chartsmaster+1 -1
miscweb: set lower resource requests for annual report micrositeoperations/deployment-chartsmaster+3 -0
miscweb: add annualreport release to miscweboperations/deployment-chartsmaster+46 -30
setup build pipelines for bzstatic and fifteenwp variantsoperations/container/miscwebmaster+8 -3
rename the production variant to bzstaticoperations/container/miscwebmaster+1 -1
define entrypoint only once instead of in each variant, simplify test variantoperations/container/miscwebmaster+3 -16
move html and httpd config for 15.wp to own directory, reorganize variantsoperations/container/miscwebmaster+61 -6
load apache module setenvifoperations/container/miscwebmaster+3 -0
blubber: copy config for 15.wikipedia into containeroperations/container/miscwebmaster+34 -0
add httpd config for 15.wikipedia.orgoperations/container/miscwebmaster+27 -0
add content of the annual report 2015 site, aka 15.wikipedia.orgoperations/container/miscwebmaster+31 K -0
backup: remove fileset for static-bugzillaoperations/puppetproduction+0 -3
httpbb: move tests for static-bugzilla to new file for miscweb-k8soperations/puppetproduction+10 -8
delete bugzilla_static after it moved from puppet to k8soperations/puppetproduction+0 -125
Show related patches Customize query in gerrit
TitleReferenceAuthorSource BranchDest Branch
replace reviewer jelto with wmf-sre-collab grouprepos/sre/miscweb/research-landing-page!15jeltoreplace-jelto-readmemaster
add .gitlab-ci.ymlrepos/sre/miscweb/bugzilla!1jeltoadd-cimain
add /repos/sre/miscweb/bugzilla to Trusted Runnersrepos/releng/gitlab-trusted-runner!42jeltoadd-bugzillamain
use timestamp in image tagrepos/sre/miscweb/annualreport!5jeltouse-timestamp-tagsmaster
use timestamp in image tagrepos/sre/miscweb/bienvenida!2jeltouse-timestamp-tagsmaster
use timestamp in image tagrepos/sre/miscweb/transparencyreport!2jeltouse-timestamp-tagsmaster
allow annualreport on trusted runnersrepos/releng/gitlab-trusted-runner!22jeltoadd-miscweb-annualreportmain
Add Ci image build and publishrepos/sre/miscweb/annualreport!2jeltoci-image-buildmaster
add blubber image build for annual report sitesrepos/sre/miscweb/annualreport!1jeltoblubber-buildmaster
Customize query in GitLab

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Dzahn added a comment.Jun 8 2023, 12:48 AM

updated the list of sites on miscweb* in this ticket to the current situation.

all classes under modules/profile/manifests/microsites/ are now covered here with one exception, peopleweb* which is used on people* VMs and not miscweb VMs despite being called a microsite

Dzahn updated the task description. (Show Details)Jun 8 2023, 12:48 AM
Dzahn updated the task description. (Show Details)
Jelto updated the task description. (Show Details)Jun 12 2023, 11:53 AM
gerritbot added a comment.Jun 17 2023, 12:04 AM

Change 930886 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/deployment-charts@master] miscweb: add statictendril release to miscweb staging

https://gerrit.wikimedia.org/r/930886

gerritbot added a comment.Jun 17 2023, 12:09 AM

Change 930887 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/deployment-charts@master] miscweb: add release statictendril to eqiad and codfw

https://gerrit.wikimedia.org/r/930887

Krinkle unsubscribed.Jun 19 2023, 12:12 PM
gerritbot added a comment.Jun 21 2023, 6:35 AM

Change 927998 merged by jenkins-bot:

[operations/deployment-charts@master] admin: add all miscweb domains as extra SANs

https://gerrit.wikimedia.org/r/927998

Dzahn updated the task description. (Show Details)Jun 21 2023, 6:08 PM
Dzahn changed the task status from Open to In Progress.Jun 21 2023, 8:56 PM
Jelto closed subtask T338781: move micro site transparency.wikimedia.org to kubernetes as Resolved.Jun 22 2023, 7:46 AM
gerritbot added a comment.Jun 22 2023, 9:27 PM

Change 932337 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: remove static_tendril classes and files

https://gerrit.wikimedia.org/r/932337

gerritbot added a comment.Jun 22 2023, 9:30 PM

Change 932338 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: move tests for static_tendril to k8s tests file

https://gerrit.wikimedia.org/r/932338

Dzahn added a comment.Jun 22 2023, 9:46 PM

created https://gitlab.wikimedia.org/repos/sre/miscweb/statictendril/-/tree/main for statictendril

Jelto mentioned this in T340182: move micro tendril.wikimedia.org and dbtree.wikimedia.org to kubernetes.Jun 23 2023, 9:06 AM
Jelto updated the task description. (Show Details)
In T300171#8957573, @Dzahn wrote:

created https://gitlab.wikimedia.org/repos/sre/miscweb/statictendril/-/tree/main for statictendril

Thanks! I opened T340182 to have a tracking task similar to the other services. I'll update the Bug in existing changes.

Dzahn removed Dzahn as the assignee of this task.Jun 23 2023, 3:02 PM

ok, thanks! Let me give this tracking / parent ticket back to the group

LSobanski moved this task from Work in Progress to Work in Progress (Tracking tasks) on the collaboration-services board.Jun 26 2023, 10:33 AM
gerritbot added a comment.Jun 27 2023, 10:20 PM

Change 932338 merged by Dzahn:

[operations/puppet@production] miscweb: move tests for static_tendril to k8s tests file

https://gerrit.wikimedia.org/r/932338

LSobanski renamed this task from move micro sites from ganeti to kubernetes to Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab.Jun 29 2023, 1:57 PM
LSobanski updated the task description. (Show Details)
LSobanski added a project: GitLab (Pipeline Services MigrationšŸ¤).Jun 29 2023, 2:00 PM
thcipriani moved this task from Backlog to Doing on the GitLab (Pipeline Services MigrationšŸ¤) board.Jun 29 2023, 2:00 PM
gerritbot added a comment.Jul 10 2023, 5:19 PM

Change 930886 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: add statictendril release to miscweb staging

https://gerrit.wikimedia.org/r/930886

Dzahn changed the status of subtask T340182: move micro tendril.wikimedia.org and dbtree.wikimedia.org to kubernetes from Open to In Progress.Jul 10 2023, 5:28 PM
gerritbot added a comment.Jul 10 2023, 7:21 PM

Change 930887 abandoned by Dzahn:

[operations/deployment-charts@master] miscweb: add release statictendril to eqiad and codfw

Reason:

duplicate of https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/936775

https://gerrit.wikimedia.org/r/930887

Dzahn updated the task description. (Show Details)Jul 10 2023, 7:26 PM

https://dbtree.wikimedia.org and https://tendril.wikimedia.org succesfully moved to k8s today. (this is "statictendril")

gerritbot added a comment.Jul 10 2023, 8:12 PM

Change 932337 merged by Dzahn:

[operations/puppet@production] miscweb: remove static_tendril classes and files

https://gerrit.wikimedia.org/r/932337

Dzahn closed subtask T340182: move micro tendril.wikimedia.org and dbtree.wikimedia.org to kubernetes as Resolved.Jul 10 2023, 8:15 PM
CodeReviewBot added a comment.Jul 13 2023, 9:27 AM

jelto opened https://gitlab.wikimedia.org/repos/sre/miscweb/transparencyreport/-/merge_requests/2

use timestamp in image tag

CodeReviewBot added a comment.Jul 13 2023, 11:11 AM

jelto merged https://gitlab.wikimedia.org/repos/sre/miscweb/transparencyreport/-/merge_requests/2

use timestamp in image tag

CodeReviewBot added a comment.Jul 13 2023, 11:23 AM

jelto opened https://gitlab.wikimedia.org/repos/sre/miscweb/bienvenida/-/merge_requests/2

use timestamp in image tag

CodeReviewBot added a comment.Jul 13 2023, 11:30 AM

jelto merged https://gitlab.wikimedia.org/repos/sre/miscweb/bienvenida/-/merge_requests/2

use timestamp in image tag

CodeReviewBot added a comment.Jul 13 2023, 11:43 AM

jelto opened https://gitlab.wikimedia.org/repos/sre/miscweb/annualreport/-/merge_requests/5

use timestamp in image tag

CodeReviewBot added a comment.Jul 13 2023, 11:48 AM

jelto merged https://gitlab.wikimedia.org/repos/sre/miscweb/annualreport/-/merge_requests/5

use timestamp in image tag

gerritbot added a comment.Jul 13 2023, 12:00 PM

Change 937938 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb: use timestamp in image tags

https://gerrit.wikimedia.org/r/937938

gerritbot added a comment.Jul 13 2023, 7:00 PM

Change 937938 merged by Dzahn:

[operations/deployment-charts@master] miscweb: use timestamp in image tags

https://gerrit.wikimedia.org/r/937938

Maintenance_bot removed a project: Patch-For-Review.Jul 13 2023, 7:10 PM
Jelto claimed this task.Jul 14 2023, 10:52 AM

Thanks for merging the above change @Dzahn . I deployed the change to all wikikube clusters. So we are using the same release workflow and tags for all services now (this issue was raised in T340182#8992829). One exception is bugzilla, but that service is still on Gerrit. Once we migrated this to GitLab, it should be similar as well.

kubectl get deployment -o=jsonpath="{range .items[*]}{'\n'}{.metadata.name}{':\t'}{range .spec.template.spec.containers[0]}{.image}{', '}{end}{end}"

miscweb-annualreport:   docker-registry.discovery.wmnet/repos/sre/miscweb/annualreport:2023-07-13-114826, 
miscweb-bienvenida:     docker-registry.discovery.wmnet/repos/sre/miscweb/bienvenida:2023-07-13-113046, 
miscweb-statictendril:  docker-registry.discovery.wmnet/repos/sre/miscweb/statictendril:2023-07-13-115511, 
miscweb-transparencyreport:     docker-registry.discovery.wmnet/repos/sre/miscweb/transparencyreport:2023-07-13-111133

I'll claim the task for now.

Dzahn added a subtask: T334511: Move research webpages to gitlab.Jul 14 2023, 4:14 PM
Dzahn mentioned this in T334511: Move research webpages to gitlab.Jul 14 2023, 4:21 PM
Dzahn added a comment.Jul 14 2023, 4:50 PM

Yep! So regarding static-bugzilla on Gerrit. That content repo is _quite large_ compared to any other microsite. Because there are thousands and thousands of HTML pages in there, though they are gzipped. I remember splitting the original upload into several patches because it was too much to have 50k files in a single Gerrit patch. So we should be careful when trying to import that into gitlab.

LSobanski updated Other Assignee, removed: Jelto.Jul 31 2023, 10:02 AM
Jelto updated the task description. (Show Details)Aug 14 2023, 9:35 AM
CodeReviewBot added a project: Patch-For-Review.Aug 24 2023, 1:39 PM

jelto opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/42

add /repos/sre/miscweb/bugzilla to Trusted Runners

CodeReviewBot added a comment.Aug 24 2023, 1:45 PM

jelto merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/42

add /repos/sre/miscweb/bugzilla to Trusted Runners

CodeReviewBot added a comment.Aug 24 2023, 1:53 PM

jelto opened https://gitlab.wikimedia.org/repos/sre/miscweb/bugzilla/-/merge_requests/1

add .gitlab-ci.yml

CodeReviewBot added a comment.Aug 24 2023, 1:54 PM

jelto merged https://gitlab.wikimedia.org/repos/sre/miscweb/bugzilla/-/merge_requests/1

add .gitlab-ci.yml

Jelto added a comment.Aug 24 2023, 2:04 PM
In T300171#9016375, @Dzahn wrote:

Yep! So regarding static-bugzilla on Gerrit. That content repo is _quite large_ compared to any other microsite. Because there are thousands and thousands of HTML pages in there, though they are gzipped. I remember splitting the original upload into several patches because it was too much to have 50k files in a single Gerrit patch. So we should be careful when trying to import that into gitlab.

I refactored the repo a bit and zipped all html files into a single gz file. This gz file is around 440Mb in size and stored in GitLabs LFS.
During image build this gz archive gets extracted and added to the image (note: I had to use multi stage builds to make sure the archive does not end up in some of the docker layers).

See also T343914 for some more reasoning for the refactoring.

Jelto added a subtask: T343914: Evaluate miscweb bugzilla architecture.Aug 24 2023, 2:04 PM
Jelto mentioned this in T343914: Evaluate miscweb bugzilla architecture.Aug 24 2023, 2:29 PM
gerritbot added a comment.Aug 24 2023, 2:38 PM

Change 952228 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb: migrate bugzilla image to GitLab

https://gerrit.wikimedia.org/r/952228

gerritbot added a comment.Aug 25 2023, 7:18 AM

Change 952228 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: migrate bugzilla image to GitLab

https://gerrit.wikimedia.org/r/952228

Jelto updated the task description. (Show Details)Aug 25 2023, 1:20 PM
Jelto closed subtask T343914: Evaluate miscweb bugzilla architecture as Resolved.Aug 29 2023, 10:45 AM
Jelto changed the status of subtask T334511: Move research webpages to gitlab from Open to Stalled.Aug 29 2023, 11:39 AM
fkaelin changed the status of subtask T334511: Move research webpages to gitlab from Stalled to In Progress.Aug 29 2023, 1:31 PM
fkaelin closed subtask T334511: Move research webpages to gitlab as Resolved.Sep 5 2023, 1:26 PM
Jelto updated the task description. (Show Details)Sep 11 2023, 8:38 AM
Jelto mentioned this in T346309: move static-codereview.wikimedia.org to kubernetes.Sep 14 2023, 9:15 AM
Jelto added a subtask: T346309: move static-codereview.wikimedia.org to kubernetes.
Jelto updated the task description. (Show Details)
Jelto closed subtask T346309: move static-codereview.wikimedia.org to kubernetes as Resolved.Sep 19 2023, 1:50 PM
Jelto updated the task description. (Show Details)
DDeSouza subscribed.Oct 29 2023, 8:30 PM
Jelto updated the task description. (Show Details)Nov 8 2023, 3:08 PM
Jelto updated the task description. (Show Details)Nov 8 2023, 3:12 PM
Jelto updated the task description. (Show Details)Nov 8 2023, 3:17 PM
Jelto updated the task description. (Show Details)Nov 8 2023, 3:19 PM
CodeReviewBot added a comment.Dec 8 2023, 9:52 AM

jelto opened https://gitlab.wikimedia.org/repos/sre/miscweb/research-landing-page/-/merge_requests/15

replace reviewer jelto with wmf-sre-collab group

CodeReviewBot added a comment.Dec 8 2023, 1:35 PM

dani merged https://gitlab.wikimedia.org/repos/sre/miscweb/research-landing-page/-/merge_requests/15

replace reviewer jelto with wmf-sre-collab group

Maintenance_bot removed a project: Patch-For-Review.Dec 8 2023, 2:10 PM
gerritbot added a comment.Jan 8 2024, 1:47 PM

Change 988495 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb: set requests and limit for bugzilla staging

https://gerrit.wikimedia.org/r/988495

gerritbot added a project: Patch-For-Review.Jan 8 2024, 1:47 PM
gerritbot added a comment.Jan 8 2024, 2:01 PM

Change 988495 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: set requests and limit for bugzilla staging

https://gerrit.wikimedia.org/r/988495

gerritbot added a comment.Jan 8 2024, 2:14 PM

Change 988499 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb: also set requests for bugzilla staging

https://gerrit.wikimedia.org/r/988499

gerritbot added a comment.Jan 8 2024, 3:39 PM

Change 988499 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: also set requests for bugzilla staging

https://gerrit.wikimedia.org/r/988499

Maintenance_bot removed a project: Patch-For-Review.Jan 8 2024, 4:32 PM
Jelto closed subtask T350791: move design.wikimedia.org to kubernetes as Resolved.Jan 18 2024, 2:39 PM
Jelto added a subtask: T357413: miscweb wikikube staging namespace exceeding quota limits.Feb 13 2024, 2:02 PM
Jelto closed subtask T357413: miscweb wikikube staging namespace exceeding quota limits as Resolved.Feb 14 2024, 2:11 PM
Jelto updated the task description. (Show Details)Feb 29 2024, 1:28 PM
Jelto mentioned this in T361090: Move k8s miscweb blackbox checks out of microsites puppet module.Mar 27 2024, 11:06 AM
Jelto added a subtask: T361090: Move k8s miscweb blackbox checks out of microsites puppet module.
Jelto changed the status of subtask T361090: Move k8s miscweb blackbox checks out of microsites puppet module from Open to Stalled.Tue, Apr 16, 1:57 PM
Jelto updated the task description. (Show Details)Tue, Apr 16, 3:08 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. Ā· Wikimedia Foundation Ā· Privacy Policy Ā· Code of Conduct Ā· Terms of Use Ā· Disclaimer Ā· CC-BY-SA Ā· GPL