Page MenuHomePhabricator
Create Task
Maniphest T300171

Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab
Closed, ResolvedPublic
Actions

Description

After T281538 is done and created the new kubernetes service miscweb, this ticket is about removing the (remaining) micro sites from "miscweb on Ganeti VMs" (miscweb1/2* machines) to the kubernetes service with the name. With the long-term goal to completely remove the miscweb* VMs.

Also part of this ticket is removing static-bugzilla code and other sites that are moved completely from the puppet repo.

The first site that has already moved was https://static-bugzilla.wikimedia.org/, it's still hosted on Gerrit so still needs to be migrated to GitLab.

One medium-term goal will be moving 3 additional sites since this is an OKR in Betterworks for this quarter.

Sites currently on the ganeti VMs are:

On Kubernetes but needs migrating from Gerrit to GitLab

This is a mix of static sites and apps though. That is because we merged "webserver misc static" and "webserver misc apps" into a single role in the past.

For the kubernetes service miscweb only the static sites are expected to move there, not the applications.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
microsites/backup: remove rt-static backup filesetoperations/puppetproduction+0 -4
miscweb: remove static-rt profile from legacy miscweb roleoperations/puppetproduction+0 -2
update links to microsites source codeintegration/docrootmaster+3 -3
miscweb: bump base and mesh templates to newest versionoperations/deployment-chartsmaster+166 -41
miscweb: also set requests for bugzilla stagingoperations/deployment-chartsmaster+3 -0
miscweb: set requests and limit for bugzilla stagingoperations/deployment-chartsmaster+4 -0
miscweb: migrate bugzilla image to GitLaboperations/deployment-chartsmaster+2 -4
miscweb: use timestamp in image tagsoperations/deployment-chartsmaster+4 -4
miscweb: remove static_tendril classes and filesoperations/puppetproduction+0 -163
miscweb: add release statictendril to eqiad and codfwoperations/deployment-chartsmaster+2 -2
miscweb: add statictendril release to miscweb stagingoperations/deployment-chartsmaster+23 -1
miscweb: move tests for static_tendril to k8s tests fileoperations/puppetproduction+17 -16
admin: add all miscweb domains as extra SANsoperations/deployment-chartsmaster+11 -0
microsites: remove bienvenida.wikimedia.org, migrated to k8soperations/puppetproduction+11 -55
microsites: remove http blackbox monitor for 15.wikipedia.orgoperations/puppetproduction+0 -10
httpbb: move tests for bienvenida.wikimedia.org to miscweb-k8soperations/puppetproduction+10 -9
trafficserver: switch bienvenida.wikimedia.org backendoperations/puppetproduction+1 -1
microsites: move blackbox checks to dedicated monitoring profileoperations/puppetproduction+29 -1
microsites: remove annualreport, migrated to k8soperations/puppetproduction+0 -74
service::catalog add miscweb 15 and annual to service catalogoperations/puppetproduction+31 -2
miscweb: set ipv4 and port for 15 and annual blackbox checkoperations/puppetproduction+14 -0
Revert "Revert "miscweb: set ipv4 and ipv6 for 15 and annual blackbox check""operations/puppetproduction+12 -0
miscweb: set ipv4 and ipv6 for 15 and annual blackbox checkoperations/puppetproduction+14 -0
trafficserver: switch annual.wikimedia.org backendoperations/puppetproduction+29 -29
microsites: remove 15.wikipedia.orgoperations/puppetproduction+0 -30
trafficserver: switch 15.wikipedia.org backendoperations/puppetproduction+1 -1
httpbb: move tests for 15.wikipedia.orgoperations/puppetproduction+8 -8
service: add host to miscweb http probeoperations/puppetproduction+1 -0
add 15.wikipedia to cert and gateway hosts for miscweb behind istio ingressoperations/deployment-chartsmaster+3 -0
miscweb: add annualreport release to eqiad and codfwoperations/deployment-chartsmaster+2 -2
miscweb: fix image name for annual report micrositeoperations/deployment-chartsmaster+1 -1
miscweb: set lower resource requests for annual report micrositeoperations/deployment-chartsmaster+3 -0
miscweb: add annualreport release to miscweboperations/deployment-chartsmaster+46 -30
setup build pipelines for bzstatic and fifteenwp variantsoperations/container/miscwebmaster+8 -3
rename the production variant to bzstaticoperations/container/miscwebmaster+1 -1
define entrypoint only once instead of in each variant, simplify test variantoperations/container/miscwebmaster+3 -16
move html and httpd config for 15.wp to own directory, reorganize variantsoperations/container/miscwebmaster+61 -6
load apache module setenvifoperations/container/miscwebmaster+3 -0
blubber: copy config for 15.wikipedia into containeroperations/container/miscwebmaster+34 -0
add httpd config for 15.wikipedia.orgoperations/container/miscwebmaster+27 -0
add content of the annual report 2015 site, aka 15.wikipedia.orgoperations/container/miscwebmaster+31 K -0
backup: remove fileset for static-bugzillaoperations/puppetproduction+0 -3
httpbb: move tests for static-bugzilla to new file for miscweb-k8soperations/puppetproduction+10 -8
delete bugzilla_static after it moved from puppet to k8soperations/puppetproduction+0 -125
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedJeltoT300171 Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab
 ResolvedDzahnT337041 move micro site annual.wikimedia.org and 15.wikipedia.org to kubernetes
 ResolvedJeltoT337047 move micro site bienvenida.wikimedia.org to kubernetes
 ResolvedJeltoT338022 ProbeDown (miscweb bienvenida puppet cleanup)
 ResolvedDzahnT338064 decom sitemaps.wikimedia.org
 ResolvedJeltoT338781 move micro site transparency.wikimedia.org to kubernetes
 ResolvedDzahnT340182 move micro tendril.wikimedia.org and dbtree.wikimedia.org to kubernetes
 ResolvedfkaelinT334511 Move research webpages to gitlab
 ResolvedRequestClement_GoubertT343957 Requesting access to deployment for fkaelin
 ResolvedSecurityJeltoT343914 Evaluate miscweb bugzilla architecture
 ResolvedJeltoT346309 move static-codereview.wikimedia.org to kubernetes
 ResolvedJeltoT350791 move design.wikimedia.org to kubernetes
 ResolvedVolker_ET352202 Final deploy of DSG
 ResolvedJeltoT350793 move query.wikidata.org to kubernetes
 ResolvedBUG REPORTJeltoT385728 Wikidata Query Builder is not loading
 ResolvedJeltoT387199 Archive wikidata/query/gui Gerrit repository
 ResolvedEBernhardsonT381909 WCQS updates for miscweb migration to k8s
 ResolvedLucas_Werkmeister_WMDET392691 Update Wikidata Query GUI URL in Codesearch
 ResolvedArnoldokothT350794 move os-reports.wikimedia.org to kubernetes
 ResolvedArnoldokothT350796 move security.wikimedia.org to kubernetes
 ResolvedJeltoT357413 miscweb wikikube staging namespace exceeding quota limits
 StalledNoneT361090 Move k8s miscweb blackbox checks out of microsites puppet module
 ResolvedDzahnT397080 decom legacy miscweb VMs
 ResolvedJeltoT397388 SystemdUnitFailed (sync_os_reports.service)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
CodeReviewBot added a comment.Jul 13 2023, 11:30 AM

jelto merged https://gitlab.wikimedia.org/repos/sre/miscweb/bienvenida/-/merge_requests/2

use timestamp in image tag

CodeReviewBot added a comment.Jul 13 2023, 11:43 AM

jelto opened https://gitlab.wikimedia.org/repos/sre/miscweb/annualreport/-/merge_requests/5

use timestamp in image tag

CodeReviewBot added a comment.Jul 13 2023, 11:48 AM

jelto merged https://gitlab.wikimedia.org/repos/sre/miscweb/annualreport/-/merge_requests/5

use timestamp in image tag

gerritbot added a comment.Jul 13 2023, 12:00 PM

Change 937938 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb: use timestamp in image tags

https://gerrit.wikimedia.org/r/937938

gerritbot added a comment.Jul 13 2023, 7:00 PM

Change 937938 merged by Dzahn:

[operations/deployment-charts@master] miscweb: use timestamp in image tags

https://gerrit.wikimedia.org/r/937938

Maintenance_bot removed a project: Patch-For-Review.Jul 13 2023, 7:10 PM
Jelto claimed this task.Jul 14 2023, 10:52 AM

Thanks for merging the above change @Dzahn . I deployed the change to all wikikube clusters. So we are using the same release workflow and tags for all services now (this issue was raised in T340182#8992829). One exception is bugzilla, but that service is still on Gerrit. Once we migrated this to GitLab, it should be similar as well.

kubectl get deployment -o=jsonpath="{range .items[*]}{'\n'}{.metadata.name}{':\t'}{range .spec.template.spec.containers[0]}{.image}{', '}{end}{end}"

miscweb-annualreport:   docker-registry.discovery.wmnet/repos/sre/miscweb/annualreport:2023-07-13-114826, 
miscweb-bienvenida:     docker-registry.discovery.wmnet/repos/sre/miscweb/bienvenida:2023-07-13-113046, 
miscweb-statictendril:  docker-registry.discovery.wmnet/repos/sre/miscweb/statictendril:2023-07-13-115511, 
miscweb-transparencyreport:     docker-registry.discovery.wmnet/repos/sre/miscweb/transparencyreport:2023-07-13-111133

I'll claim the task for now.

Dzahn added a subtask: T334511: Move research webpages to gitlab.Jul 14 2023, 4:14 PM
Dzahn mentioned this in T334511: Move research webpages to gitlab.Jul 14 2023, 4:21 PM
Dzahn added a comment.Jul 14 2023, 4:50 PM

Yep! So regarding static-bugzilla on Gerrit. That content repo is _quite large_ compared to any other microsite. Because there are thousands and thousands of HTML pages in there, though they are gzipped. I remember splitting the original upload into several patches because it was too much to have 50k files in a single Gerrit patch. So we should be careful when trying to import that into gitlab.

LSobanski updated Other Assignee, removed: Jelto.Jul 31 2023, 10:02 AM
Jelto updated the task description. (Show Details)Aug 14 2023, 9:35 AM
CodeReviewBot added a project: Patch-For-Review.Aug 24 2023, 1:39 PM

jelto opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/42

add /repos/sre/miscweb/bugzilla to Trusted Runners

CodeReviewBot added a comment.Aug 24 2023, 1:45 PM

jelto merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/42

add /repos/sre/miscweb/bugzilla to Trusted Runners

CodeReviewBot added a comment.Aug 24 2023, 1:53 PM

jelto opened https://gitlab.wikimedia.org/repos/sre/miscweb/bugzilla/-/merge_requests/1

add .gitlab-ci.yml

CodeReviewBot added a comment.Aug 24 2023, 1:54 PM

jelto merged https://gitlab.wikimedia.org/repos/sre/miscweb/bugzilla/-/merge_requests/1

add .gitlab-ci.yml

Jelto added a comment.Aug 24 2023, 2:04 PM
In T300171#9016375, @Dzahn wrote:

Yep! So regarding static-bugzilla on Gerrit. That content repo is _quite large_ compared to any other microsite. Because there are thousands and thousands of HTML pages in there, though they are gzipped. I remember splitting the original upload into several patches because it was too much to have 50k files in a single Gerrit patch. So we should be careful when trying to import that into gitlab.

I refactored the repo a bit and zipped all html files into a single gz file. This gz file is around 440Mb in size and stored in GitLabs LFS.
During image build this gz archive gets extracted and added to the image (note: I had to use multi stage builds to make sure the archive does not end up in some of the docker layers).

See also T343914 for some more reasoning for the refactoring.

Jelto added a subtask: T343914: Evaluate miscweb bugzilla architecture.Aug 24 2023, 2:04 PM
Jelto mentioned this in T343914: Evaluate miscweb bugzilla architecture.Aug 24 2023, 2:29 PM
gerritbot added a comment.Aug 24 2023, 2:38 PM

Change 952228 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb: migrate bugzilla image to GitLab

https://gerrit.wikimedia.org/r/952228

gerritbot added a comment.Aug 25 2023, 7:18 AM

Change 952228 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: migrate bugzilla image to GitLab

https://gerrit.wikimedia.org/r/952228

Jelto updated the task description. (Show Details)Aug 25 2023, 1:20 PM
Jelto closed subtask T343914: Evaluate miscweb bugzilla architecture as Resolved.Aug 29 2023, 10:45 AM
Jelto changed the status of subtask T334511: Move research webpages to gitlab from Open to Stalled.Aug 29 2023, 11:39 AM
fkaelin changed the status of subtask T334511: Move research webpages to gitlab from Stalled to In Progress.Aug 29 2023, 1:31 PM
fkaelin closed subtask T334511: Move research webpages to gitlab as Resolved.Sep 5 2023, 1:26 PM
Jelto updated the task description. (Show Details)Sep 11 2023, 8:38 AM
Jelto mentioned this in T346309: move static-codereview.wikimedia.org to kubernetes.Sep 14 2023, 9:15 AM
Jelto added a subtask: T346309: move static-codereview.wikimedia.org to kubernetes.
Jelto updated the task description. (Show Details)
Jelto closed subtask T346309: move static-codereview.wikimedia.org to kubernetes as Resolved.Sep 19 2023, 1:50 PM
Jelto updated the task description. (Show Details)
DDeSouza subscribed.Oct 29 2023, 8:30 PM
Jelto updated the task description. (Show Details)Nov 8 2023, 3:08 PM
Jelto updated the task description. (Show Details)Nov 8 2023, 3:12 PM
Jelto updated the task description. (Show Details)Nov 8 2023, 3:17 PM
Jelto updated the task description. (Show Details)Nov 8 2023, 3:19 PM
CodeReviewBot added a comment.Dec 8 2023, 9:52 AM

jelto opened https://gitlab.wikimedia.org/repos/sre/miscweb/research-landing-page/-/merge_requests/15

replace reviewer jelto with wmf-sre-collab group

CodeReviewBot added a comment.Dec 8 2023, 1:35 PM

dani merged https://gitlab.wikimedia.org/repos/sre/miscweb/research-landing-page/-/merge_requests/15

replace reviewer jelto with wmf-sre-collab group

Maintenance_bot removed a project: Patch-For-Review.Dec 8 2023, 2:10 PM
gerritbot added a comment.Jan 8 2024, 1:47 PM

Change 988495 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb: set requests and limit for bugzilla staging

https://gerrit.wikimedia.org/r/988495

gerritbot added a project: Patch-For-Review.Jan 8 2024, 1:47 PM
gerritbot added a comment.Jan 8 2024, 2:01 PM

Change 988495 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: set requests and limit for bugzilla staging

https://gerrit.wikimedia.org/r/988495

gerritbot added a comment.Jan 8 2024, 2:14 PM

Change 988499 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb: also set requests for bugzilla staging

https://gerrit.wikimedia.org/r/988499

gerritbot added a comment.Jan 8 2024, 3:39 PM

Change 988499 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: also set requests for bugzilla staging

https://gerrit.wikimedia.org/r/988499

Maintenance_bot removed a project: Patch-For-Review.Jan 8 2024, 4:32 PM
Jelto closed subtask T350791: move design.wikimedia.org to kubernetes as Resolved.Jan 18 2024, 2:39 PM
Jelto added a subtask: T357413: miscweb wikikube staging namespace exceeding quota limits.Feb 13 2024, 2:02 PM
Jelto closed subtask T357413: miscweb wikikube staging namespace exceeding quota limits as Resolved.Feb 14 2024, 2:11 PM
Jelto updated the task description. (Show Details)Feb 29 2024, 1:28 PM
Jelto mentioned this in T361090: Move k8s miscweb blackbox checks out of microsites puppet module.Mar 27 2024, 11:06 AM
Jelto added a subtask: T361090: Move k8s miscweb blackbox checks out of microsites puppet module.
Jelto changed the status of subtask T361090: Move k8s miscweb blackbox checks out of microsites puppet module from Open to Stalled.Apr 16 2024, 1:57 PM
Jelto updated the task description. (Show Details)Apr 16 2024, 3:08 PM
Arnoldokoth closed subtask T350796: move security.wikimedia.org to kubernetes as Resolved.Jun 6 2024, 3:03 AM
Jelto mentioned this in T364367: Create dedicated UIs for wdqs graph split endpoints.Jun 18 2024, 8:09 AM
gerritbot added a comment.Oct 10 2024, 1:12 PM

Change #1079279 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb: bump base and mesh templates to newest version

https://gerrit.wikimedia.org/r/1079279

gerritbot added a project: Patch-For-Review.Oct 10 2024, 1:12 PM
gerritbot added a comment.Oct 10 2024, 2:08 PM

Change #1079279 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: bump base and mesh templates to newest version

https://gerrit.wikimedia.org/r/1079279

bking closed subtask T350793: move query.wikidata.org to kubernetes as Invalid.Dec 10 2024, 7:39 PM
bking reopened subtask T350793: move query.wikidata.org to kubernetes as In Progress.
gerritbot added a comment.Feb 22 2025, 11:13 PM

Change #1121748 had a related patch set uploaded (by Jelto; author: Jelto):

[integration/docroot@master] update links to microsites source code

https://gerrit.wikimedia.org/r/1121748

gerritbot added a comment.Feb 24 2025, 10:33 AM

Change #1121748 merged by jenkins-bot:

[integration/docroot@master] update links to microsites source code

https://gerrit.wikimedia.org/r/1121748

Stashbot added a comment.Feb 24 2025, 10:34 AM

Mentioned in SAL (#wikimedia-operations) [2025-02-24T10:34:27Z] <hashar@deploy2002> Started deploy [integration/docroot@59d9e3f]: update links to microsites source code - T300171

Stashbot added a comment.Feb 24 2025, 10:34 AM

Mentioned in SAL (#wikimedia-operations) [2025-02-24T10:34:37Z] <hashar@deploy2002> Finished deploy [integration/docroot@59d9e3f]: update links to microsites source code - T300171 (duration: 00m 10s)

Aklapper changed the task status from In Progress to Open.Apr 11 2025, 10:15 PM

Resetting task status from "In Progress" to "Open" as this task has been "in progress" for more than one and a half years (see T380300).

Jelto closed subtask T350793: move query.wikidata.org to kubernetes as Resolved.Apr 23 2025, 11:21 AM
Jelto updated the task description. (Show Details)
gerritbot added a comment.Apr 29 2025, 4:33 PM

Change #1137486 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] microsites/backup: remove rt-static backup fileset

https://gerrit.wikimedia.org/r/1137486

gerritbot added a comment.Apr 29 2025, 4:46 PM

Change #1137486 merged by Dzahn:

[operations/puppet@production] microsites/backup: remove rt-static backup fileset

https://gerrit.wikimedia.org/r/1137486

gerritbot added a comment.Apr 29 2025, 8:09 PM

Change #1137484 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] miscweb: remove static-rt profile from legacy miscweb role

https://gerrit.wikimedia.org/r/1137484

gerritbot added a comment.Apr 29 2025, 8:14 PM

Change #1137484 merged by Dzahn:

[operations/puppet@production] miscweb: remove static-rt profile from legacy miscweb role

https://gerrit.wikimedia.org/r/1137484

Dzahn added a comment.Apr 29 2025, 11:33 PM

I removed the static-rt site from the legacy miscweb VMs.

Now os-reports (T350794) is the only remaining site.

root@miscweb1003:/home/dzahn# apache2ctl -S
VirtualHost configuration:
[::1]:80               localhost (/etc/apache2/conf-enabled/50-server-status.conf:15)
127.0.0.1:80           localhost (/etc/apache2/conf-enabled/50-server-status.conf:15)
*:80                   os-reports.wikimedia.org (/etc/apache2/sites-enabled/50-os-reports-wikimedia-org.conf:1)
Dzahn updated the task description. (Show Details)Apr 29 2025, 11:33 PM
Dzahn mentioned this in T385777: moscovium and RT sunset.
Dzahn added a subtask: T397080: decom legacy miscweb VMs.Jul 1 2025, 10:36 PM
Dzahn added a subscriber: Arnoldokoth.

@Arnoldokoth Can we check os_reports off the list here?

Dzahn added a comment.Jul 1 2025, 10:41 PM

T397264 is for moving the monitoring checks to a new place.

T397080 is for decom'ing the legacy VMs, which happened today.

Dzahn updated the task description. (Show Details)Jul 2 2025, 5:36 PM

legacy VMs are shut down.

DNS discovery records deleted.

confirmed with Arnold os-reports is done.

We can close this ticket :)

Dzahn closed this task as Resolved.Jul 2 2025, 5:37 PM
Dzahn closed subtask T397080: decom legacy miscweb VMs as Resolved.Jul 2 2025, 5:53 PM
LSobanski closed subtask T350794: move os-reports.wikimedia.org to kubernetes as Resolved.Jul 3 2025, 1:08 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 3 2025, 1:32 PM
Dzahn reopened subtask T397080: decom legacy miscweb VMs as Open.Nov 21 2025, 4:29 PM
Dzahn changed the status of subtask T397080: decom legacy miscweb VMs from Open to In Progress.Nov 24 2025, 8:12 PM
Dzahn closed subtask T397080: decom legacy miscweb VMs as Resolved.Nov 25 2025, 5:04 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits