Page MenuHomePhabricator
Create Task
Maniphest T362323

Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons)
Open, HighPublic
Actions

Description

This is (almost) the final step!

Progressively forward the remaining 30% of external traffic to MW-on-K8s

What?

Wikikube cluster will be fully serving:

  • External traffic (API, web, mobile)
  • Internal traffic
  • MediaWiki jobs (former jobrunners)

In MW-on-K8s terms, this translates to following deployments

  • mw-web
  • mw-api-int
  • mw-api-ext
  • mw-jobrunner
  • mw-parsoid
  • mw-wikifunctions

What we are not migrating to Wikikube yet

External traffic

Jobs

Internal traffic

Progression

  • 75%
  • 80%
  • 85%
  • 90%
  • 95%
  • 100%

Notes

The above is per DC.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Clement_Goubert triaged this task as High priority.Thu, Apr 11, 12:14 PM
Clement_Goubert created this task.
hnowlan awarded a token.Thu, Apr 11, 12:16 PM
Ladsgroup awarded a token.Thu, Apr 11, 12:22 PM
jijiki awarded a token.Thu, Apr 11, 12:23 PM
Clement_Goubert mentioned this in T290536: Serve production traffic via Kubernetes.Thu, Apr 11, 12:30 PM
taavi awarded a token.Thu, Apr 11, 12:33 PM
Clement_Goubert updated the task description. (Show Details)Thu, Apr 11, 12:46 PM
Clement_Goubert updated the task description. (Show Details)Thu, Apr 11, 12:49 PM
jijiki updated the task description. (Show Details)Tue, Apr 16, 9:31 AM
jijiki added a project: MoveComms-Support.Tue, Apr 16, 9:34 AM
jijiki added a subscriber: Trizek-WMF.
gerritbot added a comment.Fri, Apr 19, 11:59 AM

Change #1021904 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mw-web, mw-api-ext: Raise replicas for 75% traffic

https://gerrit.wikimedia.org/r/1021904

gerritbot added a project: Patch-For-Review.Fri, Apr 19, 11:59 AM
gerritbot added a comment.Fri, Apr 19, 12:02 PM

Change #1021905 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] trafficserver: move 75% of traffic to mw on k8s

https://gerrit.wikimedia.org/r/1021905

gerritbot added a comment.Fri, Apr 19, 3:59 PM

Change #1021904 merged by jenkins-bot:

[operations/deployment-charts@master] mw-web, mw-api-ext: Raise replicas for 75% traffic

https://gerrit.wikimedia.org/r/1021904

gerritbot added a comment.Tue, Apr 23, 9:49 AM

Change #1021905 merged by Clément Goubert:

[operations/puppet@production] trafficserver: move 75% of traffic to mw on k8s

https://gerrit.wikimedia.org/r/1021905

Clement_Goubert updated the task description. (Show Details)Tue, Apr 23, 9:51 AM
gerritbot added a comment.Tue, Apr 23, 10:28 AM

Change #1023397 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] kubernetes: move 5 eqiad appservers to kubernetes

https://gerrit.wikimedia.org/r/1023397

gerritbot added a comment.Tue, Apr 23, 10:48 AM

Change #1023397 merged by Clément Goubert:

[operations/puppet@production] kubernetes: move 5 eqiad appservers to kubernetes

https://gerrit.wikimedia.org/r/1023397

gerritbot added a comment.Tue, Apr 23, 11:58 AM

Change #1023412 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mw-web, mw-api-ext: Raise replicas for 80% traffic

https://gerrit.wikimedia.org/r/1023412

gerritbot added a comment.Tue, Apr 23, 12:00 PM

Change #1023413 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] trafficserver: move 80% of traffic to mw on k8s

https://gerrit.wikimedia.org/r/1023413

Trizek-WMF added a comment.Tue, Apr 23, 12:50 PM

Checking after MoveComms-Support was added to this task: what kind of support do you need, if any?

gerritbot added a comment.Wed, Apr 24, 8:52 AM

Change #1023412 merged by jenkins-bot:

[operations/deployment-charts@master] mw-web, mw-api-ext: Raise replicas for 80% traffic

https://gerrit.wikimedia.org/r/1023412

gerritbot added a comment.Wed, Apr 24, 8:57 AM

Change #1023413 merged by Clément Goubert:

[operations/puppet@production] trafficserver: move 80% of traffic to mw on k8s

https://gerrit.wikimedia.org/r/1023413

Clement_Goubert updated the task description. (Show Details)Wed, Apr 24, 9:15 AM
Stashbot added a comment.Wed, Apr 24, 9:29 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-24T09:29:39Z] <claime> 80% of external traffix to mw-on-k8s - T362323

hnowlan subscribed.Fri, Apr 26, 4:00 PM
Maintenance_bot removed a project: Patch-For-Review.Fri, Apr 26, 6:31 PM
gerritbot added a comment.Wed, May 1, 3:00 PM

Change #1026159 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: move 80% of traffic to mw-on-k8s

https://gerrit.wikimedia.org/r/1026159

gerritbot added a project: Patch-For-Review.Wed, May 1, 3:00 PM

Change #1026160 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] mw-we, mw-api-ext: bump replicas

https://gerrit.wikimedia.org/r/1026160

gerritbot added a comment.Wed, May 1, 3:13 PM

Change #1026158 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] k8s: move 5 eqiad appservers to kubernetes

https://gerrit.wikimedia.org/r/1026158

gerritbot added a comment.Thu, May 2, 10:55 AM

Change #1026158 merged by Hnowlan:

[operations/puppet@production] k8s: move 5 eqiad appservers to kubernetes

https://gerrit.wikimedia.org/r/1026158

gerritbot added a comment.Thu, May 2, 10:59 AM

Change #1026520 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] scap: make mw1407 a scap proxy

https://gerrit.wikimedia.org/r/1026520

gerritbot added a comment.Thu, May 2, 11:05 AM

Change #1026520 merged by Hnowlan:

[operations/puppet@production] scap: make mw1407 a scap proxy

https://gerrit.wikimedia.org/r/1026520

ops-monitoring-bot added a comment.Thu, May 2, 11:15 AM

Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1371.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, May 2, 11:17 AM

Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1409.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, May 2, 11:21 AM

Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1435.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, May 2, 11:21 AM

Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1399.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, May 2, 11:21 AM

Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw1405.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, May 2, 11:51 AM

Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1371.eqiad.wmnet with OS bullseye completed:

  • mw1371 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405021132_hnowlan_3953702_mw1371.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Thu, May 2, 11:53 AM

Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1409.eqiad.wmnet with OS bullseye completed:

  • mw1409 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405021135_hnowlan_3953708_mw1409.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Thu, May 2, 11:55 AM

Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1405.eqiad.wmnet with OS bullseye completed:

  • mw1405 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405021137_hnowlan_3953738_mw1405.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Thu, May 2, 11:57 AM

Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1435.eqiad.wmnet with OS bullseye completed:

  • mw1435 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405021139_hnowlan_3953714_mw1435.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Thu, May 2, 12:00 PM

Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw1399.eqiad.wmnet with OS bullseye completed:

  • mw1399 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405021143_hnowlan_3953743_mw1399.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Thu, May 2, 2:08 PM

Change #1026160 merged by jenkins-bot:

[operations/deployment-charts@master] mw-web, mw-api-ext: bump replicas

https://gerrit.wikimedia.org/r/1026160

gerritbot added a comment.Thu, May 2, 3:04 PM

Change #1026159 merged by Hnowlan:

[operations/puppet@production] trafficserver: move 85% of traffic to mw-on-k8s

https://gerrit.wikimedia.org/r/1026159

Maintenance_bot removed a project: Patch-For-Review.Thu, May 2, 3:30 PM
gerritbot added a comment.Tue, May 7, 2:13 PM

Change #1028840 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] kubernetes: make 5 eqiad api appservers k8s workers

https://gerrit.wikimedia.org/r/1028840

gerritbot added a project: Patch-For-Review.Tue, May 7, 2:13 PM
gerritbot added a comment.Tue, May 7, 2:17 PM

Change #1028842 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] mw-web, mw-api-ext: bump replicas in advance of traffic shift

https://gerrit.wikimedia.org/r/1028842

gerritbot added a comment.Tue, May 7, 2:20 PM

Change #1028844 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] trafficserver: move k8s traffic shift to 90%

https://gerrit.wikimedia.org/r/1028844

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL