Page MenuHomePhabricator

Move 10% of mediawiki external requests to mw on k8s
Closed, ResolvedPublic

Description

Move external traffic progressively at 6.5, 8, 10% to mw on k8s.

Before we can do it we need to complete the parent task, and have the new hardware in place. Once this is done, we can start cannibalizing the mw clusters turning them in k8s nodes as well.

Event Timeline

Change 957857 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] trafficserver: move 6.5% of traffic to mw on k8s

https://gerrit.wikimedia.org/r/957857

Change 957858 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] trafficserver: move 8% of traffic to mw on k8s

https://gerrit.wikimedia.org/r/957858

Change 957859 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] trafficserver: move 10% of traffic to mw on k8s

https://gerrit.wikimedia.org/r/957859

Change 961048 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mw-web, mw-api-ext: Raise replicas, raise apcu size

https://gerrit.wikimedia.org/r/961048

Change 961048 merged by jenkins-bot:

[operations/deployment-charts@master] mw-web, mw-api-ext: Raise replicas, raise apcu size

https://gerrit.wikimedia.org/r/961048

Mentioned in SAL (#wikimedia-operations) [2023-09-26T09:35:30Z] <claime> Raised replicas to 20 for mw-api-ext and mw-web - T346422

Change 957857 merged by Effie Mouzeli:

[operations/puppet@production] trafficserver: move 6.5% of traffic to mw on k8s

https://gerrit.wikimedia.org/r/957857

Mentioned in SAL (#wikimedia-operations) [2023-09-26T14:38:05Z] <effie> Rump up traffic to mw-on-k8s to 6.5% - T346422

Clement_Goubert changed the task status from Open to In Progress.Sep 27 2023, 8:56 AM
Clement_Goubert triaged this task as High priority.

Change 961337 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/alerts@master] mw-on-k8s: Raise idle worker alerting threshold to 50%

https://gerrit.wikimedia.org/r/961337

Change 961341 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mw-api-ext, mw-web: raise replicas for traffic bump

https://gerrit.wikimedia.org/r/961341

Change 961337 merged by jenkins-bot:

[operations/alerts@master] mw-on-k8s: Raise idle worker alerting threshold to 50%

https://gerrit.wikimedia.org/r/961337

Mentioned in SAL (#wikimedia-operations) [2023-09-27T09:43:31Z] <claime> Bumping mw-on-k8s traffic to 8% - T346422

Change 957858 merged by Clément Goubert:

[operations/puppet@production] trafficserver: move 8% of traffic to mw on k8s

https://gerrit.wikimedia.org/r/957858

Change 961353 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mw-web: Raise main replicas to 22

https://gerrit.wikimedia.org/r/961353

Change 961353 merged by jenkins-bot:

[operations/deployment-charts@master] mw-web: Raise main replicas to 22

https://gerrit.wikimedia.org/r/961353

Change 961357 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] mw-web: Raise main replicas to 25

https://gerrit.wikimedia.org/r/961357

Change 961357 merged by jenkins-bot:

[operations/deployment-charts@master] mw-web: Raise main replicas to 25

https://gerrit.wikimedia.org/r/961357

Change 961362 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/alerts@master] mw-on-k8s: Fold canaries into global php-fpm idle alert

https://gerrit.wikimedia.org/r/961362

Change 961362 merged by jenkins-bot:

[operations/alerts@master] mw-on-k8s: Fold canaries into global php-fpm idle alert

https://gerrit.wikimedia.org/r/961362

Change 957859 merged by Giuseppe Lavagetto:

[operations/puppet@production] trafficserver: move 10% of traffic to mw on k8s

https://gerrit.wikimedia.org/r/957859

Clement_Goubert claimed this task.

We are now serving 10% of global appserver requests from mw-on-k8s \o/

Resolving.

Change 961341 abandoned by Clément Goubert:

[operations/deployment-charts@master] mw-api-ext, mw-web: raise replicas for traffic bump

Reason:

Way past this.

https://gerrit.wikimedia.org/r/961341