Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	akosiaris
	Feb 13 2024, 11:12 AM

Description

Based on mediawiki helm chart in order to migrate the traffic that makes it to the current parsoid cluster to MW-on-K8s

Status

The deployment has been created, successfully deployed to and is now ready to start receiving traffic for parsoid-php.discovery.wmnet

Plan is:

1 host per DC for a couple of days
3 hosts per DC for a couple of more days
7 hosts per DC for a few more days
8 hosts for a few hours (we are going to be pretty confident here)
Everything Else, aka 100%
Declare it done. Cleanups tracked at T359387

Details

Subject	Repo	Branch	Lines +/-
restbase: Switch the default to mw-parsoid	operations/puppet	production	+1 -1
Switch restbase1026-1033, restbase20289-2034 to mw-parsoid	operations/puppet	production	+15 -0
mw-parsoid: replicas x2 for hopefully the last time	operations/deployment-charts	master	+2 -2
Switch more eqiad parsoid hosts to mw-parsoid	operations/puppet	production	+14 -0
Switch restbase102[2345], restbase202[4567] to mw-parsoid	operations/puppet	production	+8 -0
mw-parsoid: Bump replicas	operations/deployment-charts	master	+2 -2
Switch restbase102[01], restbase202[23] to mw-parsoid	operations/puppet	production	+4 -0
Switch restbase1019, restbase2021 to mw-parsoid	operations/puppet	production	+2 -0
ClusterConfig: Add kube-wiki-parsoid test	operations/mediawiki-config	master	+3 -0
services_proxy: Add mw-parsoid in the mesh	operations/puppet	production	+10 -0
Revert "conftool: Add some kubernetes hosts to parsoid"	operations/puppet	production	+0 -49
mw-parsoid: Add parsoid.discovery.wmnet in cert SANs	operations/deployment-charts	master	+6 -0
conftool: Add some kubernetes hosts to parsoid	operations/puppet	production	+49 -0
mw-parsoid: Switch to production and have it page	operations/puppet	production	+2 -2
Add mw-parsoid	operations/dns	master	+7 -1
mw-parsoid: Switch to lvs_setup	operations/puppet	production	+1 -1
mw-parsoid: Add LVS backends on wikikube servers	operations/puppet	production	+1 -0
service::catalog: Add mw-parsoid service	operations/puppet	production	+36 -0
conftool: Add mw-parsoid stanzas	operations/puppet	production	+1 -0
mw-parsoid: Have deployments happening	operations/puppet	production	+6 -0
mw-parsoid: Introduce it	operations/deployment-charts	master	+100 -0
admin_ng: mw-parsoid stanzas	operations/deployment-charts	master	+24 -0
deploy: Add mw-parsoid namespace stanzas	operations/puppet	production	+4 -0

Related Objects
Search...

Status	Assigned	Task
Stalled	None	T255792 Quibble runs core:unit tests twice!
Open	None	T328919 Upgrade to PHPUnit 10
Open	None	T338103 Micro-optimize ApiResult::isMetadataKey with str_starts_with once we support PHP8+
Open	None	T328921 Drop PHP 7.4 support from MediaWiki
Stalled	None	T334726 Use return type `never` in Wikibase
Open	None	T328922 Drop PHP 8.0 support from MediaWiki
Stalled	None	T319055 Upgrade to psr/container 2.x
Stalled	Krinkle	T319432 Migrate WMF production from PHP 7.4 to PHP 8.1
Open	None	T291916 Tracking task for Bullseye migrations in production
Stalled	None	T356293 Migrate MW appservers' base images to bullseye
Open	None	T290536 Serve production traffic via Kubernetes
Resolved	akosiaris	T357392 Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it
Open	None	T358588 A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle
Resolved	akosiaris	T358752 Reimage parse* hosts as kubernetes nodes

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Jdforrester-WMF added a project: Content-Transform-Team.Feb 13 2024, 3:34 PM

Jdforrester-WMF subscribed.

MSantos subscribed.Feb 15 2024, 3:03 PM

MSantos moved this task from Backlog to Tracking on the Content-Transform-Team board.Feb 15 2024, 3:07 PM

Seems fine, most parsoid traffic is probably happening on the main cluster these days anyway.
We just want to make sure that scandium doesn't get trampled on, since we still use this for pre-deployment testing.

Lucas_Werkmeister_WMDE unsubscribed.Feb 15 2024, 3:10 PM

Also worth keeping in mind: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/965608 -- in order to avoid inadventently getting external non-WMF clients for the internal Parsoid REST API, we currently expose that API only in the Parsoid cluster. If/when we move away from the Parsoid cluster we'll need to use some other mechanism to protect those endpoints from external use.

Change 1004138 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/dns@master] Add mw-parsoid

https://gerrit.wikimedia.org/r/1004138

gerritbot added a project: Patch-For-Review.Feb 16 2024, 3:13 PM

Change 1004149 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] deploy: Add mw-parsoid namespace stanzas

https://gerrit.wikimedia.org/r/1004149

Change 1004150 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] mw-parsoid: Have deployments happening

https://gerrit.wikimedia.org/r/1004150

Change 1004151 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] conftool: Add mw-parsoid stanzas

https://gerrit.wikimedia.org/r/1004151

Change 1004152 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] service::catalog: Add mw-parsoid service

https://gerrit.wikimedia.org/r/1004152

Change 1004153 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] mw-parsoid: Add LVS backends on wikikube servers

https://gerrit.wikimedia.org/r/1004153

Change 1004154 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] mw-parsoid: Switch to lvs_setup

https://gerrit.wikimedia.org/r/1004154

Change 1004155 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] mw-parsoid: Switch to production and have it page

https://gerrit.wikimedia.org/r/1004155

Novem_Linguae unsubscribed.Feb 16 2024, 3:32 PM

Change 1004157 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] mw-parsoid: Introduce it

https://gerrit.wikimedia.org/r/1004157

Change 1004739 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] admin_ng: mw-parsoid stanzas

https://gerrit.wikimedia.org/r/1004739

Change 1004149 merged by Alexandros Kosiaris:

[operations/puppet@production] deploy: Add mw-parsoid namespace stanzas

https://gerrit.wikimedia.org/r/1004149

Change 1004739 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: mw-parsoid stanzas

https://gerrit.wikimedia.org/r/1004739

Change 1004157 merged by jenkins-bot:

[operations/deployment-charts@master] mw-parsoid: Introduce it

https://gerrit.wikimedia.org/r/1004157

Change 1004150 merged by Alexandros Kosiaris:

[operations/puppet@production] mw-parsoid: Have deployments happening

https://gerrit.wikimedia.org/r/1004150

Change 1004151 merged by Alexandros Kosiaris:

[operations/puppet@production] conftool: Add mw-parsoid stanzas

https://gerrit.wikimedia.org/r/1004151

Change 1004152 merged by Alexandros Kosiaris:

[operations/puppet@production] service::catalog: Add mw-parsoid service

https://gerrit.wikimedia.org/r/1004152

Change 1004153 merged by Alexandros Kosiaris:

[operations/puppet@production] mw-parsoid: Add LVS backends on wikikube servers

https://gerrit.wikimedia.org/r/1004153

Change 1004154 merged by Alexandros Kosiaris:

[operations/puppet@production] mw-parsoid: Switch to lvs_setup

https://gerrit.wikimedia.org/r/1004154

Mentioned in SAL (#wikimedia-operations) [2024-02-21T12:01:13Z] <akosiaris> restart pybal on lvs1020 to pickup mw-parsoid service. T357392

Mentioned in SAL (#wikimedia-operations) [2024-02-21T12:02:22Z] <akosiaris> restart pybal on lvs2014 to pickup mw-parsoid service. T357392

Mentioned in SAL (#wikimedia-operations) [2024-02-21T12:10:19Z] <akosiaris> restart pybal on lvs2013, lvs 1019 to pickup mw-parsoid service. T357392

Change 1004138 merged by Alexandros Kosiaris:

[operations/dns@master] Add mw-parsoid

https://gerrit.wikimedia.org/r/1004138

Change 1004155 merged by Alexandros Kosiaris:

[operations/puppet@production] mw-parsoid: Switch to production and have it page

https://gerrit.wikimedia.org/r/1004155

Maintenance_bot removed a project: Patch-For-Review.Feb 21 2024, 12:31 PM

akosiaris changed the task status from Open to In Progress.Feb 21 2024, 2:50 PM

akosiaris triaged this task as Medium priority.

akosiaris moved this task from Backlog to Done on the MW-on-K8s board.Feb 22 2024, 9:28 AM

akosiaris moved this task from Done to In Progress on the MW-on-K8s board.Feb 22 2024, 9:31 AM

akosiaris moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.

Change 1005723 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/mediawiki-config@master] ClusterConfig: Add kube-wiki-parsoid test

https://gerrit.wikimedia.org/r/1005723

gerritbot added a project: Patch-For-Review.Feb 22 2024, 10:56 AM

In T357392#9546852, @cscott wrote:

Seems fine, most parsoid traffic is probably happening on the main cluster these days anyway.
We just want to make sure that scandium doesn't get trampled on, since we still use this for pre-deployment testing.

We aren't touching scandium at all for this, so I think we are safe on that front.

In T357392#9547408, @cscott wrote:

Also worth keeping in mind: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/965608 -- in order to avoid inadventently getting external non-WMF clients for the internal Parsoid REST API, we currently expose that API only in the Parsoid cluster. If/when we move away from the Parsoid cluster we'll need to use some other mechanism to protect those endpoints from external use.

Good point. I 've posted https://gerrit.wikimedia.org/r/1005723 to test for this and it passes already. SERVERGROUP btw is for this deployment kube-mw-parsoid which matches both parsoid and k8s traits.

I 've also used curl -s -H "Host: en.wikipedia.org" https://mw-parsoid.discovery.wmnet:4452/wiki/Special:Version to see whether Parsoid extension is installed and as expected, it is.

Change 1005730 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] conftool: Add some kubernetes hosts to parsoid

https://gerrit.wikimedia.org/r/1005730

Change 1005730 merged by Alexandros Kosiaris:

[operations/puppet@production] conftool: Add some kubernetes hosts to parsoid

https://gerrit.wikimedia.org/r/1005730

Change 1005770 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] mw-parsoid: Add parsoid.discovery.wmnet in cert SANs

https://gerrit.wikimedia.org/r/1005770

Change 1005770 merged by jenkins-bot:

[operations/deployment-charts@master] mw-parsoid: Add parsoid.discovery.wmnet in cert SANs

https://gerrit.wikimedia.org/r/1005770

akosiaris renamed this task from Create parsoid mediawiki deployment to Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it.Feb 22 2024, 3:09 PM

akosiaris updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2024-02-22T15:12:45Z] <akosiaris> Bump weight of old parsoid hosts from 10 to 110. This is a noop right now but will makes calculations later spelled out in T357392 possible.

Mentioned in SAL (#wikimedia-operations) [2024-02-22T15:15:11Z] <akosiaris> T357392 pool 46 kubernetes hosts of parsoid-php with a weight of 1. Since the 42 parse hosts are at weight 110, that means 1% goes to mw-parsoid deployment, aka mw-on-k8s

Change 1006890 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] Revert "conftool: Add some kubernetes hosts to parsoid"

https://gerrit.wikimedia.org/r/1006890

Change 1006892 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] services_proxy: Add mw-parsoid in the mesh

https://gerrit.wikimedia.org/r/1006892

Change 1006893 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] Switch restbase1019, restbase2021 to mw-parsoid

https://gerrit.wikimedia.org/r/1006893

Change 1006894 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] Switch restbase102[01], restbase202[23] to mw-parsoid

https://gerrit.wikimedia.org/r/1006894

Change 1006895 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] Switch restbase102[2345], restbase202[4567] to mw-parsoid

https://gerrit.wikimedia.org/r/1006895

Change 1006896 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] Switch restbase102[6789], restbase103[0123], restbase202[89], restbase203[01234] to mw-parsoid

https://gerrit.wikimedia.org/r/1006896

Change 1006897 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] Switch the remaining parsoid hosts to mw-parsoid

https://gerrit.wikimedia.org/r/1006897

Change 1006898 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] restbase: Switch the default to mw-parsoid

https://gerrit.wikimedia.org/r/1006898

Change 1006899 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] Clean up all the RESTBase hosts's parsoid uri changes

https://gerrit.wikimedia.org/r/1006899

Change 1006900 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] services_proxy: Remove parsoid-php, parsoid-async

https://gerrit.wikimedia.org/r/1006900

The LVS traffic approach was doomed to fail, since scap utilizes the same data structure to figure out which hosts to deploy to. I 've re-ran numbers on services_proxy and parsoid cluster to make sure I ain't missing anything and it appears that indeed the only direct client is RESTBase and monitoring/healthchecks. So, the services_proxy approach should work fine. I 've updated the plan in the task and I 'll start executing it.

Change 1006890 merged by Alexandros Kosiaris:

[operations/puppet@production] Revert "conftool: Add some kubernetes hosts to parsoid"

https://gerrit.wikimedia.org/r/1006890

Change 1006892 merged by Alexandros Kosiaris:

[operations/puppet@production] services_proxy: Add mw-parsoid in the mesh

https://gerrit.wikimedia.org/r/1006892

Change 1005723 merged by jenkins-bot:

[operations/mediawiki-config@master] ClusterConfig: Add kube-wiki-parsoid test

https://gerrit.wikimedia.org/r/1005723

Change 1006893 merged by Alexandros Kosiaris:

[operations/puppet@production] Switch restbase1019, restbase2021 to mw-parsoid

https://gerrit.wikimedia.org/r/1006893

Migration started, we are batch 1 for the next few days.

akosiaris mentioned this in T358588: A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle.Feb 27 2024, 2:24 PM

Change 1006894 merged by Alexandros Kosiaris:

[operations/puppet@production] Switch restbase102[01], restbase202[23] to mw-parsoid

https://gerrit.wikimedia.org/r/1006894

akosiaris updated the task description. (Show Details)Feb 29 2024, 8:13 AM

akosiaris mentioned this in T358752: Reimage parse* hosts as kubernetes nodes.Feb 29 2024, 10:59 AM

Change 1008828 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] mw-parsoid: Bump replicas

https://gerrit.wikimedia.org/r/1008828

Change 1008828 merged by jenkins-bot:

[operations/deployment-charts@master] mw-parsoid: Bump replicas

https://gerrit.wikimedia.org/r/1008828

Change 1006895 merged by Alexandros Kosiaris:

[operations/puppet@production] Switch restbase102[2345], restbase202[4567] to mw-parsoid

https://gerrit.wikimedia.org/r/1006895

We at ~50% mw-parsoid right now.

Change 1006897 merged by Alexandros Kosiaris:

[operations/puppet@production] Switch more eqiad parsoid hosts to mw-parsoid

https://gerrit.wikimedia.org/r/1006897

akosiaris updated the task description. (Show Details)Mar 6 2024, 7:37 AM

Change 1009227 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] mw-parsoid: replicas x2 for hopefully the last time

https://gerrit.wikimedia.org/r/1009227

Change 1009227 merged by jenkins-bot:

[operations/deployment-charts@master] mw-parsoid: replicas x2 for hopefully the last time

https://gerrit.wikimedia.org/r/1009227

Change 1006896 merged by Alexandros Kosiaris:

[operations/puppet@production] Switch restbase1026-1033, restbase20289-2034 to mw-parsoid

https://gerrit.wikimedia.org/r/1006896

Change 1006898 merged by Alexandros Kosiaris:

[operations/puppet@production] restbase: Switch the default to mw-parsoid

https://gerrit.wikimedia.org/r/1006898

akosiaris updated the task description. (Show Details)Mar 6 2024, 2:19 PM

akosiaris closed subtask T358752: Reimage parse* hosts as kubernetes nodes as Resolved.Mar 6 2024, 2:24 PM

akosiaris closed this task as Resolved.Mar 6 2024, 2:32 PM

akosiaris mentioned this in T359387: Cleanup parsoid-php service.