Page MenuHomePhabricator
Create Task
Maniphest T420468

Retire mw-parsoid LVS service
Closed, ResolvedPublic
Actions

Description

After repurposing mw-parsoid in T386246, we can retire the LVS service as it is not used anymore.

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects

Event Timeline

jijiki created this task.Mar 18 2026, 1:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 18 2026, 1:15 PM
Scott_French triaged this task as Medium priority.Mar 19 2026, 12:54 AM
Scott_French added a project: ServiceOps-Services-Oids.
Scott_French moved this task from Inbox to Scheduled (this Q) on the ServiceOps board.

@jijiki - I've optimistically triaged this to Medium and moved it to Scheduled, since my understanding is that you'll be working on this soon. Please adjust if needed. Thank you!

jijiki closed this task as Invalid.Mar 20 2026, 10:11 AM
jijiki raised the priority of this task from Medium to High.
jijiki reopened this task as In Progress.Mar 20 2026, 10:15 AM
MLechvien-WMF mentioned this in T420336: mw-parsoid improvements.Mar 23 2026, 10:29 AM
gerritbot added a comment.Mar 26 2026, 12:54 PM

Change #1261428 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] mw-parsoid: remove DNS of mw-parsoid LVS service 1

https://gerrit.wikimedia.org/r/1261428

gerritbot added a project: Patch-For-Review.Mar 26 2026, 12:54 PM
gerritbot added a comment.Mar 26 2026, 12:57 PM

Change #1261430 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] mw-parsoid: Remove probes 2

https://gerrit.wikimedia.org/r/1261430

gerritbot added a comment.Mar 26 2026, 1:01 PM

Change #1261433 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] mw-parsoid: switch to service_setup 3

https://gerrit.wikimedia.org/r/1261433

MLechvien-WMF assigned this task to jijiki.Mar 26 2026, 4:59 PM
MLechvien-WMF moved this task from Scheduled (this Q) to In Progress on the ServiceOps board.
gerritbot added a comment.Mar 27 2026, 9:52 AM

Change #1262025 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/dns@master] mw-parsoid: add CNAMES

https://gerrit.wikimedia.org/r/1262025

gerritbot added a comment.Mar 27 2026, 9:53 AM

Change #1262026 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] trafficserver: update mw-parsoid XWD entries

https://gerrit.wikimedia.org/r/1262026

gerritbot added a comment.Mar 27 2026, 11:02 AM

Change #1262052 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] mw-parsoid: remove service definition 4

https://gerrit.wikimedia.org/r/1262052

gerritbot added a comment.Mar 27 2026, 11:03 AM

Change #1262054 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] envoy: remove mw-parsoid listener

https://gerrit.wikimedia.org/r/1262054

gerritbot added a comment.Mar 30 2026, 6:48 PM

Change #1261428 merged by BCornwall:

[operations/dns@master] mw-parsoid: remove DNS of mw-parsoid LVS service 1

https://gerrit.wikimedia.org/r/1261428

gerritbot added a comment.Mar 30 2026, 6:54 PM

Change #1261430 merged by BCornwall:

[operations/puppet@production] mw-parsoid: Remove probes 2

https://gerrit.wikimedia.org/r/1261430

gerritbot added a comment.Mar 30 2026, 7:00 PM

Change #1261433 merged by BCornwall:

[operations/puppet@production] mw-parsoid: switch to service_setup 3

https://gerrit.wikimedia.org/r/1261433

Stashbot added a comment.Mar 30 2026, 7:06 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T19:05:59Z] <brett@cumin2002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-codfw and A:lvs (T420468)

Stashbot added a comment.Mar 30 2026, 7:06 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T19:06:30Z] <brett@cumin2002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-codfw and A:lvs (T420468)

Stashbot added a comment.Mar 30 2026, 7:08 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T19:08:06Z] <brett@cumin2002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-eqiad and A:lvs (T420468)

Stashbot added a comment.Mar 30 2026, 7:10 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T19:10:32Z] <brett@cumin2002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-eqiad and A:lvs (T420468)

Stashbot added a comment.Mar 30 2026, 7:11 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T19:11:28Z] <brett@cumin2002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-low-traffic-codfw and A:lvs (T420468)

Stashbot added a comment.Mar 30 2026, 7:12 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T19:11:59Z] <brett@cumin2002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-low-traffic-codfw and A:lvs (T420468)

Stashbot added a comment.Mar 30 2026, 7:12 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T19:12:15Z] <brett@cumin2002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-low-traffic-eqiad and A:lvs (T420468)

Stashbot added a comment.Mar 30 2026, 7:12 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T19:12:57Z] <brett@cumin2002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-low-traffic-eqiad and A:lvs (T420468)

Stashbot added a comment.Mar 30 2026, 7:15 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T19:15:15Z] <brett> Delete mw-parsoid lvs service via sudo -i cumin A:lvs-low-traffic-codfw 'ipvsadm --delete-service --tcp-service 10.2.1.92:4452' - T420468

Stashbot added a comment.Mar 30 2026, 7:16 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T19:16:18Z] <brett> Delete mw-parsoid lvs service via sudo -i cumin A:lvs-secondary-codfw 'ipvsadm --delete-service --tcp-service 10.2.1.92:4452' - T420468

Stashbot added a comment.Mar 30 2026, 7:18 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T19:18:09Z] <brett> Delete mw-parsoid lvs service via sudo -i cumin A:lvs-secondary-eqiad 'ipvsadm --delete-service --tcp-service 10.2.2.92:4452' - T420468

Stashbot added a comment.Mar 30 2026, 7:18 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T19:18:18Z] <brett> Delete mw-parsoid lvs service via sudo -i cumin A:lvs-low-traffic-eqiad 'ipvsadm --delete-service --tcp-service 10.2.2.92:4452' - T420468

gerritbot added a comment.Mar 30 2026, 7:22 PM

Change #1262052 merged by BCornwall:

[operations/puppet@production] mw-parsoid: remove service definition 4

https://gerrit.wikimedia.org/r/1262052

gerritbot added a comment.Mar 30 2026, 7:27 PM

Change #1262025 merged by Effie Mouzeli:

[operations/dns@master] mw-parsoid: add CNAMES

https://gerrit.wikimedia.org/r/1262025

gerritbot added a comment.Mar 30 2026, 7:30 PM

Change #1262026 merged by Effie Mouzeli:

[operations/puppet@production] trafficserver: update mw-parsoid XWD entries

https://gerrit.wikimedia.org/r/1262026

jijiki closed this task as Resolved.Mar 30 2026, 7:34 PM
jijiki added a subscriber: BCornwall.

thanks @BCornwall for all the help, this work is done!

BCornwall added a parent task: T411097: Deprecate low-traffic proxoid service and O:hcaptcha_proxy for the older hcaptcha proxy setup.Mar 30 2026, 9:00 PM
BCornwall removed a parent task: T411097: Deprecate low-traffic proxoid service and O:hcaptcha_proxy for the older hcaptcha proxy setup.Mar 30 2026, 11:01 PM
gerritbot added a comment.Mar 31 2026, 1:32 PM

Change #1265420 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::service_proxy::envoy: remove mw-parsoid

https://gerrit.wikimedia.org/r/1265420

gerritbot added a comment.Mar 31 2026, 2:13 PM

Change #1262054 merged by Brouberol:

[operations/puppet@production] envoy: remove mw-parsoid listener

https://gerrit.wikimedia.org/r/1262054

Eevans reopened this task as Open.Mar 31 2026, 3:05 PM
Eevans added subscribers: daniel, Eevans.

Puppet on RESTBase is failing with:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, One and only one listener with name 'mw-parsoid' is expected (file: /srv/puppet_code/environments/production/modules/wmflib/functions/service/get_url.pp, line: 77, column: 13) on node restbase1031.eqiad.wmnet
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

In this case 'One and only one...' is because there are zero listeners, this happened after the merging of r1262054. Allegedly we're not using RESTBase's parsoid endpoints for anything any more (/cc @daniel ?), but I don't know what it will do with a url that doesn't work at all (refuse to start? logspam? nothing at all?). Spoiler alert: I think we are about to find out.

gerritbot added a comment.Mar 31 2026, 3:06 PM

Change #1265450 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: unbreak puppet; address missing mw-parsoid listener

https://gerrit.wikimedia.org/r/1265450

gerritbot added a comment.Mar 31 2026, 3:11 PM

Change #1265452 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] fixtures: remove mw-parsoid listener from fixtures after it's been dropped

https://gerrit.wikimedia.org/r/1265452

gerritbot added a comment.Mar 31 2026, 3:14 PM

Change #1265452 merged by Brouberol:

[operations/deployment-charts@master] fixtures: remove mw-parsoid listener from fixtures after it's been dropped

https://gerrit.wikimedia.org/r/1265452

gerritbot added a comment.Mar 31 2026, 3:58 PM

Change #1265450 merged by Eevans:

[operations/puppet@production] restbase: address missing mw-parsoid listener (unbreak puppet)

https://gerrit.wikimedia.org/r/1265450

Eevans added a comment.Mar 31 2026, 4:26 PM
In T420468#11773435, @Eevans wrote:

[ ... ]

In this case 'One and only one...' is because there are zero listeners, this happened after the merging of r1262054. Allegedly we're not using RESTBase's parsoid endpoints for anything any more (/cc @daniel ?), but I don't know what it will do with a url that doesn't work at all (refuse to start? logspam? nothing at all?). Spoiler alert: I think we are about to find out.

r1265450 (merged) updated mw-parsoid to mw-api-int instead, and puppet is now running successfully. The transform endpoint does not work anymore though...

check-restbase
/en.wikipedia.org/v1/page/html/{title} (Get html by title from storage) is CRITICAL: Test Get html by title from storage returned the unexpected status 504 (expecting: 200); /en.wikipedia.org/v1/transform/wikitext/to/html/{title} (Transform wikitext to html) is CRITICAL: Test Transform wikitext to html returned the unexpected status 504 (expecting: 200)

Because there is no local listener present...

eevans@restbase1031:~$ sudo grep parsoid_uri /etc/restbase/config-vars.yaml 
parsoid_uri: http://localhost:6501/w/rest.php
eevans@restbase1031:~$ sudo ss -npl |grep 6501
eevans@restbase1031:~$

This results in some ECONNREFUSED errors...

image.png (1,911×816 px, 131 KB)

... but as far as I can tell, we're only emitting errors for service checks (i.e. when check-restbase is invoked), so I guess this is "OK"?

Eevans closed this task as Resolved.Mar 31 2026, 4:43 PM
JMeybohm reopened this task as Open.Apr 1 2026, 10:00 AM
JMeybohm subscribed.

mw-parsoid is still referenced as discovery listener in multiple places of deployment-charts CI causes various failures when checking charts/deployments

jijiki changed the task status from Open to In Progress.Apr 1 2026, 12:20 PM
gerritbot added a comment.Apr 1 2026, 12:44 PM

Change #1266216 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] Update fixtures and remove mw-parsoid

https://gerrit.wikimedia.org/r/1266216

gerritbot added a comment.Apr 1 2026, 1:59 PM

Change #1266216 merged by jenkins-bot:

[operations/deployment-charts@master] Update fixtures and remove mw-parsoid

https://gerrit.wikimedia.org/r/1266216

gerritbot added a comment.Apr 2 2026, 9:41 AM

Change #1265420 abandoned by Effie Mouzeli:

[operations/puppet@production] profile::service_proxy::envoy: remove mw-parsoid

Reason:

this was merged in I2c626f98f0b3627f8c690a7b0f5e94552decec73

https://gerrit.wikimedia.org/r/1265420

Maintenance_bot removed a project: Patch-For-Review.Apr 2 2026, 10:32 AM
jijiki closed this task as Resolved.Apr 2 2026, 9:45 PM
jijiki mentioned this in T432445: Remove api-gateway service traffic.Fri, Jul 17, 1:27 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits