Implement the MediaWiki Singleversion experimentation outlined in FY 2024-2025 WE6.2.5 and WE6.2.9 Hypothesis
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	akosiaris
	Jan 28 2025, 3:59 PM

Description

Intro

SingleVersion MW: Routing Options document tries to document how HTTP traffic flows into MediaWiki Multiversion work as well as a breakdown of various options on how to route traffic in a single version container orchestrated environment. While the doc is (hopefully) a good read, the intent of this task is to document the progress with implementing the chosen approached, dubbed *WikiKube Ingress Gateway*. Some preliminary PoCs have already proved functional, this is about proving this will work at scale and with all the details taken into account.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
hieradata: remove mw-wikifunctions discovery services	operations/puppet	production	+0 -5
Remove old mw-wikifunctions RRs	operations/dns	master	+2 -2
mw-wikifunctions: Remove default from gatewayHosts	operations/deployment-charts	master	+0 -1
service: Cleanup of wikifunctions	operations/puppet	production	+0 -3
mesh: Use http_host as well for mw-wikifunctions	operations/puppet	production	+1 -0
mesh: Use sets_sni for mw-wikifuctions	operations/puppet	production	+1 -0
mw-wikifunctions: Add a missing SAN	operations/deployment-charts	master	+2 -0
wikifunctions: Switch to ingress service	operations/puppet	production	+3 -19
wikifunctions: Switch to ingress service	operations/puppet	production	+2 -19
Add group{0,1,2} and pretrain releases in mw-api-int staging	operations/deployment-charts	master	+92 -0
mw-wikifunctions: Switch DNS to use ingress	operations/dns	master	+4 -6
wikifunctions: Disable paging	operations/puppet	production	+1 -1
admin_ng: Preserve Server header in ingressgateway	operations/deployment-charts	master	+34 -1
Revert^4 "cache::backend: Switch mw-wikifunctions to ingress"	operations/puppet	production	+6 -15
mw-wikifunctions-ingress: Switch to k8s-ingress-wikikube-rw	operations/dns	master	+1 -1
mw-wikifuctions: Add main FQDNS in tlsExtraSANs	operations/deployment-charts	master	+3 -1
cache::backend: Switch mw-wikifunctions to ingress	operations/puppet	production	+3 -3
Add wikifunctions-ingress-ro records	operations/dns	master	+4 -0
mw-wikifunctions: Add an extra SAN	operations/deployment-charts	master	+7 -1
wikifunctions: Remove ports from httproutes	operations/deployment-charts	master	+0 -6
scap: Add 3 releases in mw-wikifunctions	operations/puppet	production	+3 -0
wikifunctions: Add group{0,1,2} releases	operations/deployment-charts	master	+94 -2
mediawiki: Bump ingress to module 1.2.0	operations/deployment-charts	master	+58 -8
ingress: Add the ability to have >1 destination rules	operations/deployment-charts	master	+28 -5
ingress: Add the ability to have >1 destination rules	operations/deployment-charts	master	+190 -0
mediawiki: Bump ingress module to 1.1.0	operations/deployment-charts	master	+8 -47

Related Objects

Mentioned In: T401396: Revisit backend routing for rest-gateway
T397874: Assess switchover behavior for mw-wikifunctions
T395451: Make the JobQueue compatible with the MediaWiki Single version HTTP routing system
T389375: Investigate Latency When Processing Large Objects in Evaluator

Event Timeline

akosiaris created this task.Jan 28 2025, 3:59 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 28 2025, 3:59 PM

JMeybohm subscribed.Jan 28 2025, 4:16 PM

• mszabo subscribed.Jan 28 2025, 4:22 PM

Change #1115889 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] Add group{0,1,2} and pretrain releases in mw-api-int staging

https://gerrit.wikimedia.org/r/1115889

gerritbot added a project: Patch-For-Review.Jan 31 2025, 3:34 PM

Quick update, the patch above works.

curl --http1.1 -k -v -H "Host: test.wikipedia.org" https://mediawiki.k8s-staging.discovery.wmnet:30443

from any host in the infra routes to a release named group0 which is introduced in the patch. group0 for now is just testwiki and test2wiki, group1 just wikidata and everything else goes to the group2 release. I 'll need to generate the entire list and populate it, which was expected per the linked doc. HTTP routing works fine, MediaWiki errors out with Uncaught MediaWiki\Config\ConfigException: Failed to load configuration from etcd: (curl error: 1) Unsupported protocol in /srv/mediawiki/php-1.44.0-wmf.14/includes/config/EtcdConfig.php:231</code></p></div> which is probably some networking issue I 'll figure out later. The good news is that the HTTP routing works.

There is also a tentative pretrain release named in the patch, but I have setup no routes for it since we haven't yet targetted a specific wiki.

jijiki moved this task from Incoming 🐫 to Doing 😎 on the serviceops-deprecated board.Feb 3 2025, 2:12 PM

Krinkle subscribed.Feb 7 2025, 4:51 AM

Change #1131636 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] mediawiki: Bump ingress module to 1.1.0

https://gerrit.wikimedia.org/r/1131636

Change #1131692 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] ingress: Add the ability to have >1 destination rules

https://gerrit.wikimedia.org/r/1131692

Change #1131693 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] ingress: Add the ability to have >1 destination rules

https://gerrit.wikimedia.org/r/1131693

Change #1131694 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] mediawiki: Bump ingress to module 1.2.0

https://gerrit.wikimedia.org/r/1131694

Change #1131636 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Bump ingress module to 1.1.0

https://gerrit.wikimedia.org/r/1131636

Using the following configuration for a mediawiki helm release named group0 deployed in the staging cluster

ingress:
  enabled: true
  gatewayHosts:
    default: "mediawiki"
    extraFQDNs:
      # And now everything we receive traffic for
      - '*.wikipedia.org'
      - '*.mediawiki.org'
      - '*.wikibooks.org'
      - '*.wikidata.org'
      - '*.wikimedia.org'
      - '*.wikimediafoundation.org'
      - '*.wikinews.org'
      - '*.wikiquote.org'
      - '*.wikisource.org'
      - '*.wikiversity.org'
      - '*.wikivoyage.org'
      - '*.wiktionary.org'
      - '*.wmfusercontent.org'
      - 'mediawiki.org'
      - 'w.wiki'
      - 'wikibooks.org'
      - 'wikidata.org'
      - 'wikimedia.org'
      - 'wikimediafoundation.org'
      - 'wikinews.org'
      - 'wikiquote.org'
      - 'wikisource.org'
      - 'wikiversity.org'
      - 'wikivoyage.org'
      - 'wiktionary.org'
      - 'wmfusercontent.org'
      - 'wikipedia.org'
  httproutes:
    - name: group0
      match:
        - authority:
            regex: (test|test2).wikipedia.org
      route:
        - destination:
            host: mediawiki-group0-tls-service.mw-api-int.svc.cluster.local
    - name: group1
      match:
        - authority:
            exact: wikidata.org
      route:
        - destination:
            host: mediawiki-group1-tls-service.mw-api-int.svc.cluster.local
    - name: group2 # default to everything in group2
      route:
        - destination:
            host: mediawiki-group2-tls-service.mw-api-int.svc.cluster.local
  destinationrulehosts:
    - mediawiki-group0-tls-service.mw-api-int.svc.cluster.local
    - mediawiki-group1-tls-service.mw-api-int.svc.cluster.local
    - mediawiki-group2-tls-service.mw-api-int.svc.cluster.local

having another 2 releases named group1, group2 that differ from group0 only in the fact they don't have the above stanza in their helm values we got

we have

$ curl -vk --http1.1 -H "Host: test.wikipedia.org" https://mediawiki.k8s-staging.discovery.wmnet:30443/wiki/Main_Page |& grep -E 'Parsed by'
Parsed by mw‐api‐int.eqiad.group0‐557f898bd5‐w6gxn
$ curl -vk --http1.1 -H "Host: wikidata.org" https://mediawiki.k8s-staging.discovery.wmnet:30443/wiki/Main_Page |& grep -iE 'location:|server:'
< server: istio-envoy
< location: https://www.wikidata.org/wiki/Main_Page
$ curl -vk --http1.1 -H "Host: en.wikipedia.org" https://mediawiki.k8s-staging.discovery.wmnet:30443/wiki/Main_Page |& grep -E 'server:'
< server: istio-envoy

which is what we expect indeed to see for all 3 calls.

Change #1131692 merged by jenkins-bot:

[operations/deployment-charts@master] ingress: Add the ability to have >1 destination rules

https://gerrit.wikimedia.org/r/1131692

Change #1131693 merged by jenkins-bot:

[operations/deployment-charts@master] ingress: Add the ability to have >1 destination rules

https://gerrit.wikimedia.org/r/1131693

Change #1131694 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Bump ingress to module 1.2.0

https://gerrit.wikimedia.org/r/1131694

Change #1132684 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] wikifunctions: Add group{0,1,2} releases

https://gerrit.wikimedia.org/r/1132684

Change #1132690 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] scap: Add 3 releases in mw-wikifunctions

https://gerrit.wikimedia.org/r/1132690

Change #1132691 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] wikifunctions: Switch to ingress service

https://gerrit.wikimedia.org/r/1132691

Change #1132684 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Add group{0,1,2} releases

https://gerrit.wikimedia.org/r/1132684

Change #1132690 merged by Alexandros Kosiaris:

[operations/puppet@production] scap: Add 3 releases in mw-wikifunctions

https://gerrit.wikimedia.org/r/1132690

Change #1133161 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] wikifunctions: Remove ports from httproutes

https://gerrit.wikimedia.org/r/1133161

Change #1133161 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Remove ports from httproutes

https://gerrit.wikimedia.org/r/1133161

Change #1133318 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] mw-wikifunctions: Add an extra SAN

https://gerrit.wikimedia.org/r/1133318

Change #1133318 merged by jenkins-bot:

[operations/deployment-charts@master] mw-wikifunctions: Add an extra SAN

https://gerrit.wikimedia.org/r/1133318

Change #1133335 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/dns@master] Add wikifunctions-ingress-ro records

https://gerrit.wikimedia.org/r/1133335

Change #1133335 merged by Alexandros Kosiaris:

[operations/dns@master] Add wikifunctions-ingress-ro records

https://gerrit.wikimedia.org/r/1133335

Change #1133343 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] cache::backend: Switch mw-wikifunctions to ingress

https://gerrit.wikimedia.org/r/1133343

Change #1133343 merged by Alexandros Kosiaris:

[operations/puppet@production] cache::backend: Switch mw-wikifunctions to ingress

https://gerrit.wikimedia.org/r/1133343

Change #1133356 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] mw-wikifuctions: Add main FQDNS in tlsExtraSANs

https://gerrit.wikimedia.org/r/1133356

Change #1133356 merged by jenkins-bot:

[operations/deployment-charts@master] mw-wikifuctions: Add main FQDNS in tlsExtraSANs

https://gerrit.wikimedia.org/r/1133356

Change #1133372 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/dns@master] mw-wikifunctions-ingress: Switch to k8s-ingress-wikikube-rw

https://gerrit.wikimedia.org/r/1133372

Change #1133372 merged by Alexandros Kosiaris:

[operations/dns@master] mw-wikifunctions-ingress: Switch to k8s-ingress-wikikube-rw

https://gerrit.wikimedia.org/r/1133372

bd808 subscribed.Apr 2 2025, 9:52 PM

Change #1133745 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] admin_ng: Preserve Server header in ingressgateway

https://gerrit.wikimedia.org/r/1133745

Change #1133812 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] Revert^4 "cache::backend: Switch mw-wikifunctions to ingress"

https://gerrit.wikimedia.org/r/1133812

Change #1133812 merged by Alexandros Kosiaris:

[operations/puppet@production] Revert^4 "cache::backend: Switch mw-wikifunctions to ingress"

https://gerrit.wikimedia.org/r/1133812

Change #1133821 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] wikifunctions: Move to lvs_setup, disabling paging

https://gerrit.wikimedia.org/r/1133821

Change #1133745 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: Preserve Server header in ingressgateway

https://gerrit.wikimedia.org/r/1133745

Change #1133878 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/dns@master] mw-wikifunctions: Switch DNS to use ingress

https://gerrit.wikimedia.org/r/1133878

Change #1133821 merged by Alexandros Kosiaris:

[operations/puppet@production] wikifunctions: Disable paging

https://gerrit.wikimedia.org/r/1133821

Change #1133878 merged by Alexandros Kosiaris:

[operations/dns@master] mw-wikifunctions: Switch DNS to use ingress

https://gerrit.wikimedia.org/r/1133878

Change #1115889 abandoned by Alexandros Kosiaris:

[operations/deployment-charts@master] Add group{0,1,2} and pretrain releases in mw-api-int staging

Reason:

This was a PoC and has worked ok but is not needed anymore, abandoning.

https://gerrit.wikimedia.org/r/1115889

Change #1133932 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] wikifunctions: Switch to ingress service

https://gerrit.wikimedia.org/r/1133932

Change #1133932 merged by Alexandros Kosiaris:

[operations/puppet@production] wikifunctions: Switch to ingress service

https://gerrit.wikimedia.org/r/1133932

Change #1133940 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] service: Cleanup of wikifunctions

https://gerrit.wikimedia.org/r/1133940

Change #1134017 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] mw-wikifunctions: Add a missing SAN

https://gerrit.wikimedia.org/r/1134017

Change #1134020 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] mesh: Use sets_sni for mw-wikifuctions

https://gerrit.wikimedia.org/r/1134020

Change #1132691 abandoned by Alexandros Kosiaris:

[operations/puppet@production] wikifunctions: Switch to ingress service

Reason:

duplicate of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1133932

https://gerrit.wikimedia.org/r/1132691

Change #1134017 merged by jenkins-bot:

[operations/deployment-charts@master] mw-wikifunctions: Add a missing SAN

https://gerrit.wikimedia.org/r/1134017

Change #1134020 merged by Alexandros Kosiaris:

[operations/puppet@production] mesh: Use sets_sni for mw-wikifuctions

https://gerrit.wikimedia.org/r/1134020

Change #1134050 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] mesh: Use http_host as well for mw-wikifunctions

https://gerrit.wikimedia.org/r/1134050

Change #1134050 abandoned by Alexandros Kosiaris:

[operations/puppet@production] mesh: Use http_host as well for mw-wikifunctions

Reason:

This wouldn't work. See last comment.

https://gerrit.wikimedia.org/r/1134050

Change #1133940 merged by Alexandros Kosiaris:

[operations/puppet@production] service: Cleanup of wikifunctions

https://gerrit.wikimedia.org/r/1133940

Maintenance_bot removed a project: Patch-For-Review.Apr 4 2025, 12:30 PM

I 'll resolve this. wikifunctions.org is now routed via the WikiKube ingress, meaning that we have the ability to create rules that work just fine. Codifying wikiversions.json in this new format is a job for another task. For completeness sake, wikifunctions.org Ingress routing routes wikifunctions.org to a Helm release named group0 and www.wikifunctions.org to a Helm release of MediaWiki called group1. group2 also exists but receives no traffic.

akosiaris closed this task as Resolved.Apr 4 2025, 2:38 PM

Change #1134251 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] mw-wikifunctions: Remove default from gatewayHosts

https://gerrit.wikimedia.org/r/1134251

gerritbot added a project: Patch-For-Review.Apr 4 2025, 3:16 PM

thcipriani awarded a token.Apr 4 2025, 3:25 PM

Change #1134251 merged by jenkins-bot:

[operations/deployment-charts@master] mw-wikifunctions: Remove default from gatewayHosts

https://gerrit.wikimedia.org/r/1134251

Maintenance_bot removed a project: Patch-For-Review.Apr 4 2025, 7:30 PM

bd808 awarded a token.Apr 5 2025, 12:01 AM

akosiaris mentioned this in T389375: Investigate Latency When Processing Large Objects in Evaluator.May 16 2025, 9:11 PM

akosiaris mentioned this in T395451: Make the JobQueue compatible with the MediaWiki Single version HTTP routing system.May 28 2025, 1:56 PM

Change #1163856 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] hieradata: remove mw-wikifunctions discovery services

https://gerrit.wikimedia.org/r/1163856

gerritbot added a project: Patch-For-Review.Jun 25 2025, 7:20 PM

Scott_French mentioned this in T397874: Assess switchover behavior for mw-wikifunctions.Jun 25 2025, 8:02 PM

Change #1164174 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/dns@master] Remove old mw-wikifunctions RRs

https://gerrit.wikimedia.org/r/1164174

Change #1164174 merged by Alexandros Kosiaris:

[operations/dns@master] Remove old mw-wikifunctions RRs

https://gerrit.wikimedia.org/r/1164174

Change #1163856 merged by Alexandros Kosiaris:

[operations/puppet@production] hieradata: remove mw-wikifunctions discovery services

https://gerrit.wikimedia.org/r/1163856

Maintenance_bot removed a project: Patch-For-Review.Jun 26 2025, 1:31 PM

Clement_Goubert mentioned this in T401396: Revisit backend routing for rest-gateway.Nov 18 2025, 11:45 AM

Implement the MediaWiki Singleversion experimentation outlined in FY 2024-2025 WE6.2.5 and WE6.2.9 HypothesisClosed, ResolvedPublicActions

Description

Intro

Details

Related Objects

Event Timeline

Implement the MediaWiki Singleversion experimentation outlined in FY 2024-2025 WE6.2.5 and WE6.2.9 Hypothesis
Closed, ResolvedPublic
Actions