Page MenuHomePhabricator
Create Task
Maniphest T420436

Migrate Wikikube k8s apiserver and services to IPIP
Open, MediumPublic
Actions

Description

With T352956: Handling inbound IPIP traffic on low traffic LVS k8s based realservers completed and verified on wikikube-staging we now need to migrate all other kubernetes related LVS VIPs to IPIP encapsulation.

I've captured the required steps at:
https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/IPIP

  • apiserves (and the apiserver service) have been migrated to IPIP
  • Prep work (enabling IPIP support on all workers) fo workload is done, we now need to migrate services to IPIP
Services
  • apertium
  • citoid
  • cxserver
  • echostore
  • eventgate-analytics
  • eventgate-logging-external
  • eventgate-analytics-external
  • eventgate-main
  • eventstreams
  • k8s-ingress-wikikube
  • kartotherian-k8s-ssl
  • kubemaster
  • mathoid (hold because T431372: Mark Mathoid endpoints as deprecated)
  • mobileapps
  • mwdebug
  • mwdebug-next
  • mw-web
  • mw-web-next
  • mw-api-ext
  • mw-api-ext-next
  • mw-api-int
  • mw-jobrunner
  • proton
  • push-notifications
  • recommendation-api
  • sessionstore
  • shellbox
  • shellbox-constraints
  • shellbox-media
  • shellbox-syntaxhighlight
  • shellbox-timeline
  • shellbox-video
  • tegola-vector-tiles
  • termbox
  • thumbor
  • toolhub
  • wikifeeds
  • zotero
  • linkrecommendation
  • linkrecommendation-external
  • device-analytics
  • rest-gateway
import yaml
with open("hieradata/common/service.yaml", "r", encoding="utf8") as f:
    data = yaml.safe_load(f)
services = data.get("service::catalog", {})
for name, config in services.items():
    lvs = config.get("lvs", {})
    conftool = lvs.get("conftool", {})
    ipip = lvs.get("ipip_encapsulation", [])
    if conftool.get("cluster") != "kubernetes":
        continue
    if "eqiad" in ipip and "codfw" in ipip:
        print(f"[x] {name}")
        continue
    print(f"[ ] {name}")

Details

Other Assignee
Jelto
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
service::catalog: Set ipip for mobileapps, proton, push-notifications eqiadJeltooperations/puppetproduction+3 -0
service::catalog: Set ipip for mobileapps, proton, push-notifications codfwJeltooperations/puppetproduction+12 -3
service::catalog: Set ipip for echostore, eventstreams, kartotherian eqiadJeltooperations/puppetproduction+3 -0
service::catalog: Set ipip for echostore, eventstreams, kartotherian codfwJeltooperations/puppetproduction+12 -3
service::catalog: Set ipip_encapsulation for eventgate-main in eqiadJeltooperations/puppetproduction+1 -0
service::catalog: Set ipip_encapsulation for eventgate-main in codfwJeltooperations/puppetproduction+4 -1
service::catalog: Set ipip_encapsulation for eventgate-analytics* in eqiadJeltooperations/puppetproduction+2 -0
service::catalog: Set ipip_encapsulation for eventgate-analytics* in codfwJeltooperations/puppetproduction+8 -2
service::catalog: Set ipip_encapsulation for eventgate-logging-external in eqiadJeltooperations/puppetproduction+1 -0
service::catalog: Set ipip_encapsulation for eventgate-logging-external in codfwJeltooperations/puppetproduction+4 -1
service::catalog: Set ipip_encapsulation for citoid, cxserver in eqiadJeltooperations/puppetproduction+2 -0
service::catalog: Set ipip_encapsulation for citoid, cxserver in codfwJeltooperations/puppetproduction+8 -2
service::catalog: Set ipip_encapsulation for apertium in eqiad.Jeltooperations/puppetproduction+1 -0
service::catalog: Set ipip_encapsulation for apertium in codfwJeltooperations/puppetproduction+4 -1
service::catalog: Set ipip_encapsulation for wikikube codfwJeltooperations/puppetproduction+4 -1
wikikube: Switch to IPIP mode on workersJMeybohmoperations/puppetproduction+11 -0
wikikube: Enable ipip_encapsulation and mh schedulerJMeybohmoperations/puppetproduction+5 -1
wikikube: Switch to IPIP mode for kube-apiserverJMeybohmoperations/puppetproduction+2 -0
wikikube: Switch to IPIP mode for kube-apiserverJMeybohmoperations/puppetproduction+6 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

JMeybohm created this task.Mar 18 2026, 9:55 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 18 2026, 9:55 AM
JMeybohm mentioned this in T352956: Handling inbound IPIP traffic on low traffic LVS k8s based realservers.Mar 18 2026, 10:01 AM
JMeybohm edited projects, added: ServiceOps (Next quarter); removed: ServiceOps.Mar 18 2026, 10:07 AM
JMeybohm created subtask T420729: Add missing wikikube workers to conftool-data.Mar 20 2026, 1:15 PM
gerritbot added a comment.Mar 24 2026, 2:55 PM

Change #1259995 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] wikikube: Switch to IPIP mode for kube-apiserver

https://gerrit.wikimedia.org/r/1259995

gerritbot added a project: Patch-For-Review.Mar 24 2026, 2:55 PM

Change #1259996 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] wikikube: Enable ipip_encapsulation and mh scheduler

https://gerrit.wikimedia.org/r/1259996

Blake subscribed.Mar 24 2026, 3:35 PM
gerritbot added a comment.Mar 25 2026, 11:11 AM

Change #1259995 merged by JMeybohm:

[operations/puppet@production] wikikube: Switch to IPIP mode for kube-apiserver

https://gerrit.wikimedia.org/r/1259995

gerritbot added a comment.Mar 25 2026, 11:23 AM

Change #1260654 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] wikikube: Switch to IPIP mode for kube-apiserver

https://gerrit.wikimedia.org/r/1260654

gerritbot added a comment.Mar 25 2026, 11:25 AM

Change #1260654 merged by JMeybohm:

[operations/puppet@production] wikikube: Switch to IPIP mode for kube-apiserver

https://gerrit.wikimedia.org/r/1260654

gerritbot added a comment.Mar 25 2026, 11:40 AM

Change #1259996 merged by JMeybohm:

[operations/puppet@production] wikikube: Enable ipip_encapsulation and mh scheduler

https://gerrit.wikimedia.org/r/1259996

Stashbot added a comment.Mar 25 2026, 11:51 AM

Mentioned in SAL (#wikimedia-operations) [2026-03-25T11:51:13Z] <jayme> migrated wikikube apiservers (eqiad and codfw) to IPIP - T420436

Maintenance_bot removed a project: Patch-For-Review.Mar 25 2026, 12:31 PM
gerritbot added a comment.Mar 25 2026, 2:51 PM

Change #1260723 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] wikikube: Switch to IPIP mode on workers

https://gerrit.wikimedia.org/r/1260723

gerritbot added a project: Patch-For-Review.Mar 25 2026, 2:51 PM
Stashbot added a comment.Mar 30 2026, 1:05 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T13:05:12Z] <jayme> disabling puppet on A:wikiube-worker-eqiad for T420436

gerritbot added a comment.Mar 30 2026, 1:07 PM

Change #1260723 merged by JMeybohm:

[operations/puppet@production] wikikube: Switch to IPIP mode on workers

https://gerrit.wikimedia.org/r/1260723

Stashbot added a comment.Mar 30 2026, 1:59 PM

Mentioned in SAL (#wikimedia-operations) [2026-03-30T13:59:17Z] <jayme> enabling puppet on A:wikiube-worker-eqiad for T420436

Maintenance_bot removed a project: Patch-For-Review.Mar 30 2026, 2:32 PM
JMeybohm added a comment.Mar 30 2026, 2:52 PM

After enabling IPIP on the workers there are two which do not accept IPIP packets:

RuntimeError: wikikube-worker2347.codfw.wmnet is not accepting incoming IPIP traffic:  
                                           outer IP header: 172.16.1.1 -> 10.192.58.10  
                                           inner IP header: 10.64.16.154 -> 10.2.1.70  
                                           destination port: 30443

 RuntimeError: wikikube-worker1347.eqiad.wmnet is not accepting incoming IPIP traffic:  
                                           outer IP header: 172.16.1.1 -> 10.64.181.4  
                                           inner IP header: 10.64.16.154 -> 10.2.2.70  
                                           destination port: 30443
JMeybohm added a comment.Mar 31 2026, 8:45 AM

wikikube-worker2347.codfw.wmnet is fine after a ferm restart. For whatever reason it was unable to reload and apply the accept rules for ipip

root@wikikube-worker2347:~# sudo iptables -L -n | grep 172.16.0.0/12
root@wikikube-worker2347:~# systemctl status ferm
● ferm.service - ferm firewall configuration
     Loaded: loaded (/usr/lib/systemd/system/ferm.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/ferm.service.d
             └─ferm-service-auto-restart.conf, ferm-service-status-restart.conf
     Active: active (exited) since Mon 2026-03-23 13:59:21 UTC; 1 week 0 days ago
 Invocation: eb0989e0768c43389eb94d678b179f63
    Process: 692638 ExecReload=/usr/local/sbin/ferm-status --start-ferm (code=exited, status=1/FAILURE)
   Main PID: 2732 (code=exited, status=0/SUCCESS)
   Mem peak: 49.1M
        CPU: 604ms

Mar 23 13:59:21 wikikube-worker2347 ferm[2732]: Starting Firewall: ferm.
Mar 23 13:59:21 wikikube-worker2347 systemd[1]: Finished ferm.service - ferm firewall configuration.
Mar 30 13:38:20 wikikube-worker2347 systemd[1]: Reloading ferm.service - ferm firewall configuration...
Mar 30 13:38:21 wikikube-worker2347 ferm-status[692638]: Starting ferm
Mar 30 13:38:21 wikikube-worker2347 ferm-status[692693]: Can't lock /run/xtables.lock: Resource temporarily unavailable
Mar 30 13:38:21 wikikube-worker2347 ferm-status[692693]: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?
Mar 30 13:38:21 wikikube-worker2347 ferm-status[692686]: Failed to run /usr/sbin/ip6tables-legacy-restore
Mar 30 13:38:21 wikikube-worker2347 ferm-status[692686]: Firewall rules rolled back.
Mar 30 13:38:21 wikikube-worker2347 systemd[1]: ferm.service: Control process exited, code=exited, status=1/FAILURE
Mar 30 13:38:21 wikikube-worker2347 systemd[1]: Reload failed for ferm.service - ferm firewall configuration.
root@wikikube-worker2347:~# lsof /run/xtables.lock
root@wikikube-worker2347:~# systemctl restart ferm
root@wikikube-worker2347:~# sudo iptables -L -n | grep 172.16.0.0/12
ACCEPT     4    --  172.16.0.0/12        0.0.0.0/0

On wikikube-worker1347.eqiad.wmnet the istio-ingressgateway is unhappy since it can't lookup istiod's IP to update/load it's config.

JMeybohm added a comment.Mar 31 2026, 3:37 PM
In T420436#11771019, @JMeybohm wrote:

On wikikube-worker1347.eqiad.wmnet the istio-ingressgateway is unhappy since it can't lookup istiod's IP to update/load it's config.

That was something similar to T417817: Test Nokia switches BGP config for k8s workers (but for a different Nokia version).

With all that out of the way we are good to go for moving services to IPIP.

JMeybohm updated the task description. (Show Details)Mar 31 2026, 3:38 PM
MLechvien-WMF moved this task from Next quarter to Backlog on the ServiceOps board.Apr 1 2026, 10:29 AM
MLechvien-WMF edited projects, added: ServiceOps; removed: ServiceOps (Next quarter).
MLechvien-WMF moved this task from Backlog to Next quarter on the ServiceOps board.May 21 2026, 10:50 AM
MLechvien-WMF edited projects, added: ServiceOps (Next quarter); removed: ServiceOps.
CDanis subscribed.Jun 3 2026, 2:42 PM
MLechvien-WMF triaged this task as Medium priority.Jun 5 2026, 7:33 AM
MLechvien-WMF moved this task from Backlog to Confirmed on the ServiceOps (Next quarter) board.Jun 10 2026, 7:40 PM
MLechvien-WMF edited projects, added: ServiceOps; removed: ServiceOps (Next quarter).Jun 30 2026, 7:50 AM
MLechvien-WMF assigned this task to JMeybohm.Jun 30 2026, 7:56 AM
MLechvien-WMF updated Other Assignee, added: Jelto.
MLechvien-WMF moved this task from Inbox to Scheduled (this Q) on the ServiceOps board.
gerritbot added a comment.Jul 16 2026, 12:24 PM

Change #1311444 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip_encapsulation for wikikube codfw

https://gerrit.wikimedia.org/r/1311444

gerritbot added a project: Patch-For-Review.Jul 16 2026, 12:24 PM
JMeybohm updated the task description. (Show Details)Mon, Aug 17, 12:14 PM
JMeybohm updated the task description. (Show Details)Thu, Aug 20, 1:50 PM
Jelto subscribed.Thu, Aug 20, 2:04 PM
gerritbot added a comment.Sun, Aug 23, 11:56 AM

Change #1328348 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip_encapsulation for apertium in codfw

https://gerrit.wikimedia.org/r/1328348

gerritbot added a comment.Sun, Aug 23, 11:56 AM

Change #1328349 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip_encapsulation for apertium in eqiad.

https://gerrit.wikimedia.org/r/1328349

gerritbot added a comment.Tue, Aug 25, 8:57 AM

Change #1328348 merged by Jelto:

[operations/puppet@production] service::catalog: Set ipip_encapsulation for apertium in codfw

https://gerrit.wikimedia.org/r/1328348

gerritbot added a comment.Tue, Aug 25, 1:21 PM

Change #1328349 merged by Jelto:

[operations/puppet@production] service::catalog: Set ipip_encapsulation for apertium in eqiad.

https://gerrit.wikimedia.org/r/1328349

Jelto updated the task description. (Show Details)Tue, Aug 25, 1:26 PM
gerritbot added a comment.Mon, Aug 31, 2:22 PM

Change #1332741 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip_encapsulation for citoid, cxserver in codfw

https://gerrit.wikimedia.org/r/1332741

gerritbot added a comment.Wed, Sep 2, 9:00 AM

Change #1333691 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip_encapsulation for citoid, cxserver in eqiad

https://gerrit.wikimedia.org/r/1333691

gerritbot added a comment.Wed, Sep 2, 12:21 PM

Change #1332741 merged by Jelto:

[operations/puppet@production] service::catalog: Set ipip_encapsulation for citoid, cxserver in codfw

https://gerrit.wikimedia.org/r/1332741

Jelto added a subscriber: ssingh.EditedThu, Sep 3, 8:53 AM

@ssingh reported in wikimedia-traffic a pending pybal restart and manually restarted pybal.

I used the sre.loadbalancer.migrate-service-ipip cookbook which should do pybal restart. In the cookbook log there is:

==> Press continue when you are ready to restart pybal.
Type "go" to proceed or "abort" to interrupt the execution
> go
User input is: "go"
START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
Running action: restart_daemons on hosts lvs2013.codfw.wmnet
...
Running action: restart_daemons on hosts lvs2014.codfw.wmnet
...
All restart_daemons were successful
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs

Also SAL logs a restart of pybal in codfw:

START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs

In SAL I also see that you @ssingh restarted LVS in eqiad:

sukhe@lvs1019:~$ sudo systemctl restart pybal.service
sukhe@lvs1020:~$ sudo systemctl restart pybal.service

So I assume the IPIP migration change from this patch also caused scheduler changes in eqiad, while IPIP was enabled in codfw only? And for the future services should be migrated in both DCs at the same time to not cause pending pybal changes?

gerritbot added a comment.Thu, Sep 3, 9:35 AM

Change #1333691 merged by Jelto:

[operations/puppet@production] service::catalog: Set ipip_encapsulation for citoid, cxserver in eqiad

https://gerrit.wikimedia.org/r/1333691

Jelto updated the task description. (Show Details)Thu, Sep 3, 9:40 AM
gerritbot added a comment.Thu, Sep 3, 10:53 AM

Change #1334801 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip_encapsulation for eventgate-logging-external in codfw

https://gerrit.wikimedia.org/r/1334801

gerritbot added a comment.Thu, Sep 3, 10:53 AM

Change #1334802 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip_encapsulation for eventgate-logging-external in eqiad

https://gerrit.wikimedia.org/r/1334802

gerritbot added a comment.Thu, Sep 3, 10:53 AM

Change #1334803 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip_encapsulation for eventgate-analytics* in codfw

https://gerrit.wikimedia.org/r/1334803

gerritbot added a comment.Thu, Sep 3, 10:53 AM

Change #1334804 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip_encapsulation for eventgate-analytics* in eqiad

https://gerrit.wikimedia.org/r/1334804

gerritbot added a comment.Thu, Sep 3, 10:53 AM

Change #1334805 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip_encapsulation for eventgate-main in codfw

https://gerrit.wikimedia.org/r/1334805

gerritbot added a comment.Thu, Sep 3, 10:53 AM

Change #1334806 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip_encapsulation for eventgate-main in eqiad

https://gerrit.wikimedia.org/r/1334806

gerritbot added a comment.Mon, Sep 7, 9:50 AM

Change #1334801 merged by Jelto:

[operations/puppet@production] service::catalog: Set ipip_encapsulation for eventgate-logging-external in codfw

https://gerrit.wikimedia.org/r/1334801

gerritbot added a comment.Mon, Sep 7, 10:08 AM

Change #1334802 merged by Jelto:

[operations/puppet@production] service::catalog: Set ipip_encapsulation for eventgate-logging-external in eqiad

https://gerrit.wikimedia.org/r/1334802

Jelto updated the task description. (Show Details)Mon, Sep 7, 10:12 AM
gerritbot added a comment.Mon, Sep 7, 12:58 PM

Change #1334803 merged by Jelto:

[operations/puppet@production] service::catalog: Set ipip_encapsulation for eventgate-analytics* in codfw

https://gerrit.wikimedia.org/r/1334803

gerritbot added a comment.Mon, Sep 7, 1:16 PM

Change #1334804 merged by Jelto:

[operations/puppet@production] service::catalog: Set ipip_encapsulation for eventgate-analytics* in eqiad

https://gerrit.wikimedia.org/r/1334804

Jelto updated the task description. (Show Details)Mon, Sep 7, 1:27 PM
gerritbot added a comment.Mon, Sep 7, 2:26 PM

Change #1334805 merged by Jelto:

[operations/puppet@production] service::catalog: Set ipip_encapsulation for eventgate-main in codfw

https://gerrit.wikimedia.org/r/1334805

gerritbot added a comment.Mon, Sep 7, 2:37 PM

Change #1334806 merged by Jelto:

[operations/puppet@production] service::catalog: Set ipip_encapsulation for eventgate-main in eqiad

https://gerrit.wikimedia.org/r/1334806

Jelto updated the task description. (Show Details)Mon, Sep 7, 2:42 PM
Jelto added a comment.Mon, Sep 7, 2:46 PM

eventgate* was migrated to IPIP. None of the migrations caused any latency peaks or other problems so far. So we should be good to convert bigger batches of services. I'll upload changes.

gerritbot added a comment.Tue, Sep 8, 8:30 AM

Change #1337851 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip for echostore, eventstreams, kartotherian

https://gerrit.wikimedia.org/r/1337851

gerritbot added a comment.Tue, Sep 8, 8:30 AM

Change #1337852 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip_encapsulation for echostore, eventstreams, kartotherian

https://gerrit.wikimedia.org/r/1337852

gerritbot added a comment.Tue, Sep 8, 12:50 PM

Change #1337851 merged by Jelto:

[operations/puppet@production] service::catalog: Set ipip for echostore, eventstreams, kartotherian codfw

https://gerrit.wikimedia.org/r/1337851

gerritbot added a comment.Tue, Sep 8, 1:05 PM

Change #1337852 merged by Jelto:

[operations/puppet@production] service::catalog: Set ipip for echostore, eventstreams, kartotherian eqiad

https://gerrit.wikimedia.org/r/1337852

Jelto updated the task description. (Show Details)Tue, Sep 8, 1:10 PM
gerritbot added a comment.Tue, Sep 8, 1:19 PM

Change #1337923 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip for mobileapps, proton, push-notifications codfw

https://gerrit.wikimedia.org/r/1337923

gerritbot added a comment.Tue, Sep 8, 1:19 PM

Change #1337924 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] service::catalog: Set ipip for mobileapps, proton, push-notifications eqiad

https://gerrit.wikimedia.org/r/1337924

gerritbot added a comment.Thu, Sep 10, 1:37 PM

Change #1337923 merged by Jelto:

[operations/puppet@production] service::catalog: Set ipip for mobileapps, proton, push-notifications codfw

https://gerrit.wikimedia.org/r/1337923

gerritbot added a comment.Thu, Sep 10, 1:51 PM

Change #1337924 merged by Jelto:

[operations/puppet@production] service::catalog: Set ipip for mobileapps, proton, push-notifications eqiad

https://gerrit.wikimedia.org/r/1337924

Jelto updated the task description. (Show Details)Thu, Sep 10, 1:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits