Page MenuHomePhabricator
Create Task
Maniphest T351069

Enable IPIP encapsulation for ncredir
Closed, ResolvedPublic
Actions

Description

We now have all the pieces needed to use IPIP encapsulation with load balanced services. A good guinea pig to test it could be ncredir. The following things are required to enable IPIP encapsulation there:

  • update pybal
  • deploy ipip-multiqueue-optimizer on LVS instances
  • deploy tcp-mss-clamper on ncredir instances
  • Allow inbound IPIP traffic on ncredir instances (T352143)
  • Disable rp filter

Enabled in:

  • eqiad
  • codfw
  • esams
  • ulsfo
  • eqsin
  • drmrs

Details

SubjectRepoBranchLines +/-
P:lvs: set monitoring enabled for IPIP-related servicesoperations/puppetproduction+10 -6
traffic: Provide a dashboard link for LVSRealServerMSSoperations/alertsmaster+5 -4
traffic: Alert on configured and observed MSS mismatchoperations/alertsmaster+56 -0
hiera: Unify ncredir IPIP encapsulation settingsoperations/puppetproduction+2 -18
hiera: Enable IPIP encapsulation on text|secondary LVS in eqiadoperations/puppetproduction+7 -0
hiera: Enable IPIP encapsulation on ncredir@eqiadoperations/puppetproduction+1 -0
hiera: Disable rp_filter on ncredir@eqiadoperations/puppetproduction+2 -0
hiera: Enable IPIP encapsulation on text|secondary LVS in esamsoperations/puppetproduction+5 -0
hiera: Enable IPIP encapsulation on ncredir@esamsoperations/puppetproduction+1 -0
hiera: Disable rp_filter for ncredir@esamsoperations/puppetproduction+2 -0
hiera: Enable IPIP on text|secondary LVS in drmrsoperations/puppetproduction+5 -0
hiera: Enable IPIP encapsulation on ncredir@drmrsoperations/puppetproduction+1 -0
hiera: Disable rp_filter for ncredir@drmrsoperations/puppetproduction+2 -0
hiera: Enable IPIP on eqsin text|secondary LVSoperations/puppetproduction+5 -0
hiera: Enable IPIP encapsulation on ncredir@eqsinoperations/puppetproduction+1 -0
hiera: Disable rp_filter on ncredir@eqsinoperations/puppetproduction+2 -0
prometheus::sysctl: Support configurable sysctlsoperations/puppetproduction+117 -24
lvs::realserver::ipip: Check that TCP MSS clamping is workingoperations/puppetproduction+147 -1
lvs::realserver::ipip: Clamp on lo toooperations/puppetproduction+1 -0
hiera: Enable IPIP on codfw text|secondary LVSoperations/puppetproduction+7 -0
ncredir: Enable IPIP encapsulation on codfwoperations/puppetproduction+3 -0
prometheus::ops: Fix lvs_realserver_clamper configoperations/puppetproduction+5 -9
service: Disable IPIP encapsulation for ncredir@ulsfooperations/puppetproduction+0 -2
service: Enable IPIP encapsulation for ncredir-https toooperations/puppetproduction+4 -0
profile::lvs: Fix ipip-multiqueue-optimizer systemd unitoperations/puppetproduction+2 -2
hiera: Enable IPIP on ulsfo text|secondary LVSoperations/puppetproduction+5 -0
interface::clsact: Fix unless cmdoperations/puppetproduction+2 -2
ncredir: Enable IPIP encapsulation on ulsfooperations/puppetproduction+6 -0
lvs,pybal: Deploy ipip-multiqueue-optimizer for IPIP enabled balancersoperations/puppetproduction+67 -6
profile: Provide a lvs::realserver::ipip profileoperations/puppetproduction+96 -1
interface::ipip: Fix absentingoperations/puppetproduction+26 -0
interface::manual: Fix absentingoperations/puppetproduction+27 -1
interface: Add a clsact helperoperations/puppetproduction+44 -6
interface: Allow creating IPIP interfaces w/o an endpointoperations/puppetproduction+113 -4
pybal,wmflib::service: Add ipip_encapsulation flag on lvsoperations/puppetproduction+31 -8
service: Add ipip_encapsulation field to ServiceLVSoperations/software/spicerackmaster+3 -0
wmflib::service: Add ipip_encapsulation flag on lvsoperations/puppetproduction+30 -8
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Nov 21 2023, 4:41 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-21T16:41:45Z] <vgutierrez@cumin1001> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs[1017-1019].eqiad.wmnet} and A:lvs (T351069)

Vgutierrez updated the task description. (Show Details)Nov 22 2023, 9:25 AM
gerritbot added a comment.Nov 22 2023, 9:27 AM

Change 974623 merged by Vgutierrez:

[operations/puppet@production] pybal,wmflib::service: Add ipip_encapsulation flag on lvs

https://gerrit.wikimedia.org/r/974623

gerritbot added a comment.Nov 22 2023, 9:48 AM

Change 975253 merged by Vgutierrez:

[operations/puppet@production] interface: Allow creating IPIP interfaces w/o an endpoint

https://gerrit.wikimedia.org/r/975253

gerritbot added a comment.Nov 22 2023, 9:50 AM

Change 975324 merged by Vgutierrez:

[operations/puppet@production] interface: Add a clsact helper

https://gerrit.wikimedia.org/r/975324

Stashbot added a comment.Nov 22 2023, 9:53 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-22T09:53:27Z] <vgutierrez> rolling restart of pybal to catch up on a NOOP config update - T351069

Stashbot added a comment.Nov 22 2023, 9:53 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-22T09:53:52Z] <vgutierrez@cumin1001> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs (T351069)

Stashbot added a comment.Nov 22 2023, 10:21 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-22T10:21:46Z] <vgutierrez@cumin1001> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs (T351069)

gerritbot added a comment.Nov 22 2023, 1:59 PM

Change 976737 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] lvs: Deploy ipip-multiqueue-optimizer for IPIP enabled balancers

https://gerrit.wikimedia.org/r/976737

gerritbot added a comment.Nov 23 2023, 11:31 AM

Change 977046 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] interface::manual: Fix absensting

https://gerrit.wikimedia.org/r/977046

gerritbot added a comment.Nov 23 2023, 12:17 PM

Change 977056 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] interface::ipip: Fix absenting

https://gerrit.wikimedia.org/r/977056

gerritbot added a comment.Nov 24 2023, 6:43 AM

Change 977046 merged by Vgutierrez:

[operations/puppet@production] interface::manual: Fix absenting

https://gerrit.wikimedia.org/r/977046

gerritbot added a comment.Nov 24 2023, 6:43 AM

Change 977056 merged by Vgutierrez:

[operations/puppet@production] interface::ipip: Fix absenting

https://gerrit.wikimedia.org/r/977056

Vgutierrez added a comment.Nov 24 2023, 4:22 PM

@ayounsi what would be the required TCP MSS clamping values? per https://phabricator.wikimedia.org/T348837#9256494 It seems that around ~1400 bytes for both IPv4/IPv6 should be ok?

ayounsi added a comment.Nov 27 2023, 9:16 AM

That's a great question. I don't think we have the resources to do an extensive investigation.

I see 2 options:

  1. either we only subtract the tunnel header from the default MSS to get the most data out of each packets
    • Fortunately we're not heavy inbound, so it's the size of the outbound packet that matter the most.
  2. or we decrease it further to try to fix the same issue larger providers investigated (clients with tunnels and odd settings)
    • this is slightly off topic, but maybe worth doing as we're there anyway. But the value will be an approximation (eg. based on what others are doing, round MTU or MSS value for ease of troubleshooting)
Vgutierrez added a comment.Nov 27 2023, 3:08 PM

thx @ayounsi we will go with option 1:

  • IPv4: 1500 - 20 (IP) - 20 (IP) - 20 (TCP) = 1440 bytes
  • IPv6: 1500 - 40 (IPv6) - 40 (IPv6) - 20 (TCP) = 1400 bytes
gerritbot added a comment.Nov 27 2023, 3:10 PM

Change 977696 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] lvs::realserver::ipip: Check that TCP MSS clamping is working

https://gerrit.wikimedia.org/r/977696

gerritbot added a comment.Nov 27 2023, 4:01 PM

Change 975342 merged by Vgutierrez:

[operations/puppet@production] profile: Provide a lvs::realserver::ipip profile

https://gerrit.wikimedia.org/r/975342

gerritbot added a comment.Nov 27 2023, 4:05 PM

Change 976737 merged by Vgutierrez:

[operations/puppet@production] lvs,pybal: Deploy ipip-multiqueue-optimizer for IPIP enabled balancers

https://gerrit.wikimedia.org/r/976737

gerritbot added a comment.Nov 27 2023, 4:11 PM

Change 975772 merged by Vgutierrez:

[operations/puppet@production] ncredir: Enable IPIP encapsulation on ulsfo

https://gerrit.wikimedia.org/r/975772

gerritbot added a comment.Nov 27 2023, 4:39 PM

Change 977736 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] interface::clsact: Fix unless cmd

https://gerrit.wikimedia.org/r/977736

gerritbot added a comment.Nov 27 2023, 4:42 PM

Change 977736 merged by Vgutierrez:

[operations/puppet@production] interface::clsact: Fix unless cmd

https://gerrit.wikimedia.org/r/977736

gerritbot added a comment.Nov 27 2023, 5:40 PM

Change 977746 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable IPIP on ulsfo LVS

https://gerrit.wikimedia.org/r/977746

Stashbot added a comment.Nov 27 2023, 5:52 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-27T17:52:21Z] <vgutierrez> upload ipip-multiqueue-optimizer 0.2 to apt.wm.o (bullseye) - T351069

gerritbot added a comment.Nov 27 2023, 5:54 PM

Change 977746 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable IPIP on ulsfo text|secondary LVS

https://gerrit.wikimedia.org/r/977746

gerritbot added a comment.Nov 27 2023, 5:59 PM

Change 977761 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] profile::lvs: Fix ipip-multiqueue-optimizer systemd unit

https://gerrit.wikimedia.org/r/977761

gerritbot added a comment.Nov 27 2023, 6:05 PM

Change 977761 merged by Vgutierrez:

[operations/puppet@production] profile::lvs: Fix ipip-multiqueue-optimizer systemd unit

https://gerrit.wikimedia.org/r/977761

Stashbot added a comment.Nov 27 2023, 6:07 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-27T18:07:25Z] <vgutierrez> restarting pybal on lvs4010 - T351069

gerritbot added a comment.Nov 27 2023, 6:10 PM

Change 977764 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] service: Enable IPIP encapsulation for ncredir-https too

https://gerrit.wikimedia.org/r/977764

gerritbot added a comment.Nov 27 2023, 7:46 PM

Change 977764 merged by Vgutierrez:

[operations/puppet@production] service: Enable IPIP encapsulation for ncredir-https too

https://gerrit.wikimedia.org/r/977764

Stashbot added a comment.Nov 27 2023, 7:50 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-27T19:50:17Z] <vgutierrez> restarting pybal on lvs4010 - T351069

Stashbot added a comment.Nov 27 2023, 7:53 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-27T19:53:19Z] <vgutierrez> restarting pybal on lvs4008 (effectively enabling IPIP encapsulation on ncredir@ulsfo) - T351069

gerritbot added a comment.Nov 27 2023, 8:10 PM

Change 977782 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] service: Disable IPIP encapsulation for ncredir@ulsfo

https://gerrit.wikimedia.org/r/977782

gerritbot added a comment.Nov 27 2023, 8:13 PM

Change 977782 merged by Vgutierrez:

[operations/puppet@production] service: Disable IPIP encapsulation for ncredir@ulsfo

https://gerrit.wikimedia.org/r/977782

Stashbot added a comment.Nov 27 2023, 8:16 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-27T20:16:14Z] <vgutierrez> rolling restart of pybal on lvs4010 and lvs4008 - T351069

Vgutierrez updated the task description. (Show Details)Nov 27 2023, 8:29 PM
Stashbot added a comment.Nov 28 2023, 10:09 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-28T10:09:24Z] <vgutierrez> rolling restart of pybal on lvs4010 and lvs4008, effectively enabling IPIP encapsulation on ncredir@ulsfo - T351069

Stashbot added a comment.Nov 28 2023, 10:21 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-28T10:21:19Z] <vgutierrez> rolling restart of pybal on lvs4010 and lvs4008, effectively disabling IPIP encapsulation on ncredir@ulsfo - T351069

Vgutierrez closed subtask T352143: Firewall rules prevent IPIP/IP6IP6 encapsulated traffic from reaching realservers as Resolved.Nov 28 2023, 10:40 AM
Vgutierrez updated the task description. (Show Details)
Vgutierrez closed subtask T352160: RP filtering drops requests incoming via IPIP tunnels on ncredir realservers as Resolved.Nov 29 2023, 12:24 PM
Vgutierrez updated the task description. (Show Details)
Vgutierrez updated the task description. (Show Details)
Stashbot added a comment.Nov 29 2023, 12:25 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-29T12:25:38Z] <vgutierrez> rolling restart of pybal on lvs4008 and lvs4010, effectively enabling IPIP encapsulation for ncredir@ulsfo - T351069

Vgutierrez updated the task description. (Show Details)Nov 29 2023, 12:34 PM
gerritbot added a comment.Nov 29 2023, 2:17 PM

Change 978608 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] prometheus::ops: Fix lvs_realserver_clamper config

https://gerrit.wikimedia.org/r/978608

gerritbot added a comment.Nov 29 2023, 2:33 PM

Change 978608 merged by Vgutierrez:

[operations/puppet@production] prometheus::ops: Fix lvs_realserver_clamper config

https://gerrit.wikimedia.org/r/978608

gerritbot added a comment.Nov 29 2023, 3:52 PM

Change 978624 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] ncredir: Enable IPIP encapsulation on codfw

https://gerrit.wikimedia.org/r/978624

gerritbot added a comment.Nov 29 2023, 3:56 PM

Change 978625 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable IPIP on codfw text|secondary LVS

https://gerrit.wikimedia.org/r/978625

gerritbot added a comment.Nov 30 2023, 9:31 AM

Change 978624 merged by Vgutierrez:

[operations/puppet@production] ncredir: Enable IPIP encapsulation on codfw

https://gerrit.wikimedia.org/r/978624

gerritbot added a comment.Nov 30 2023, 9:52 AM

Change 978625 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable IPIP on codfw text|secondary LVS

https://gerrit.wikimedia.org/r/978625

Stashbot added a comment.Nov 30 2023, 9:59 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-30T09:59:19Z] <vgutierrez> rolling restart of pybal on lvs2011 and lvs2014, effectively enabling IPIP encapsulation on ncredir@codfw - T351069

gerritbot added a comment.Dec 1 2023, 9:04 AM

Change 979297 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] prometheus::sysctl: Support configurable sysctls

https://gerrit.wikimedia.org/r/979297

gerritbot added a comment.Dec 4 2023, 10:42 AM

Change 979903 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] lvs::realserver::ipip: Clamp on lo too

https://gerrit.wikimedia.org/r/979903

gerritbot added a comment.Dec 4 2023, 11:40 AM

Change 979903 merged by Vgutierrez:

[operations/puppet@production] lvs::realserver::ipip: Clamp on lo too

https://gerrit.wikimedia.org/r/979903

gerritbot added a comment.Dec 4 2023, 3:16 PM

Change 977696 merged by Vgutierrez:

[operations/puppet@production] lvs::realserver::ipip: Check that TCP MSS clamping is working

https://gerrit.wikimedia.org/r/977696

gerritbot added a comment.Dec 4 2023, 3:35 PM

Change 979984 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Disable rp_filter on ncredir@eqsin

https://gerrit.wikimedia.org/r/979984

gerritbot added a comment.Dec 4 2023, 3:35 PM

Change 979985 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable IPIP encapsulation on ncredir@eqsin

https://gerrit.wikimedia.org/r/979985

gerritbot added a comment.Dec 4 2023, 3:50 PM

Change 979297 merged by Vgutierrez:

[operations/puppet@production] prometheus::sysctl: Support configurable sysctls

https://gerrit.wikimedia.org/r/979297

gerritbot added a comment.Dec 4 2023, 4:05 PM

Change 979984 merged by Vgutierrez:

[operations/puppet@production] hiera: Disable rp_filter on ncredir@eqsin

https://gerrit.wikimedia.org/r/979984

gerritbot added a comment.Dec 4 2023, 4:19 PM

Change 979994 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable IPIP on eqsin text|secondary LVS

https://gerrit.wikimedia.org/r/979994

Vgutierrez updated the task description. (Show Details)Dec 4 2023, 4:20 PM
gerritbot added a comment.Dec 5 2023, 6:45 AM

Change 979985 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable IPIP encapsulation on ncredir@eqsin

https://gerrit.wikimedia.org/r/979985

gerritbot added a comment.Dec 5 2023, 6:52 AM

Change 979994 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable IPIP on eqsin text|secondary LVS

https://gerrit.wikimedia.org/r/979994

Stashbot added a comment.Dec 5 2023, 6:55 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-05T06:55:11Z] <vgutierrez> rolling restart of text|secondary LVS on eqsin effectively enabling IPIP encapsulation for ncredir@eqsin - T351069

Vgutierrez updated the task description. (Show Details)Dec 5 2023, 6:58 AM
gerritbot added a comment.Dec 5 2023, 7:25 AM

Change 980272 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Disable rp_filter for ncredir@drmrs

https://gerrit.wikimedia.org/r/980272

gerritbot added a comment.Dec 5 2023, 7:25 AM

Change 980273 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable IPIP encapsulation on ncredir@drmrs

https://gerrit.wikimedia.org/r/980273

gerritbot added a comment.Dec 5 2023, 7:25 AM

Change 980274 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable IPIP on text|secondary LVS in drmrs

https://gerrit.wikimedia.org/r/980274

gerritbot added a comment.Dec 5 2023, 8:04 AM

Change 980280 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/alerts@master] traffic: Alert on configured and observed MSS mismatch

https://gerrit.wikimedia.org/r/980280

gerritbot added a comment.Dec 5 2023, 11:07 AM

Change 980272 merged by Vgutierrez:

[operations/puppet@production] hiera: Disable rp_filter for ncredir@drmrs

https://gerrit.wikimedia.org/r/980272

gerritbot added a comment.Dec 5 2023, 3:12 PM

Change 980273 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable IPIP encapsulation on ncredir@drmrs

https://gerrit.wikimedia.org/r/980273

gerritbot added a comment.Dec 5 2023, 5:42 PM

Change 980274 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable IPIP on text|secondary LVS in drmrs

https://gerrit.wikimedia.org/r/980274

Stashbot added a comment.Dec 5 2023, 5:46 PM

Mentioned in SAL (#wikimedia-operations) [2023-12-05T17:45:59Z] <vgutierrez> rolling restart of text|secondary LVS on drmrs effectively enabling IPIP encapsulation for ncredir@drmrs- T351069

Vgutierrez updated the task description. (Show Details)Dec 5 2023, 5:49 PM
gerritbot added a comment.Dec 11 2023, 9:33 AM

Change 981955 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Disable rp_filter for ncredir@esams

https://gerrit.wikimedia.org/r/981955

gerritbot added a comment.Dec 11 2023, 9:54 AM

Change 982038 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable IPIP encapsulation on ncredir@esams

https://gerrit.wikimedia.org/r/982038

gerritbot added a comment.Dec 11 2023, 10:02 AM

Change 982040 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable IPIP encapsulation on text|secondary LVS in esams

https://gerrit.wikimedia.org/r/982040

gerritbot added a comment.Dec 11 2023, 10:13 AM

Change 981955 merged by Vgutierrez:

[operations/puppet@production] hiera: Disable rp_filter for ncredir@esams

https://gerrit.wikimedia.org/r/981955

gerritbot added a comment.Dec 11 2023, 10:48 AM

Change 982038 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable IPIP encapsulation on ncredir@esams

https://gerrit.wikimedia.org/r/982038

gerritbot added a comment.Dec 11 2023, 11:16 AM

Change 982040 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable IPIP encapsulation on text|secondary LVS in esams

https://gerrit.wikimedia.org/r/982040

Stashbot added a comment.Dec 11 2023, 11:20 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-11T11:20:03Z] <vgutierrez> rolling restart of pybal on lvs3010 and lvs3008 effectively enabling IPIP encapsulation on ncredir@esams - T351069

gerritbot added a comment.Dec 11 2023, 12:26 PM

Change 982063 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Disable rp_filter on ncredir@eqiad

https://gerrit.wikimedia.org/r/982063

gerritbot added a comment.Dec 11 2023, 12:28 PM

Change 982070 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable IPIP encapsulation on ncredir@eqiad

https://gerrit.wikimedia.org/r/982070

gerritbot added a comment.Dec 11 2023, 2:14 PM

Change 982096 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Enable IPIP encapsulation on text|secondary LVS in eqiad

https://gerrit.wikimedia.org/r/982096

gerritbot added a comment.Dec 11 2023, 3:18 PM

Change 982063 merged by Vgutierrez:

[operations/puppet@production] hiera: Disable rp_filter on ncredir@eqiad

https://gerrit.wikimedia.org/r/982063

gerritbot added a comment.Dec 11 2023, 3:46 PM

Change 982070 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable IPIP encapsulation on ncredir@eqiad

https://gerrit.wikimedia.org/r/982070

gerritbot added a comment.Dec 11 2023, 4:05 PM

Change 982096 merged by Vgutierrez:

[operations/puppet@production] hiera: Enable IPIP encapsulation on text|secondary LVS in eqiad

https://gerrit.wikimedia.org/r/982096

Stashbot added a comment.Dec 11 2023, 4:13 PM

Mentioned in SAL (#wikimedia-operations) [2023-12-11T16:13:21Z] <vgutierrez> rolling restart of pybal on lvs1020 and lvs1017 effectively enabling IPIP encapsulation on ncredir@eqiad - T351069

Vgutierrez closed this task as Resolved.Dec 11 2023, 4:15 PM
Vgutierrez updated the task description. (Show Details)
gerritbot added a comment.Dec 11 2023, 4:19 PM

Change 982124 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Unify ncredir IPIP encapsulation settings

https://gerrit.wikimedia.org/r/982124

gerritbot added a comment.Dec 11 2023, 4:29 PM

Change 982124 merged by Vgutierrez:

[operations/puppet@production] hiera: Unify ncredir IPIP encapsulation settings

https://gerrit.wikimedia.org/r/982124

gerritbot added a comment.Dec 13 2023, 2:01 PM

Change 980280 merged by Vgutierrez:

[operations/alerts@master] traffic: Alert on configured and observed MSS mismatch

https://gerrit.wikimedia.org/r/980280

gerritbot added a comment.Dec 13 2023, 2:49 PM

Change 982808 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/alerts@master] traffic: Provide a dashboard link for LVSRealServerMSS

https://gerrit.wikimedia.org/r/982808

gerritbot added a comment.Dec 18 2023, 5:42 PM

Change 982808 merged by Vgutierrez:

[operations/alerts@master] traffic: Provide a dashboard link for LVSRealServerMSS

https://gerrit.wikimedia.org/r/982808

gerritbot added a comment.Jan 19 2024, 2:51 PM

Change 991785 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] P:lvs: set monitoring enabled for IPIP-related services

https://gerrit.wikimedia.org/r/991785

gerritbot added a comment.Jan 23 2024, 4:00 PM

Change 991785 merged by Ssingh:

[operations/puppet@production] P:lvs: set monitoring enabled for IPIP-related services

https://gerrit.wikimedia.org/r/991785

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL