Page MenuHomePhabricator
Create Task
Maniphest T367511

Migrate Cloudelastic load balancing to IPIP encapsulation (LVS)
Closed, ResolvedPublic
Actions

Description

Migration steps:

  • depool cloudelastic (but keep monitoring its cluster health)
  • stop puppet on cloudelastic, lvs1020 and lvs1018
  • apply puppet on cloudelastic (one server at a time, watching cluster health checks) and validate that IPIP is working as expected [we use https://gitlab.wikimedia.org/-/snippets/107 for that]
  • apply puppet on lvs1020 and lvs1018
  • rolling restart of pybal on the impacted LVS servers (via sre.loadbalancer.restart-pybal cookbook ?)
  • validate from the bastion hosts that traffic towards the VIP works as expected
  • repool cloudelastic

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
cloudelastic: enable IPIP for LVSoperations/puppetproduction+19 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

bking created this task.Jun 14 2024, 1:20 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptJun 14 2024, 1:20 PM
Vgutierrez added a parent task: T367312: Migrate services behind high-traffic2 LVS to IPIP encapsulation.Jun 17 2024, 9:46 AM
Vgutierrez mentioned this in T367312: Migrate services behind high-traffic2 LVS to IPIP encapsulation.
Gehel triaged this task as Medium priority.Jun 18 2024, 8:46 AM
Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.
Gehel moved this task from Scratch to Infrastructure on the Data-Platform-SRE board.
gerritbot added a comment.Jun 18 2024, 9:03 AM

Change #1043302 had a related patch set uploaded (by Gehel; author: Bking):

[operations/puppet@production] cloudelastic: enable IPIP for LVS

https://gerrit.wikimedia.org/r/1043302

gerritbot added a project: Patch-For-Review.Jun 18 2024, 9:03 AM
Gehel mentioned this in T365616: Consider migrating Search Platform-owned clusters to IPIP encapsulation (LVS).Jun 18 2024, 9:04 AM
bking updated the task description. (Show Details)Jun 18 2024, 9:19 PM
bking updated the task description. (Show Details)Jun 18 2024, 9:23 PM
bking updated the task description. (Show Details)
ops-monitoring-bot added a comment.Jun 20 2024, 2:17 PM

Icinga downtime and Alertmanager silence (ID=81308338-4f99-4220-b27b-84a614289b52) set by bking@cumin2002 for 2:00:00 on 6 host(s) and their services with reason: IPIP migration

cloudelastic[1005-1010].eqiad.wmnet
gerritbot added a comment.Jun 20 2024, 2:21 PM

Change #1043302 merged by Bking:

[operations/puppet@production] cloudelastic: enable IPIP for LVS

https://gerrit.wikimedia.org/r/1043302

Stashbot added a comment.Jun 20 2024, 2:47 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-20T14:47:44Z] <vgutierrez> rolling restart of pybal on lvs1020 and lvs1018 - T367511

bking added a comment.Jun 20 2024, 3:08 PM

I'm happy to report the migration completed successfully. As such, I'm closing out this ticket.

However, if you are noticing issues post-migration, please do re-open.

bking closed this task as Resolved.Jun 20 2024, 3:09 PM
bking claimed this task.
bking edited projects, added Data-Platform-SRE (2024.06.17 - 2024.07.07); removed Data-Platform-SRE.
bking moved this task from Backlog to Done on the Data-Platform-SRE (2024.06.17 - 2024.07.07) board.
Don-vip subscribed.Jun 22 2024, 12:22 PM

Did this change fix T365154? video2commons became live again at the general surprise, around the time this change was applied.

Gehel subscribed.Jun 28 2024, 7:48 AM
In T367511#9915026, @Don-vip wrote:

Did this change fix T365154? video2commons became live again at the general surprise, around the time this change was applied.

I doubt very much this is related. I can't imagine a link between video2commons and cloudelastic (but I might lack imagination) and this change did not change anything functionally, it just changed the very low level way in which the traffic is flowing into cloudelastic.

Don-vip added a comment.Jun 28 2024, 12:12 PM

Yes, taavi pointed us towards the change that repaired video2commons: https://phabricator.wikimedia.org/T365154#9915037

bking mentioned this in T394062: Migrate CirrusSearch LVS pools to IPIP.May 13 2025, 3:51 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits