Page MenuHomePhabricator
Create Task
Maniphest T330075

[cloudvirt] Enable jumbo frames on cloud-hosts/cloud-private interfaces
Closed, ResolvedPublic
Actions

Description

We currently have our ceph osd hosts setup to use jumbo frames, but the cloudvirts are configured to use 1500 MTU.

This is not optimal as jumbo frames will allow for higher throughput to the VM storage layer (ceph).

All the switches (cloudsw*) are configured with jumbo frames too.

This task is to enable the jumbo frames on the cloudvirts making sure nothing breaks in-between.

Related: T329711: [ceph] Add monitoring for inter-osd/mon/cloudvirt connectivity, T329778: [ceph] Investigate if there's a way to degrade instead of failing when jumbo frames are being dropped in the network

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
P:wmcs::cloud_private_subnet: Cleanup feature flag for jumbo framesoperations/puppetproduction+12 -41
hieradata: Enable jumbo frames on remaining eqiad1 nodesoperations/puppetproduction+8 -0
hieradata: Enable jumbo frames on eqiad1 cloudnetsoperations/puppetproduction+1 -0
hieradata: Enable jumbo frames on all eqiad1 cloudvirtsoperations/puppetproduction+6 -1
hieradata: Enable jumbo frames on cloudvirt1062operations/puppetproduction+1 -0
hieradata: Enable jumbo frames on remaining codfw1dev nodesoperations/puppetproduction+7 -0
hieradata: Enable jumbo frames on codfw1dev cloudnetsoperations/puppetproduction+1 -0
hieradata: Enable jumbo frames on all codwf1dev cloudvirtsoperations/puppetproduction+2 -1
P:wmcs::cloud_private_subnet: Support enabling jumbo framesoperations/puppetproduction+19 -10
Customize query in gerrit

Related Objects
Search...

Event Timeline

dcaro created this task.Feb 20 2023, 12:51 PM
dcaro triaged this task as High priority.
• aborrero subscribed.Feb 20 2023, 1:30 PM

Questions for NetOps: they live in the cloud-hosts vlan. It is OK if some hosts attached to that VLAN use high MTU and other don't?

dcaro mentioned this in T329711: [ceph] Add monitoring for inter-osd/mon/cloudvirt connectivity.Feb 20 2023, 2:01 PM
• aborrero added a project: netops.Feb 21 2023, 5:14 PM
• aborrero added subscribers: ayounsi, cmooney.
Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptFeb 21 2023, 5:14 PM
ayounsi added a comment.Feb 22 2023, 11:13 AM
In T330075#8629532, @aborrero wrote:

Questions for NetOps: they live in the cloud-hosts vlan. It is OK if some hosts attached to that VLAN use high MTU and other don't?

TCP handshake contains MSS data of each side so they will both send packets at the max size they can send and max size the other side can receive.
And they can be asymmetric, host A can send packets at 9000 to host B, and host B send 1500 packets to host A.

UDP by default is also < 1500.

So for UDP it comes down to the application, and TCP should do the right thing.

• aborrero added a comment.Feb 22 2023, 11:18 AM

Ok, thanks!

Next question would be:

Because T319184: Move WMCS servers to 1 single NIC some cloudvirts have a single NIC trunked with 1 native + 1 tagged VLAN. I wonder if there is any problem if each have a different MTU set on either the linux side or the switch port side.

ayounsi added a comment.Feb 22 2023, 11:25 AM

Switch side is already at 9192.

Server TCP stack will define the MSS based on the interface used to reach the other endpoint (so based on the routing table).

So here too it should just work.

fnegri subscribed.Feb 22 2023, 2:52 PM
ayounsi moved this task from Backlog to Watching on the netops board.Mar 16 2023, 9:11 PM
• nskaggs removed a project: cloud-services-team (FY2022/2023-Q3).Apr 28 2023, 1:53 PM
• joanna_borun removed projects: Infrastructure-Foundations, netops.Feb 12 2024, 3:51 PM
dcaro removed dcaro as the assignee of this task.May 22 2025, 7:41 AM
cmooney added a comment.May 22 2025, 9:15 AM
In T330075#8629532, @aborrero wrote:

Questions for NetOps: they live in the cloud-hosts vlan. It is OK if some hosts attached to that VLAN use high MTU and other don't?

I would say thread very carefully here and fully test any implementation in codfw before going live. As Arzhel says for TCP the MSS negotiation should largely keep things working ok, but MTU problems can be tricky.

taavi added a project: Cloud-VPS.Nov 5 2025, 2:33 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptNov 5 2025, 2:33 PM
Andrew subscribed.Nov 5 2025, 2:33 PM
fgiunchedi mentioned this in T408543: MTU setting in IPv6 VMs causes issues with Docker.Nov 5 2025, 2:34 PM
taavi renamed this task from [cloudvirt] Move to jumbo frames to [cloudvirt] Enable jumbo frames on cloud-hosts/cloud-private interfaces.Nov 7 2025, 12:06 PM
taavi claimed this task.
taavi mentioned this in T273596: Investigate and enable jumbo frames in cloudvirt nodes.
taavi merged a task: T273596: Investigate and enable jumbo frames in cloudvirt nodes.
taavi added subscribers: taavi, RhinosF1, Aklapper.
taavi mentioned this in T409544: Tell Neutron the physical network has jumbo frames enabled.Nov 7 2025, 12:08 PM
taavi added a parent task: T409544: Tell Neutron the physical network has jumbo frames enabled.
gerritbot added a comment.Nov 7 2025, 12:20 PM

Change #1203003 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:wmcs::cloud_private_subnet: Support enabling jumbo frames

https://gerrit.wikimedia.org/r/1203003

gerritbot added a project: Patch-For-Review.Nov 7 2025, 12:20 PM
gerritbot added a comment.Nov 7 2025, 12:58 PM

Change #1203003 merged by Majavah:

[operations/puppet@production] P:wmcs::cloud_private_subnet: Support enabling jumbo frames

https://gerrit.wikimedia.org/r/1203003

Maintenance_bot removed a project: Patch-For-Review.Nov 7 2025, 1:31 PM
gerritbot added a comment.Nov 10 2025, 10:04 AM

Change #1203398 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Enable jumbo frames on all codwf1dev cloudvirts

https://gerrit.wikimedia.org/r/1203398

gerritbot added a project: Patch-For-Review.Nov 10 2025, 10:04 AM

Change #1203399 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Enable jumbo frames on codfw1dev cloudnets

https://gerrit.wikimedia.org/r/1203399

gerritbot added a comment.Nov 10 2025, 10:04 AM

Change #1203400 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Enable jumbo frames on remaining codfw1dev nodes

https://gerrit.wikimedia.org/r/1203400

gerritbot added a comment.Nov 10 2025, 10:08 AM

Change #1203398 merged by Majavah:

[operations/puppet@production] hieradata: Enable jumbo frames on all codwf1dev cloudvirts

https://gerrit.wikimedia.org/r/1203398

gerritbot added a comment.Nov 10 2025, 1:16 PM

Change #1203399 merged by Majavah:

[operations/puppet@production] hieradata: Enable jumbo frames on codfw1dev cloudnets

https://gerrit.wikimedia.org/r/1203399

gerritbot added a comment.Nov 10 2025, 1:27 PM

Change #1203400 merged by Majavah:

[operations/puppet@production] hieradata: Enable jumbo frames on remaining codfw1dev nodes

https://gerrit.wikimedia.org/r/1203400

Maintenance_bot removed a project: Patch-For-Review.Nov 10 2025, 1:30 PM
gerritbot added a comment.Nov 11 2025, 8:55 AM

Change #1203760 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Enable jumbo frames on cloudvirt1062

https://gerrit.wikimedia.org/r/1203760

gerritbot added a project: Patch-For-Review.Nov 11 2025, 8:55 AM
gerritbot added a comment.Nov 11 2025, 8:59 AM

Change #1203760 merged by Majavah:

[operations/puppet@production] hieradata: Enable jumbo frames on cloudvirt1062

https://gerrit.wikimedia.org/r/1203760

Maintenance_bot removed a project: Patch-For-Review.Nov 11 2025, 9:31 AM
gerritbot added a comment.Nov 12 2025, 4:12 PM

Change #1204623 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Enable jumbo frames on all eqiad1 cloudvirts

https://gerrit.wikimedia.org/r/1204623

gerritbot added a project: Patch-For-Review.Nov 12 2025, 4:13 PM

Change #1204624 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Enable jumbo frames on eqiad1 cloudnets

https://gerrit.wikimedia.org/r/1204624

gerritbot added a comment.Nov 12 2025, 4:13 PM

Change #1204625 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] hieradata: Enable jumbo frames on remaining eqiad1 nodes

https://gerrit.wikimedia.org/r/1204625

gerritbot added a comment.Nov 12 2025, 4:13 PM

Change #1204626 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:wmcs::cloud_private_subnet: Cleanup feature flag for jumbo frames

https://gerrit.wikimedia.org/r/1204626

gerritbot added a comment.Nov 13 2025, 10:03 AM

Change #1204623 merged by Majavah:

[operations/puppet@production] hieradata: Enable jumbo frames on all eqiad1 cloudvirts

https://gerrit.wikimedia.org/r/1204623

gerritbot added a comment.Nov 13 2025, 10:31 AM

Change #1204624 merged by Majavah:

[operations/puppet@production] hieradata: Enable jumbo frames on eqiad1 cloudnets

https://gerrit.wikimedia.org/r/1204624

gerritbot added a comment.Nov 13 2025, 10:37 AM

Change #1204625 merged by Majavah:

[operations/puppet@production] hieradata: Enable jumbo frames on remaining eqiad1 nodes

https://gerrit.wikimedia.org/r/1204625

gerritbot added a comment.Nov 13 2025, 10:44 AM

Change #1204626 merged by Majavah:

[operations/puppet@production] P:wmcs::cloud_private_subnet: Cleanup feature flag for jumbo frames

https://gerrit.wikimedia.org/r/1204626

taavi closed this task as Resolved.Nov 13 2025, 10:45 AM
Maintenance_bot removed a project: Patch-For-Review.Nov 13 2025, 11:30 AM
Andrew awarded a token.Nov 13 2025, 3:06 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits