Change Details

**Background** While investigating poor performance for backup traffic between eqiad and codfw, it was discovered packets were being dropped by asw devices in eqiad (see T274234). Average usage is well within link capacity, but it is likely we are seeing microbursts as described here: https://kb.juniper.net/InfoCenter/index?page=content&id=KB36095 **Mitigation - Buffer Allocation** By default the switches reserve 50% of buffer memory for a "lossless" traffic class we don't use. It seems we can re-partition the space to dedicate the majority of it to best-effort instead. https://www.juniper.net/documentation/us/en/software/junos/traffic-mgmt-qfx/topics/example/cos-shared-buffer-allocation-lossy-ucast-qfx-series-configuring.html This change should be rolled out to all EX/QFX switches across the network. Currently scheduling as follows: |Row|Date|Time|Task| |-----|------|-----|------| |A|Tues July 20th 2021|17:00 CET|TBC| |B|Thurs July 22nd 2021|17:00 CET|TBC| |C|Tues July 27th 2021|17:00 CET|TBC| |D|Thurs July 29th 2021|17:00 CET|TBC|

**Background** While investigating poor performance for backup traffic between eqiad and codfw, it was discovered packets were being dropped by asw devices in eqiad (see T274234). Average usage is well within link capacity, but it is likely we are seeing microbursts as described here: https://kb.juniper.net/InfoCenter/index?page=content&id=KB36095 **Mitigation - Buffer Allocation** By default the switches reserve 50% of buffer memory for a "lossless" traffic class we don't use. It seems we can re-partition the space to dedicate the majority of it to best-effort instead. https://www.juniper.net/documentation/us/en/software/junos/traffic-mgmt-qfx/topics/example/cos-shared-buffer-allocation-lossy-ucast-qfx-series-configuring.html This change should be rolled out to all EX/QFX switches across the network. Currently scheduling as follows: |Row|Date|Time|Task| |-----|------|-----|------| |A|Tues July 20th 2021|15:00 UTC (08:00 PDT / 11:00 EDT / 17:00 CEST)|TBC| |B|Thurs July 22nd 2021|15:00 UTC (08:00 PDT / 11:00 EDT / 17:00 CEST)|TBC| |C|Tues July 27th 2021|15:00 UTC (08:00 PDT / 11:00 EDT / 17:00 CEST)|TBC| |D|Thurs July 29th 2021|15:00 UTC (08:00 PDT / 11:00 EDT / 17:00 CEST)|TBC|

**Background** While investigating poor performance for backup traffic between eqiad and codfw, it was discovered packets were being dropped by asw devices in eqiad (see T274234). Average usage is well within link capacity, but it is likely we are seeing microbursts as described here: https://kb.juniper.net/InfoCenter/index?page=content&id=KB36095 **Mitigation - Buffer Allocation** By default the switches reserve 50% of buffer memory for a "lossless" traffic class we don't use. It seems we can re-partition the space to dedicate the majority of it to best-effort instead. https://www.juniper.net/documentation/us/en/software/junos/traffic-mgmt-qfx/topics/example/cos-shared-buffer-allocation-lossy-ucast-qfx-series-configuring.html This change should be rolled out to all EX/QFX switches across the network. Currently scheduling as follows: |Row|Date|Time|Task| |-----|------|-----|------| |A|Tues July 20th 2021|15:00 UTC (08:00 PDT / 11:00 EDT / 17:00 CETST)|TBC| |B|Thurs July 22nd 2021|15:00 UTC (08:00 PDT / 11:00 EDT / 17:00 CETST)|TBC| |C|Tues July 27th 2021|15:00 UTC (08:00 PDT / 11:00 EDT / 17:00 CETST)|TBC| |D|Thurs July 29th 2021|15:00 UTC (08:00 PDT / 11:00 EDT / 17:00 CETST)|TBC|