Change 701499 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Adding 'quality-of-service' template for use on QFX/EX series switches.

https://gerrit.wikimedia.org/r/701499

What is the expected length of service interupption for any of these days? I'm looking on the impact on the dumpsdata/snapshot hosts, and depending on the legth of time of the outage, we might be able to get by with some minor shuffling of hosts around.

I’ve no reason to think anything other than what’s in the child tasks at this point. Having put out feelers externally, I’ve got some anecdotal reports that a few seconds is correct to expect.

The exact duration of the impact is unknown at this time - we hope to be able to test on some real switches before the date and get a firm indication. Best estimate is it will be in the order of seconds, certainly no longer than a minute, but we should plan for up to a 5-minute interruption, and be aware as always that there is a small potential something will go wrong and cause a longer disturbance.

In T284592#7198669, @cmooney wrote:

I’ve no reason to think anything other than what’s in the child tasks at this point. Having put out feelers externally, I’ve got some anecdotal reports that a few seconds is correct to expect.

The exact duration of the impact is unknown at this time - we hope to be able to test on some real switches before the date and get a firm indication. Best estimate is it will be in the order of seconds, certainly no longer than a minute, but we should plan for up to a 5-minute interruption, and be aware as always that there is a small potential something will go wrong and cause a longer disturbance.

This is quite good, even taking into account the risk that something goes awry. Thanks!

@cmooney I haven't been able to get ahold of you this week, so leaving the comment I left on IRC here:
My preferred order for the switches maintenance would be: row d, c, b a (again, this is what would work best for dbas, as it would give us more time to work on row a and row b replacements)

@Marostegui Ok thanks for the comments. I've not been feeling so good so hadn't been online.

Will review Monday against feedback from other teams but I'm sure we can accommodate. Also please advise if you expect timelines to be workable, or if they are a little tight we can look at pushing out so everyone has time to prepare.

I hope you get better soon. I am off next week but someone from the team will contact you next week.
From my point of view, I think if we follow that row order, we should be ok with the given dates.

With the new schedule I think I can swap one dumpsdata host and one snapshot host and avoid any impact whatsoever on XMl/SQL dumps. This is great, thank you!

@cmooney should we sent out an email about this to ops@ and possibly add those times/dates to the maintenance calendar? Thank you!

@jijiki thanks yes good suggestions both. I will send a mail to ops@ later today as a reminder for people to review.

In terms of maintenance do you mean the "Ops vendor maintenance" one? Or something else? I tried to add it to that one but I don't think I have permissions, I can probably sort that I'm sure though.

Change 701499 merged by jenkins-bot:

[operations/homer/public@master] Adding 'quality-of-service' template for use on QFX/EX series switches.

https://gerrit.wikimedia.org/r/701499

Change 705722 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Previous change had incorrect file extention on the 'class-of-service' config template. Correcting here.

https://gerrit.wikimedia.org/r/705722

Change 705722 merged by jenkins-bot:

[operations/homer/public@master] Previous change had incorrect file extention on the 'class-of-service' config template. Correcting here.

https://gerrit.wikimedia.org/r/705722

Change 706568 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Adding flag for asw2-c-eqiad to configure class-of-service shared buffer config. This will keep it in line with config added manually in T286065.

https://gerrit.wikimedia.org/r/706568

Change 706568 merged by jenkins-bot:

[operations/homer/public@master] Adding flag for asw2-c-eqiad to configure class-of-service shared buffer config. This will keep it in line with config added manually in T286065.

https://gerrit.wikimedia.org/r/706568

Change 708784 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Adding flag for asw2-a-eqiad and asw2-b-eqiad to configure class-of-service shared buffer config. This will keep it in line with config added manually under T286061 and T286032

https://gerrit.wikimedia.org/r/708784

Change 708784 merged by jenkins-bot:

[operations/homer/public@master] Adding flag for asw2-a-eqiad and asw2-b-eqiad to configure class-of-service shared buffer config. This will keep it in line with config added manually under T286061 and T286032

https://gerrit.wikimedia.org/r/708784

Mentioned in SAL (#wikimedia-operations) [2021-07-30T08:56:08Z] <topranks> running homer against asw2-a-eqiad and asw2-b-eqiad to bring homer in line with manual config added for buffer mem. T284592

Ok well we're about a week after DC switchover back to eqiad so we can make some conclusions on the results of the changes in eqiad.

Overall there definitely seem to be less discards on the relevant links in the past week, compared to earlier in the year. In some cases the graphs are somewhat skewed by unrelated bursts of activity, but in general the pattern is fairly clear (comparing the last week with stats prior to July):

asw2-a-eqiad:

asw2-b-eqiad:

asw2-c-eqiad:

asw2-d-eqiad:

The original issue that made us look at this was low throughput on the regular transfers from eqiad backup hosts to those in codfw. Checking the most recent graphs it seems speeds for these have not degraded back to the levels observed in May/June. This is despite link usage in eqiad returning to normal following last weeks DC switch back, and the resulting increase in discards on the switch->cr links (albeit less than before as shown above):

We could leave it further time to allow more time for the new pattern to emerge, but broadly I think we can say:

• The change has definitely reduced the number of discards we observe day-to-day on these links.
• The reduction has improved real-world performance as seen for the backup hosts.
• There is still a significant number of drops, which will unquestionably be causing degraded performance for services.

I am creating a parent task for this, so we can track the overall issue of these drops, and any other actions we may wish to take to address the continuing problem.

Change 929689 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Remove optional var to set COS buffers for QFX/EX switches

https://gerrit.wikimedia.org/r/929689

Mentioned in SAL (#wikimedia-operations) [2023-06-14T13:42:47Z] <topranks> adjusting port buffer partition asw-a-codfw T284592

Mentioned in SAL (#wikimedia-operations) [2023-06-14T13:46:01Z] <topranks> adjusting port buffer partition asw-b-codfw T284592

Mentioned in SAL (#wikimedia-operations) [2023-06-14T13:49:33Z] <topranks> adjusting port buffer partition asw-c-codfw T284592

Mentioned in SAL (#wikimedia-operations) [2023-06-14T13:53:11Z] <topranks> adjusting port buffer partition asw-d-codfw T284592

Mentioned in SAL (#wikimedia-operations) [2023-06-14T13:57:24Z] <topranks> adjusting port buffer partition asw2-ulsfo T284592

Mentioned in SAL (#wikimedia-operations) [2023-06-14T13:58:24Z] <topranks> adjusting port buffer partition asw1-eqsin T284592

Mentioned in SAL (#wikimedia-operations) [2023-06-14T13:59:38Z] <topranks> adjusting port buffer partition asw2-esams T284592

Change 929689 merged by jenkins-bot:

[operations/homer/public@master] Remove optional var to set COS buffers for QFX/EX switches

https://gerrit.wikimedia.org/r/929689

Change 930754 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Do not push class-of-service buffer partition to ex4300

https://gerrit.wikimedia.org/r/930754

Change 930754 merged by jenkins-bot:

[operations/homer/public@master] Do not push class-of-service buffer partition to ex4300

https://gerrit.wikimedia.org/r/930754