Page MenuHomePhabricator
Create Task
Maniphest T221904

swift backend decomms / rebalances are noisy
Closed, ResolvedPublic
Actions

Description

It's very common, when Swift backend machines are doing lots of rebalancing after their weights have been changed, to get a ton of spammy alerts:

17:58:08	<+icinga-wm>	PROBLEM - swift-container-server on ms-be2024 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.192.48.60: Connection reset by peer https://wikitech.wikimedia.org/wiki/Swift
17:58:34	<+icinga-wm>	PROBLEM - MD RAID on ms-be2019 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.192.16.161: Connection reset by peer
17:58:34	<+icinga-wm>	PROBLEM - puppet last run on ms-be2019 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.192.16.161: Connection reset by peer

While the operation of Swift itself doesn't seem to be obviously adversely affected by all the I/O of rebalancing, certainly Icinga NRPEs are sensitive to it.

This is a task to verify the truth of the former part of the above statement, and to minimize the impact of rebalancing upon monitoring

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
swift: limit rsync and swift-object-replicator memory to 5% in eqiadoperations/puppetproduction+1 -1
hieradata: add defaults for profile::swift::storageoperations/puppetproduction+5 -0
swift: limit rsync and swift-object-replicator memory to 5% in codfwoperations/puppetproduction+43 -31
role: default swift rsync memory limitoperations/puppetproduction+2 -0
swift: limit rsync to 10% memory in codfwoperations/puppetproduction+5 -1
swift: limit rsync service memoryoperations/puppetproduction+14 -0
Customize query in gerrit

Related Objects

Event Timeline

CDanis created this task.Apr 25 2019, 10:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 25 2019, 10:05 PM
• herron triaged this task as Medium priority.Apr 26 2019, 10:44 PM
• herron added a project: observability.
CDanis claimed this task.May 7 2019, 12:59 PM
Stashbot added a comment.May 7 2019, 1:02 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-07T13:02:45Z] <cdanis> T221904 cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin -p95 -m async -b5 'ms-be2*' 'run-puppet-agent -q' 'systemctl restart swift-object-replicator' 'systemctl restart swift-object-auditor'

CDanis added a comment.EditedMay 7 2019, 1:05 PM

Trying out a few things here:

  • ionice'ing swift-object-replicator lower than everything else, except
  • ionice'ing swift-object-auditor even lower than that
  • pick a handful of hosts in codfw: all hosts ending in a 4 or 7 -- 8 hosts or about 20% of the cluster. ms-be2*[4,7].codfw.wmnet
  • and change their scheduler algorithm on sd? to cfq (which respects ionice)
  • start off a new round of replication traffic from the to-be-decommed hosts (c.f. T221068 )
  • afterwards, do some combing through logs to see how swift-object-server performance and also monitoring noise was affected between control/experiment
Stashbot added a comment.May 7 2019, 1:17 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-07T13:17:17Z] <cdanis> T221904 cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin -p95 -m async -b5 'ms-be1*' 'run-puppet-agent -q' 'systemctl restart swift-object-replicator' 'systemctl restart swift-object-auditor'

Stashbot added a comment.May 8 2019, 7:01 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-08T19:01:04Z] <cdanis> T221904 cdanis@cumin1001.eqiad.wmnet ~ % sudo cumin 'ms-be2*[4,7].codfw.wmnet' 'for DISK in /sys/block/sd*/queue/scheduler ; do echo cfq > $DISK ; done'

fgiunchedi subscribed.Aug 19 2019, 1:11 PM

FWIW this should be slightly less noisy since individual swift daemons won't produce alerts anymore as per https://gerrit.wikimedia.org/r/c/operations/puppet/+/530080

fgiunchedi added a comment.Dec 2 2019, 12:20 PM

AFAICS through the latest rebalances we haven't observed any alerts, possibly also due to using multiple servers per port (T222366)

CDanis closed this task as Resolved.May 7 2020, 1:57 PM

Optimistically resolving this ticket, I think both https://gerrit.wikimedia.org/r/c/operations/puppet/+/530080 and T222366 have fixed it.

fgiunchedi reopened this task as Open.Oct 22 2020, 12:14 PM

Unfortunately reopening, we've been seeing failures (e.g. systemd, ssh) during latest codfw rebalances

• lmata removed a project: observability.Oct 26 2020, 3:45 PM
• lmata subscribed.

I'm going to un tag Observability for now as this is more swift related and less o11y related. :-) if this changes please retag

fgiunchedi mentioned this in T273453: ms-fe.svc.codfw.wmnet paged during Swift rebalance.Feb 1 2021, 1:27 PM
gerritbot added a comment.Feb 1 2021, 3:05 PM

Change 660854 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] swift: limit rsync service memory

https://gerrit.wikimedia.org/r/660854

gerritbot added a project: Patch-For-Review.Feb 1 2021, 3:05 PM

Change 660855 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] swift: limit rsync to 10% memory in codfw

https://gerrit.wikimedia.org/r/660855

gerritbot added a comment.Feb 3 2021, 10:27 AM

Change 660854 merged by Filippo Giunchedi:
[operations/puppet@production] swift: limit rsync service memory

https://gerrit.wikimedia.org/r/660854

gerritbot added a comment.Feb 3 2021, 10:29 AM

Change 660855 merged by Filippo Giunchedi:
[operations/puppet@production] swift: limit rsync to 10% memory in codfw

https://gerrit.wikimedia.org/r/660855

gerritbot added a comment.Feb 3 2021, 10:33 AM

Change 661343 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] role: default swift rsync memory limit

https://gerrit.wikimedia.org/r/661343

gerritbot added a comment.Feb 3 2021, 10:36 AM

Change 661343 merged by Filippo Giunchedi:
[operations/puppet@production] role: default swift rsync memory limit

https://gerrit.wikimedia.org/r/661343

fgiunchedi added a project: User-fgiunchedi.Feb 3 2021, 10:38 AM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.
Maintenance_bot removed a project: Patch-For-Review.Feb 3 2021, 11:10 AM
Stashbot added a comment.Feb 3 2021, 2:19 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-03T14:19:55Z] <godog> test memory limits on swift-object-replicator on ms-be2050 - T221904

gerritbot added a comment.Feb 3 2021, 4:35 PM

Change 661408 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] swift: limit rsync and swift-object-replicator memory to 5% in codfw

https://gerrit.wikimedia.org/r/661408

gerritbot added a project: Patch-For-Review.Feb 3 2021, 4:35 PM
gerritbot added a comment.Feb 4 2021, 2:10 PM

Change 661408 merged by Filippo Giunchedi:
[operations/puppet@production] swift: limit rsync and swift-object-replicator memory to 5% in codfw

https://gerrit.wikimedia.org/r/661408

Maintenance_bot removed a project: Patch-For-Review.Feb 4 2021, 3:10 PM
fgiunchedi added a comment.Feb 5 2021, 1:48 PM

Limiting the memory of rsync (receive side) and swift-object-replicator (sender side) has helped quite a bit in bounding the read/write latency experienced by clients. See the screenshot below, with feb 1st rebalance causing big spikes in latency, whereas subsequent rebalances (where latency spikes, but not as much)

2021-02-05-144735_948x615_scrot.png (615×948 px, 123 KB)

gerritbot added a comment.Feb 8 2021, 1:18 PM

Change 662703 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: add defaults for profile::swift::storage::replication_limit_memory_percent

https://gerrit.wikimedia.org/r/662703

gerritbot added a project: Patch-For-Review.Feb 8 2021, 1:18 PM
fgiunchedi mentioned this in T274092: Beta cluster Swift backend instances are missing profile::swift::storage::rsync_limit_memory_percent (puppet fails).Feb 8 2021, 1:25 PM
gerritbot added a comment.Feb 9 2021, 9:26 AM

Change 662703 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: add defaults for profile::swift::storage

https://gerrit.wikimedia.org/r/662703

gerritbot added a comment.Feb 9 2021, 9:30 AM

Change 662907 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] swift: limit rsync and swift-object-replicator memory to 5% in eqiad

https://gerrit.wikimedia.org/r/662907

gerritbot added a comment.Feb 9 2021, 1:32 PM

Change 662907 merged by Filippo Giunchedi:
[operations/puppet@production] swift: limit rsync and swift-object-replicator memory to 5% in eqiad

https://gerrit.wikimedia.org/r/662907

fgiunchedi closed this task as Resolved.Feb 15 2021, 2:09 PM

I'm boldly resolving this again since limiting memory usage for object replication processes helped a whole lot to make rebalances quiet

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits