With the default settings of c-foreach-restart it's no longer possible to reliably restart Cassandra on the restbase hosts (probably due to increased data set sizes or higher I/O), resulting in tracebacks like the following:
```
2018-06-14 08:46:12,288 INFO [b] Disabling client ports...
2018-06-14 08:46:21,521 INFO [b] Draining...
2018-06-14 08:51:33,693 INFO [b] Stopping service cassandra-b
2018-06-14 08:51:37,530 INFO [b] Starting service cassandra-b
2018-06-14 08:51:37,539 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:51:43,539 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:51:49,540 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:51:55,543 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:52:01,547 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:52:07,553 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:52:13,559 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:52:19,563 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:52:25,567 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:52:31,572 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:52:37,575 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:52:43,579 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:52:49,583 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:52:55,587 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:53:01,592 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:53:07,598 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:53:13,605 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:53:19,611 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:53:25,615 WARNING [b] CQL (10.192.32.135:9042) not listening (will retry)...
2018-06-14 08:53:37,616 ERROR [b] CQL (10.192.32.135:9042) DOWN
Traceback (most recent call last):
File "/usr/bin/c-foreach-restart", line 60, in <module>
main()
File "/usr/bin/c-foreach-restart", line 29, in main
post_shutdown=args.execute_post_shutdown
File "/usr/share/cassandra-tools-wmf/cassandra/tools/instances.py", line 92, in restart
raise Exception("{} restart FAILED".format(self.service_name))
Exception: cassandra-b restart FAILED
```
After some discussion with Eric I switched to "c-foreach-restart -d 10 -a 20 -r 12" which worked reliably (the remaining ~15 servers restarted without any issues with these settings).
Given that cassandra-wmf-tools is an internal package, should be update the defaults, so that this works out of the box?