Current plan for this is:
- change three new osd nodes to ADD the public network ip (actually production private) as an additional cluster network (so they can talk to both the new cluster network and the old one) (via local hack, will be reverted by puppet)
- add those new osd nodes to the existing pool
- remove old osds one by one, wait for rebuilds
- rebuild new osds with modern settings
- remove the public network IP from the cluster network config on our three new osd nodes one at a time because it will restart OSD daemons