It would be nice able to connect to ceph clusters across DCs using encrypted traffic (see [1]).
To do so we have to enable client traffic encryption on the ceph clusters:
https://docs.ceph.com/en/latest/rados/configuration/msgr2/#confval-ms_client_mode
This might require reconfiguring nova/cinder/glance too
We have to make sure though that the current versions of librbd (openstack) support it.
I think the current bdsync-based backup solution for NFS shares doesn't use any kind of encryption when flying across DCs. This is to say: enabling encryption for ceph traffic is nice, but in my opinion shouldn't be a blocker for introducing the cinder-backup solution.
I think the current bdsync-based backup solution for NFS shares doesn't use any kind of encryption when flying across DCs. This is to say: enabling encryption for ceph traffic is nice, but in my opinion shouldn't be a blocker for introducing the cinder-backup solution.
Ack, I'll reword the task :)
Do we know if by any chance the change would be trivial, or if a major client/server config rework would be involved?
it's a ceph setting (unless we want to use some kinda proxy) that does not need a keypair, but requires (for what I've looked into it so far) cold restarting the whole cluster.
It seems that the current WMCS cluster does support encryption (secure mode) already. I guess probably a by-product of some cluster upgrades?
$ sudo ceph config show-with-defaults mon.cloudcephmon1004 -f json-pretty | grep -B1 -A3 'ms_.*_mode' { "name": "ms_client_mode", "value": "crc secure", "source": "default" }, { "name": "ms_cluster_mode", "value": "crc secure", "source": "default" }, -- { "name": "ms_mon_client_mode", "value": "secure crc", "source": "default" }, { "name": "ms_mon_cluster_mode", "value": "secure crc", "source": "default" }, { "name": "ms_mon_service_mode", "value": "secure crc", "source": "default" }, -- { "name": "ms_osd_compress_mode", "value": "none", "source": "default" }, -- { "name": "ms_service_mode", "value": "crc secure", "source": "default" },
(related documentation: https://docs.ceph.com/en/latest/rados/configuration/msgr2/#connection-mode-configuration-options )
From a quick test from cloudbackup2004, forcing the secure connection in the config file:
[global] ms_client_mode = secure
does seem to work fine:
$ sudo ceph health HEALTH_OK
(and tcpdump looks more gibberish than before, even though for health is already not totally in clear).
As a result of this I'm sending a patch to add this setting via a hiera setting and a subsequent patch to enable it for the wmcs::openstack::codfw1dev::cinder_backups hosts (cloudback1*-dev, cross-dc backups) so that we can test it.
Change #1302904 had a related patch set uploaded (by Volans; author: Volans):
[operations/puppet@production] ceph: allow to set client transport encryption
Change #1302905 had a related patch set uploaded (by Volans; author: Volans):
[operations/puppet@production] Cinder backups: enable transport encryption part 1
Change #1302904 merged by Volans:
[operations/puppet@production] ceph: allow to set client transport encryption
Change #1302905 merged by Volans:
[operations/puppet@production] Cinder backups: enable transport encryption part 1
Change #1303961 had a related patch set uploaded (by Volans; author: Volans):
[operations/puppet@production] Cinder backups: enable transport encryption part 2
Change #1303961 merged by Volans:
[operations/puppet@production] Cinder backups: enable transport encryption part 2
With the above patches the cross-site traffic (cinder volume backups) is now encrypted and the backups should fail if they are unable, not falling back to crc.
It would be very easy to expand the above patches to make all the traffic encrypted.