@CDanis says that we might be able to use one of the Ceph clusters we have in production now as we have a few existing data streams that get picked up by puppetservers that way.

cc @jgleeson @Ejegg and @Lars - feel free to ask questions here if you need to dig into details beforehand about how we will handle the transfer from CiviCRM to this environment (for example if you want to get guidance on encryption options).

In particular it'd be good to know if you only care about encryption going over the wire, or if encryption at rest is also necessary.

Hi @CDanis, nice to meet you. We are keen to use encryption for this project to protect PII. The current thinking is to agree on a shared secret and use symmetric encryption (AES) to protect the data.

For the initial full exports, we estimate the file size to be around 1–2 GB, which should allow for efficient encryption and decryption. Following that first export, we plan to send diffs, which will dramatically reduce the file size for subsequent transfers.

Change #1255036 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] Add fundraising-data-uploader role user

https://gerrit.wikimedia.org/r/1255036

Change #1255036 merged by CDanis:

[operations/puppet@production] deployment_server: fundraising-data-uploader role

https://gerrit.wikimedia.org/r/1255036