The use-case here is TLS private keys as well as various more-ephemeral keys which are rotated frequently (e.g. TLS Ticket keys, TCP Fastopen keys, DNS Cookie Keys, etc). There are really 3 sub-problems to solve here:
- For the ticket-like cases (as opposed to actual TLS private keys from externally-signed certs), having a system which manages the rotation of small fixed secrets into keyfiles in some secure central location, in memory (tmpfs) only. Every case may require unique solutions here in terms of how the output is formatted and how many prior keys are kept, etc. Most cases will end up something like: generate a new 48-byte random output every 12 hours, and store the last 3 outputs in time order in a single file.
- Having a way to distribute these securely to many servers, storing them only in memory (e.g. tmpfs), yet ensuring they're available before the consuming daemons start on reboot. There was some past work on this in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/447804/ and related, and I think that's probably still the best approximate path forward.
- For the TLS privkey types of problems: migrating key paths to the new location (various downtimes and moves of private data to new paths, etc).