* We have a distributed data store cobbled together with shell scripts, rsync, etc. It's incredibly opaque
* It's the biggest bottleneck on the deployment process. But truly novel data (i.e., new messages/translations) represents a tiny sliver of the byte payload, which consists primarily of unmodified messages that already exist on each deployment target.
One thing that makes our current deployments unreliable is the fact that we have many workarounds to make it possible to avoid rewriting the l10n cache. Ideally, we would use "scap" no matter how small the change we're making to the site, but that's not practical due to l10n cache rebuild and distribution often taking over 10 minutes. Some of us have some nascent ideas about how to make this faster, but we haven't yet agreed on a solution we want to pursue.