== Motivation
Currently configuration updates are applied immediately. This makes it impossible to do any sanity checks before deploying, e.g. using the canary feature we have for code deployments. Canary has worked perfectly to avoid downtime due to bad code deployments, but at 2020-07-29 we had 6 minute downtime due to mistake in a configuration update.
== Proposal
Use normal deployment process also for configuration updates.
=== Process comparison
Currently, to do a configuration update:
# Submit a patch to translatewiki repository
# Have the patch merged
# Log in to web2
# Run `twn-update-config` (change is now deployed)
# (not enforced) test in production
# (not enforced) monitor logs
If it would go through normal deployments:
# Submit a patch to translatewiki repository
# Have the patch merged
# Log in to web2
# Run `twn-update-config`
# (not enforced) test in canary
# (not enforced) check logs
# `cd /srv/mediawiki`
# `b oregano tag`
# `b oregano deploy` (change is now deployed)
# (not enforced) monitor logs
We could a a single command to do steps 7-9 to make it a bit easier.
=== Pros and cons
| Current process | New process |
| {icon plus-circle} Simple and fast | {icon plus-circle} Will not cause downtime if checked on canary first
| {icon minus-circle} Risky, can cause downtime | {icon plus-circle} Same process, no surprises
| {icon minus-circle} Different, surprising process compared to code deployments | {icon minus-circle} Additional steps
|| {icon minus-circle} Requires learning how to use canary, and it cannot be enforced
|| {icon minus-circle} May cause "split-brain" scenario as caches are shared (already happens for code, but all such changes (database schemas, message keys) should be done to take this into account
== Implementation notes
* Currently the configuration resides outside the deployment directory. Either we need to move it inside it, or have required files copied over.