This is a follow-up actionable for the T239874 incident.
Avoid future cases of unintentionally leaving a former master pooled as zero-weight replica.
What we know
- Masters usually have weight: 0.
- There is usually no replica with weight: 0.
- We sometimes use weight: 1 for replicas that should receive little to no traffic but should still be waited for in terms of avoiding replication lag (and to use as backup in case of issues).
- We know that weight: 0 does currently result in at least some attempted connections from MW (per T239874, MW tried at least 4000 times per hour every hour throughout a 24 hour period). It is the topic of T239900 to discuss whether this is desirable long-term.
Enforce with some validation logic or schema in dbctl that a replica cannot have zero weight. This is meant to avoid the footgun scenario where a new master is prepended via dbctl but then to forget changing the configuration for the former master.
Making a zero-weight replica illegal means the operator will either have to depool it properly or to set its weight to at least 1.
I don't have a strong preference for this proposal. I don't know this area very well, so if I got something wrong or if there's a different/better way we can/should do this instead, please suggest it :)