When we operate sX sections, if a host that is being promoted to master via sudo dbctl --scope XXX section sX set-master is depooled, dbctl will error with:
Execution FAILED Reported errors: Section sX has no master
This is good as otherwise you'll end up with a section with no master. The future new master always needs to be pooled (even with weight 0).
When testing dbctl + parsercache this wasn't the case, and I was allowed to promote a depooled replica to master, which means the section ended up with no master. How to reproduce (this is safe to run as of today as right now the main config for parsercache is still MW):
root@cumin1002:~# sudo dbctl --scope codfw section pc4 set-master pc2015 WARNING: 'pc2015' is not a candidate master for section 'pc4'
I ignore the warning, as it is just a warning and commit:
root@cumin1002:~# dbctl config commit -m "Test pc4 master switch"
codfw/externalLoads/pc4 live codfw/externalLoads/pc4 generated
[ {}
{
"pc2016": 1
},
{}
]
Enter y or yes to confirm: y
Previous configuration saved. To restore it run: dbctl config restore /var/cache/conftool/dbconfig/20240516-103039-marostegui.json
WARNING:conftool.announce:dbctl commit (dc=all): 'Test pc4 master switch', diff saved to https://phabricator.wikimedia.org/P62494 and previous config saved to /var/cache/conftool/dbconfig/20240516-103039-marostegui.jsonThe section ends empty.
pc2015 is a replica on pc4 but it is depooled:
root@cumin1002:~# dbctl instance pc2015 get
{
"pc2015": {
"host_ip": "10.192.32.132",
"note": "",
"port": 3306,
"sections": {
"pc4": {
"percentage": 100,
"pooled": false,
"weight": 1
}
}
},
"tags": "datacenter=codfw"
}