Change Details

Most of the daily toil on our DBA team is caused by repetitive actions to change the state of individual databases or of entire sections. This right now is done via code deploys; a typical maintenance window on a single server can require at least 3 code deployments. This is not acceptable, and we decided to move the storage of such state from the code repository to our consistent datastore, etcd. Given how critical and complex those data are, our standard `confctl` tool won't be enough to ensure data are correctly stored and to minimize the risk of errors and the strain on DBAs from using the tool. This CLI tool, that I propose to call `dbconfig`, will still be part of conftool, and will have the following functionalities: - Depool/warmup/repool fully a database server - Change weights per server - Set which server is the master for each section - Set read-only per-section or globally I also have a basic proposal for a UI, and the corresponding safety checks. # Get the current configuration * `dbconfig get NAME` gets you all the current configuration of a mysql instance (so either host:port or host only if it's the default port) * `dbconfig section [SECTION|all] show` shows what is configured for that specific section, or for all of them ** As a well formatted yaml ** If `--mediawiki` is added, the output will be in the format used by MediaWiki # Depool/pool/warmup * `dbconfig [depool|warmup|pool] [NAME] [selector tags]`, where selector tags are on function (main, vslow, rc, etc.) ** depool means removing from `sectionLoads` and `groupLoadsBySection` *** Sanity check 1: we're not removing the master *** Sanity check 2: we're not leaving a section with less than N slaves (configurable) *** Sanity check 3: we're not leaving a section that had databases for `vslow` without any ** pool means re-adding in `sectionLoads` and `groupLoadsBySection` with the previously-stored weights *** No sanity check needed, really ** warmup will work like pool, but will set the weight to a fraction of the declared one *** `dbconfig warmup NAME 0.1` pools a depooled database with weights set to `ceil(weights*0.1)`" *** No sanity check needed # Change the weights * `dbconfig edit NAME` allows you to edit the details of the configuration, including weights, in an editor ** All sanity checks we described above for depooling # Set master for a section * `dbconfig section SECTION get` shows the current configuration for the session metadata (master, enabled functions, minimum number of slaves) * `dbconfig section SECTION set-master NAME` sets the master for that specific section to NAME ** Sanity check 1: Will verify that the server is pooled for that section # Set a section read-only * `dbconfig section SECTION [ro|rw] REASON` will set the section to readonly with the specified reason

Most of the daily toil on our DBA team is caused by repetitive actions to change the state of individual databases or of entire sections. This right now is done via code deploys; a typical maintenance window on a single server can require at least 3 code deployments. This is not acceptable, and we decided to move the storage of such state from the code repository to our consistent datastore, etcd. Given how critical and complex those data are, our standard `confctl` tool won't be enough to ensure data are correctly stored and to minimize the risk of errors and the strain on DBAs from using the tool. This CLI tool, that I propose to call `dbconfig`, will still be part of conftool, and will have the following functionalities: - Depool/warmup/repool fully a database server - Change weights per server - Set which server is the master for each section - Set read-only per-section or globally I also have a basic proposal for a UI, and the corresponding safety checks. # Get the current configuration * `dbconfig get NAME` gets you all the current configuration of a mysql instance (so either host:port or host only if it's the default port) * `dbconfig section [SECTION|all] show` shows what is configured for that specific section, or for all of them ** As a well formatted yaml ** If `--mediawiki` is added, the output will be in the format used by MediaWiki # Depool/pool/warmup * `dbconfig [depool|pool] [NAME] [selector tags] [pooled fraction]`, where selector tags are on the group (main, vslow, rc, etc.) ** depool means removing from `sectionLoads` and `groupLoadsBySection` *** Sanity check 1: we're not removing the master *** Sanity check 2: we're not leaving a section with less than N slaves (configurable) *** Sanity check 3: we're not leaving a section that had databases for `vslow` without any ** pool means re-adding in `sectionLoads` and `groupLoadsBySection` with the previously-stored weights *** No sanity check needed, really ** warmup will work like pool, but will set the weight to a fraction of the declared one *** `dbconfig warmup NAME 0.1` pools a depooled database with weights set to `ceil(weights*0.1)`" *** No sanity check needed # Change the weights * `dbconfig edit NAME` allows you to edit the details of the configuration, including weights, in an editor ** All sanity checks we described above for depooling # Set master for a section * `dbconfig section SECTION get` shows the current configuration for the session metadata (master, enabled functions, minimum number of slaves) * `dbconfig section SECTION set-master NAME` sets the master for that specific section to NAME ** Sanity check 1: Will verify that the server is pooled for that section # Set a section read-only * `dbconfig section SECTION [ro|rw] REASON` will set the section to readonly with the specified reason

Most of the daily toil on our DBA team is caused by repetitive actions to change the state of individual databases or of entire sections. This right now is done via code deploys; a typical maintenance window on a single server can require at least 3 code deployments. This is not acceptable, and we decided to move the storage of such state from the code repository to our consistent datastore, etcd. Given how critical and complex those data are, our standard `confctl` tool won't be enough to ensure data are correctly stored and to minimize the risk of errors and the strain on DBAs from using the tool. This CLI tool, that I propose to call `dbconfig`, will still be part of conftool, and will have the following functionalities: - Depool/warmup/repool fully a database server - Change weights per server - Set which server is the master for each section - Set read-only per-section or globally I also have a basic proposal for a UI, and the corresponding safety checks. # Get the current configuration * `dbconfig get NAME` gets you all the current configuration of a mysql instance (so either host:port or host only if it's the default port) * `dbconfig section [SECTION|all] show` shows what is configured for that specific section, or for all of them ** As a well formatted yaml ** If `--mediawiki` is added, the output will be in the format used by MediaWiki # Depool/pool/warmup * `dbconfig [depool|warmup|pool] [NAME] [selector tags]` [pooled fraction]`, where selector tags are on functionthe group (main, vslow, rc, etc.) ** depool means removing from `sectionLoads` and `groupLoadsBySection` *** Sanity check 1: we're not removing the master *** Sanity check 2: we're not leaving a section with less than N slaves (configurable) *** Sanity check 3: we're not leaving a section that had databases for `vslow` without any ** pool means re-adding in `sectionLoads` and `groupLoadsBySection` with the previously-stored weights *** No sanity check needed, really ** warmup will work like pool, but will set the weight to a fraction of the declared one *** `dbconfig warmup NAME 0.1` pools a depooled database with weights set to `ceil(weights*0.1)`" *** No sanity check needed # Change the weights * `dbconfig edit NAME` allows you to edit the details of the configuration, including weights, in an editor ** All sanity checks we described above for depooling # Set master for a section * `dbconfig section SECTION get` shows the current configuration for the session metadata (master, enabled functions, minimum number of slaves) * `dbconfig section SECTION set-master NAME` sets the master for that specific section to NAME ** Sanity check 1: Will verify that the server is pooled for that section # Set a section read-only * `dbconfig section SECTION [ro|rw] REASON` will set the section to readonly with the specified reason