Change Details

**Status quo:** For primary switchovers, we currently create a CRS support ticket ([[https://phabricator.wikimedia.org/T302283|example]]), Schedule it around two weeks in advance, add a note about it in Tech news, and CRS sometimes sends an email about it to mailing lists and set up banners before the window. We usually request half an hour of read-only but switchovers barely take more than a minute (while it used to be longer). **Issues:** - This is drastically reducing speed of doing database maintenance. We are currently doing a lot more changes and sometimes we have more than five schema changes waiting for primary switchovers ([[https://phabricator.wikimedia.org/T300471|example]]). Beside most of schema changes, other maintenance works like OS upgrades, reboots for security updates, MariaDB upgrades, etc. need primary switchovers as well. - A schema change that doesn't need a primary switchover takes around a week. A similar one but needing primary switchovers take at least two months or so. Hindering other teams' ability to do their work. - This is taking a lot of CRS resources, for the coordination and work. - This also takes a lot of volunteer time to translate the message while users wouldn't even notice this. - It adds noise to the tech news leading to other more important notes being drowned. **Notes:** - The read-only time is negligible and extremely small, usually around 30s, some large writes can cause longer read-only times. - The reason we ask for half an hour is that things might go wrong and read-only take longer but 1- This has not ever happened since this process started 2- This can happen with any type of maintenance work. - I tested locally and if the site is read-only and you try to save your edit (assuming you opened the edit page before read-only), The response still contains the edit and you can simply click on the Submit button, nothing gets lost according to my test. **Proposed solution** Add two 30 minutes slots per week (Tue/Thu) in the deployment calendar. 7:00 UTC which is an hour before morning EU backport window. This is when the db load is the lowest. DBAs would do the switchovers during that window (they can just not use that window if they don't have anything planned). No announcement will be sent but DBAs add the planned tickets in the "Changes" section of that window for bookkeeping.

**Status quo:** For primary switchovers, we currently create a ticket for CRS team ([[https://phabricator.wikimedia.org/T302283|example]]), Schedule it around two weeks in advance, add a note about it in Tech news, and CRS sometimes sends an email about it to mailing lists and set up banners before the window. We usually request half an hour of read-only but switchovers barely take more than a minute (while it used to be longer). **Issues:** - This is drastically reducing speed of doing database maintenance. We are currently doing a lot more changes and sometimes we have more than five schema changes waiting for primary switchovers ([[https://phabricator.wikimedia.org/T300471|example]]). Beside most of schema changes, other maintenance works like OS upgrades, reboots for security updates, MariaDB upgrades, etc. need primary switchovers as well. - A schema change that doesn't need a primary switchover takes around a week. A similar one but needing primary switchovers take at least two months or so. Hindering other teams' ability to do their work. - This is taking a lot of CRS resources, for the coordination and work. - This also takes a lot of volunteer time to translate the message while users wouldn't even notice this. - It adds noise to the tech news leading to other more important notes being drowned. **Notes:** - The read-only time is negligible and extremely small, usually around 30s, some large writes can cause longer read-only times. - The reason we ask for half an hour is that things might go wrong and read-only take longer but 1- This has not ever happened since this process started 2- This can happen with any type of maintenance work. - I tested locally and if the site is read-only and you try to save your edit (assuming you opened the edit page before read-only), The response still contains the edit and you can simply click on the Submit button, nothing gets lost according to my test. **Proposed solution** Add two 30 minutes slots per week (Tue/Thu) in the deployment calendar. 7:00 UTC which is an hour before morning EU backport window. This is when the db load is the lowest. DBAs would do the switchovers during that window (they can just not use that window if they don't have anything planned). No announcement will be sent but DBAs add the planned tickets in the "Changes" section of that window for bookkeeping.

**Status quo:** For primary switchovers, we currently create a CRS support ticketticket for CRS team ([[https://phabricator.wikimedia.org/T302283|example]]), Schedule it around two weeks in advance, add a note about it in Tech news, and CRS sometimes sends an email about it to mailing lists and set up banners before the window. We usually request half an hour of read-only but switchovers barely take more than a minute (while it used to be longer). **Issues:** - This is drastically reducing speed of doing database maintenance. We are currently doing a lot more changes and sometimes we have more than five schema changes waiting for primary switchovers ([[https://phabricator.wikimedia.org/T300471|example]]). Beside most of schema changes, other maintenance works like OS upgrades, reboots for security updates, MariaDB upgrades, etc. need primary switchovers as well. - A schema change that doesn't need a primary switchover takes around a week. A similar one but needing primary switchovers take at least two months or so. Hindering other teams' ability to do their work. - This is taking a lot of CRS resources, for the coordination and work. - This also takes a lot of volunteer time to translate the message while users wouldn't even notice this. - It adds noise to the tech news leading to other more important notes being drowned. **Notes:** - The read-only time is negligible and extremely small, usually around 30s, some large writes can cause longer read-only times. - The reason we ask for half an hour is that things might go wrong and read-only take longer but 1- This has not ever happened since this process started 2- This can happen with any type of maintenance work. - I tested locally and if the site is read-only and you try to save your edit (assuming you opened the edit page before read-only), The response still contains the edit and you can simply click on the Submit button, nothing gets lost according to my test. **Proposed solution** Add two 30 minutes slots per week (Tue/Thu) in the deployment calendar. 7:00 UTC which is an hour before morning EU backport window. This is when the db load is the lowest. DBAs would do the switchovers during that window (they can just not use that window if they don't have anything planned). No announcement will be sent but DBAs add the planned tickets in the "Changes" section of that window for bookkeeping.