Intro
As of Nov 2022 (T323349), we have a scap helping script populated on WikiKube workers to prefetch mediawiki images so that deployment times in MW-on-K8s work fast enough.
Problem statement
Since then, there have been
- a number of improvements in the infrastructure as well as (e.g. we no longer have nodes with HDDs)
- a number of regression that have been addressed (e.g. T366094)
- Plans to see how trimming down OCI image size (via altering how we do MultiVersion)
Furthermore, this script has also caused a number of failed deployments as it makes scap WikiKube deployments aware of individual hosts (e.g. a hardware failed WikiKube host shows up in scap output) and requiring manual actions by an SRE to set the host as inactive in conftool. The UX is also not great for the deployer as they don't get to benefit from one of the advantages of deploying to a kubernetes cluster, which is to abstract away specific nodes.
We need a way to gauge the the overall effects this script has and whether it's still needed now, in the short term and in the medium term future.
Description of plan
Add the ability to the script we have populated to skip execution in a probabilistic way. We define a percentage based threshold and then lower that threshold every few weeks and gauge the results.
Support has been added in https://gerrit.wikimedia.org/r/q/topic:%22mediawiki-image-download_pct%22
Some rough steps for the percentage would be:
- 99%
- 90%
- 80%
- 75%
- 66%
- 50%
- 25%
- 5%
- 0%
At each of steps listed above, we stop and gauge the effects of the change. Given the duration needed to show something worthy of talking about, there is a good chance we will see regressions or fixes that mess with the numbers. This is something we will have to deal with.
Measurement
A simple plan to gauge this is to rely on scap action timings. The data exists in Logstash and should allow us to figure out the impact of our changes (up to a degree)
