Change Details

We want to be able to easily switch what is visible as "staging" to the developers/deployers between the two staging clusters (staging-eqiad and staging-codfw). That would give us the freedom to easily test out things in one of the clusters and then switch deploys over once we're ready. This should hide the fact that there are actually two staging clusters as far away from devs/deployers as possible. When switching from staging-eqiad to staging-codfw (and vice versa) we would need to: * Ensure all services currently deployed on staging-eqiad are deployed to staging-codfw * Switch/link /etc/kubernetes/apertium-staging.config on deploy* hosts to /etc/kubernetes/apertium-staging-codfw.config Open questions: * How to deal with Prometheus, Kibana and Grafana dashboards (we probably can't/should not hide the two clusters there as we will need to be able to observe both)? * How to deal with DNS (staging.svc.eqiad.wmnet / staging.svc.codfw.wmnet). Should we have something DC agnostic that we can switch transparently as well?

We want to be able to easily switch what is visible as "staging" to the developers/deployers between the two staging clusters (staging-eqiad and staging-codfw). That would give us the freedom to easily test out things in one of the clusters and then switch deploys over once we're ready. This should hide the fact that there are actually two staging clusters as far away from devs/deployers as possible. When switching from staging-eqiad to staging-codfw (and vice versa) we would need to: * Ensure all services currently deployed on staging-eqiad are deployed to staging-codfw * Switch/link /etc/kubernetes/apertium-staging.config on deploy* hosts to /etc/kubernetes/apertium-staging-codfw.config Open questions: * How to deal with Prometheus, Kibana and Grafana dashboards (we probably can't/should not hide the two clusters there as we will need to be able to observe both)? * How to deal with DNS (staging.svc.eqiad.wmnet / staging.svc.codfw.wmnet). Should we have something DC agnostic that we can switch transparently as well? We have a manual process documented at: https://wikitech.wikimedia.org/wiki/Kubernetes#Switch_the_active_staging_cluster_(eqiad%3C-%3Ecodfw) This does not include changes to monitoring. We currently expect switches to be very short lived, so there probably is no need to invest any additional time.

We want to be able to easily switch what is visible as "staging" to the developers/deployers between the two staging clusters (staging-eqiad and staging-codfw). That would give us the freedom to easily test out things in one of the clusters and then switch deploys over once we're ready. This should hide the fact that there are actually two staging clusters as far away from devs/deployers as possible. When switching from staging-eqiad to staging-codfw (and vice versa) we would need to: * Ensure all services currently deployed on staging-eqiad are deployed to staging-codfw * Switch/link /etc/kubernetes/apertium-staging.config on deploy* hosts to /etc/kubernetes/apertium-staging-codfw.config Open questions: * How to deal with Prometheus, Kibana and Grafana dashboards (we probably can't/should not hide the two clusters there as we will need to be able to observe both)? * How to deal with DNS (staging.svc.eqiad.wmnet / staging.svc.codfw.wmnet). Should we have something DC agnostic that we can switch transparently as well? We have a manual process documented at: https://wikitech.wikimedia.org/wiki/Kubernetes#Switch_the_active_staging_cluster_(eqiad%3C-%3Ecodfw) This does not include changes to monitoring. We currently expect switches to be very short lived, so there probably is no need to invest any additional time.