+1 to the idea of deprecating the Beta Cluster, once there's a better replacement. I would suggest retitling this task to make the objective clear, either "Blue-green deployment" if the goal is to focus on a single replacement infrastructure, or "Deprecate Beta Cluster" if the goal is simply to get rid of the cluster by any possible means.

I see this task is brand new so this is just a recommendation for future work, but as you refine the RFC please link to existing work around this topic, such as T168843: Continous Delivery, https://wikitech.wikimedia.org/wiki/Deployment_pipeline and so on, to improve the quality of discussion here.

Am I reading this correctly that the proposed Beta replacement would be directly using and updating production data and also share the production bottlenecks (such as databases)?

In T308283#7927987, @Majavah wrote:

Am I reading this correctly that the proposed Beta replacement would be directly using and updating production data and also share the production bottlenecks (such as databases)?

You are reading that right.

The biggest unknown around this idea is data.

Using separate database has pros and cons:

• (pro) ability to perform destructive end-to-end testing
• (pro) could be updated on-the-fly with an automated process without worry
• (con) limits our ability to reproduce production scenarios
• Unknowns: what data? Where would it come from and where would it live?

Likewise using production data has pros and cons:

• (con) may be destructive to production data (although, if this is a post-merge environment, we may find that out with the train in a less controlled way)
• (con) New code can access PII (although, again, in a post-merge environment, this code will be doing that in a week to an hour anyway)
• (pro) new code can be manually tested with production data
• (pro) no maintenance of a secondary system
• Unknowns: how to update database schema?

In T308283#7928181, @thcipriani wrote:

You are reading that right.

Thank you for that clarification! I have a couple of comments regarding your points though.

• (con) limits our ability to reproduce production scenarios

Does it? We already have the ability to hack mwdebug servers to run whatever code we need for debugging purposes. I don't think the alternative (merging debugging code to master and then reverting it) is any better.

• (con) may be destructive to production data (although, if this is a post-merge environment, we may find that out with the train in a less controlled way)

This doesn't take into account that some code is behind a feature flag and only enabled when the developer manually wants it to be enabled. For example the extension I help maintain, CentralAuth, is a security-critical component that we can't test on a single isolated wiki on the production realm. There are two major use cases where a proper staging environment gives us something that local testing can't give us:

• There will always be weird edge cases that we can't think of when testing things locally. These are usually created by small behavior changes when the code is updated or when a new set of data is introduced to new entries that can't be backfilled to old rows. A widely-used staging environment will have a small amount of these edge cases that would otherwise be found in production.
• Some components, such as various levels of caching, are difficult to replicate locally with production-like behavior. Being able to test new code with the components and overall setup similar to production makes me more confident that the production deployment will not have any serious problems.

In T308283#7928242, @Majavah wrote:

In T308283#7928181, @thcipriani wrote:

You are reading that right.

Thank you for that clarification! I have a couple of comments regarding your points though.

• (con) limits our ability to reproduce production scenarios

Does it? We already have the ability to hack mwdebug servers to run whatever code we need for debugging purposes. I don't think the alternative (merging debugging code to master and then reverting it) is any better.

This was something mentioned by @cscott about parsoid testing. But you're right that many (most? some? unknown) have access to mwdebug machines.

• (con) may be destructive to production data (although, if this is a post-merge environment, we may find that out with the train in a less controlled way)

This doesn't take into account that some code is behind a feature flag and only enabled when the developer manually wants it to be enabled. For example the extension I help maintain, CentralAuth, is a security-critical component that we can't test on a single isolated wiki on the production realm. There are two major use cases where a proper staging environment gives us something that local testing can't give us:

• There will always be weird edge cases that we can't think of when testing things locally. These are usually created by small behavior changes when the code is updated or when a new set of data is introduced to new entries that can't be backfilled to old rows. A widely-used staging environment will have a small amount of these edge cases that would otherwise be found in production.
• Some components, such as various levels of caching, are difficult to replicate locally with production-like behavior. Being able to test new code with the components and overall setup similar to production makes me more confident that the production deployment will not have any serious problems.

I'm trying to follow you're saying, but I'm not clear: Are you saying that rollout on the train is more controlled than I think it is? Or that a true pre-production environment would be helpful for testing? Or something else?

Regarding data and databases in production. You could potentially have a dedicated section in databases and make sure the appservers in test environment only be able to connect that section. We already have a test-s4 section. This would need some work wrt our grants and dedicated hardware (2 boxes per dc = 4) but not impossible to do.