I'm very interested in this topic. Over the past couple of years, I've been involved in a number of discussions on the topic, especially that of making beta cluster more "production-like". However, I don't think we've defined what "production-like" means. I'm of the belief that we may need to approach the question differently. Right now the discussions seem to be focused on a single production-like staging environment. Perhaps viewing it as several testing environments that have characteristics of the production would yield some alternatives that are more achievable. I think coupling that discussion with adiscussion about testing approaches would also be beneficial.

Along the same lines as what @Jrbranaa said, it would also be interesting to think about "local-development-environment-like" in addition to "production-like", meaning is it possible to have a future where we can share configuration and tooling among local/QA/CI/production.

Perhaps viewing it as several testing environments that have characteristics of the production would yield some alternatives that are more achievable

Yes, agreed, and maybe also broadening this out a little bit to talk about Netlify-style on-demand QA environments that can be created on a per-patch basis, with predefined recipes for content / users / configuration / extensions in these environments.

Thanks @Jrbranaa and @kostajh for the input. The topic is indeed pretty broad, and, to actually make the work at the conference productive, its scope should probably be narrowed down, or split into multiple sessions.

I find the following ideas of yours very interesting:

Right now the discussions seem to be focused on a single production-like staging environment. Perhaps viewing it as several testing environments that have characteristics of the production would yield some alternatives that are more achievable.

Yes, agreed, and maybe also broadening this out a little bit to talk about Netlify-style on-demand QA environments that can be created on a per-patch basis, with predefined recipes for content / users / configuration / extensions in these environments.

I've recently discussed with a colleague of mine that what we miss from the existing staging environment (the fact it is a single one is also relevant/problematic!) is the ability to define and launch (in an automated way) an environment with particular production-like configuration (extensions, services, etc) but, what seems often overlooked, also the given set of feature flags. This would make development, testing and releasing new features to the complex systems we deal with significantly easier, less error-prone and simply nicer.

Does this blurry idea somehow corresponds, at least partly, to what you had in mind @Jrbranaa @kostajh. Do you think it would make sense to try to specify this session further in this direction?

Does this blurry idea somehow corresponds, at least partly, to what you had in mind @Jrbranaa @kostajh. Do you think it would make sense to try to specify this session further in this direction?

Yes, it does, especially the part about "also the given set of feature flags. This would make development, testing and releasing new features to the complex systems we deal with significantly easier, less error-prone and simply nicer."

It doesn't seem like we are that far away, I think what is missing is the ability to easily define CI settings (which could be done per-patch/branch), for example in extension.json the ability to have:

 json
"SomeVar": {
  "description": "Some feature flag.",
  "value": false,
  "ci": true
}

Would help get us part of the way. I haven't seen this done, but I imagine we could also standardize on writing maintenance scripts that run on install which could populate content or make further changes by checking to see if the wgWikimediaJenkinsCI variable is true.

These things would both be useful for CI, but they'd apply as well to creating a throwaway QA environment as well.

Maybe we could use Quibble for this since it already knows how to clone dependencies and such -- on toolforge, we could have an app that listens for particular comments coming from Gerrit ("make environment") which would then use Quibble to run a container with a publicly exposed port for a certain period of time, and report that URL back to Gerrit.

Do you think it would make sense to try to specify this session further in this direction?

@WMDE-leszek Yes, I think this direction makes sense. As @kostajh mentioned, I think there are things in place that we can build on already from a tech perspective. I do think that it will require some change in the way people see/plan/execute test though. In my experience, some of the most difficult discussions around this have been what "production-like" actually means and the testing context in which those production-like attributes matter (i.e., you don't need production-like scale if you're primarily interesting in compatibility with other extensions).

Good points being raised. Personally I see beta as actually two things.

• shadow service (use the latest software against existing data for verification), which it is not, since it uses separate data structures
• pre-production staging of significant software configuration changes and features. Which it is not, since it doesn't really reproduce production

I think this sort of causes the -like descriptions that people have given. Ideally both cases are more separated and more true to their intended purpose. This was not really possible before due to resource limitations, but maybe now we can be closer ?