I'm very interested in this topic. Over the past couple of years, I've been involved in a number of discussions on the topic, especially that of making beta cluster more "production-like". However, I don't think we've defined what "production-like" means. I'm of the belief that we may need to approach the question differently. Right now the discussions seem to be focused on a single production-like staging environment. Perhaps viewing it as several testing environments that have characteristics of the production would yield some alternatives that are more achievable. I think coupling that discussion with adiscussion about testing approaches would also be beneficial.

Along the same lines as what @Jrbranaa said, it would also be interesting to think about "local-development-environment-like" in addition to "production-like", meaning is it possible to have a future where we can share configuration and tooling among local/QA/CI/production.

Perhaps viewing it as several testing environments that have characteristics of the production would yield some alternatives that are more achievable

Yes, agreed, and maybe also broadening this out a little bit to talk about Netlify-style on-demand QA environments that can be created on a per-patch basis, with predefined recipes for content / users / configuration / extensions in these environments.

Thanks @Jrbranaa and @kostajh for the input. The topic is indeed pretty broad, and, to actually make the work at the conference productive, its scope should probably be narrowed down, or split into multiple sessions.

I find the following ideas of yours very interesting:

Right now the discussions seem to be focused on a single production-like staging environment. Perhaps viewing it as several testing environments that have characteristics of the production would yield some alternatives that are more achievable.

Yes, agreed, and maybe also broadening this out a little bit to talk about Netlify-style on-demand QA environments that can be created on a per-patch basis, with predefined recipes for content / users / configuration / extensions in these environments.

I've recently discussed with a colleague of mine that what we miss from the existing staging environment (the fact it is a single one is also relevant/problematic!) is the ability to define and launch (in an automated way) an environment with particular production-like configuration (extensions, services, etc) but, what seems often overlooked, also the given set of feature flags. This would make development, testing and releasing new features to the complex systems we deal with significantly easier, less error-prone and simply nicer.

Does this blurry idea somehow corresponds, at least partly, to what you had in mind @Jrbranaa @kostajh. Do you think it would make sense to try to specify this session further in this direction?

Does this blurry idea somehow corresponds, at least partly, to what you had in mind @Jrbranaa @kostajh. Do you think it would make sense to try to specify this session further in this direction?

Yes, it does, especially the part about "also the given set of feature flags. This would make development, testing and releasing new features to the complex systems we deal with significantly easier, less error-prone and simply nicer."

It doesn't seem like we are that far away, I think what is missing is the ability to easily define CI settings (which could be done per-patch/branch), for example in extension.json the ability to have:

 json
"SomeVar": {
  "description": "Some feature flag.",
  "value": false,
  "ci": true
}

Would help get us part of the way. I haven't seen this done, but I imagine we could also standardize on writing maintenance scripts that run on install which could populate content or make further changes by checking to see if the wgWikimediaJenkinsCI variable is true.

These things would both be useful for CI, but they'd apply as well to creating a throwaway QA environment as well.

Maybe we could use Quibble for this since it already knows how to clone dependencies and such -- on toolforge, we could have an app that listens for particular comments coming from Gerrit ("make environment") which would then use Quibble to run a container with a publicly exposed port for a certain period of time, and report that URL back to Gerrit.

Do you think it would make sense to try to specify this session further in this direction?

@WMDE-leszek Yes, I think this direction makes sense. As @kostajh mentioned, I think there are things in place that we can build on already from a tech perspective. I do think that it will require some change in the way people see/plan/execute test though. In my experience, some of the most difficult discussions around this have been what "production-like" actually means and the testing context in which those production-like attributes matter (i.e., you don't need production-like scale if you're primarily interesting in compatibility with other extensions).

Good points being raised. Personally I see beta as actually two things.

• shadow service (use the latest software against existing data for verification), which it is not, since it uses separate data structures
• pre-production staging of significant software configuration changes and features. Which it is not, since it doesn't really reproduce production

I think this sort of causes the -like descriptions that people have given. Ideally both cases are more separated and more true to their intended purpose. This was not really possible before due to resource limitations, but maybe now we can be closer ?

(Programming note)

This session was accepted and will be scheduled.

Notes to the session leader

• Please continue to scope this session and post the session's goals and main questions into the task description.
  • If your topic is too big for one session, work with your Program Committee contact to break it down even further.
  • Session descriptions need to be completely finalized by November 1, 2019.
• Please build your session collaboratively!
  • You should consider breakout groups with report-backs, using posters / post-its to visualize thoughts and themes, or any other collaborative meeting method you like.
  • If you need to have any large group discussions they must be planned out, specific, and focused.
  • A brief summary of your session format will need to go in the associated Phabricator task.
  • Some ideas from the old WMF Team Practices Group.
• If you have any pre-session suggested reading or any specific ideas that you would like your attendees to think about in advance of your session, please state that explicitly in your session’s task.
  • Please put this at the top of your Phabricator task under the label “Pre-reading for all Participants.”

Notes to those interested in attending this session

(or those wanting to engage before the event because they are not attending)

• If the session leader is asking for feedback, please engage!
• Please do any pre-session reading that the leader would like you to do.

Wikimedia Technical Conference
Atlanta, GA USA
November 12 - 15, 2019

Session Name / Topic
Quo Vadis Beta Cluster? Towards production-like testing and staging
environments
Session Leader: Jean-Rene + Leszek; Facilitator: James; Scribe: Nick +
Leszek

Session Attendees
greg, andre, bryan, giuseppe, thedj, tobi, daniel, musikanimal,
niharika, daniel Z, reedy, maté, joaquin, petr, tpt, kosta

Notes:

• About multiple possible solutions, depending on requirements
• Desired outcome: a set of high level requirements / use-cases
  • these will become the seed elements for a Working Group
• Seed questions:
  • Why and when is parity with production important? -- not necessarily required for all types of testing
  • What are the most impactful problems wrt testing environments today?
  • What are some of the high level ideas to improve the testing environments? You can ask "why" to help focus
• Break them down into "Must have" and "Wants"

[Group discussions]

session overview comments

• 1 BD
  • our group ended up with too many people with too many scars, so kept rabbit-holing
  • spent most of the time talking about"when is parity important" - service integration, piece interacts with job queue and kafka and and and, then you need a more complete stack than one might normally use locally
    • Also need content complexity and variety, for different types of testing - E.g. templates can be hard to test for, because often not present in testing environments we have.
    • Full hardware and network parity for integration testing
    • last job of one person, had 3 separate identical testing environments, like our staging environment but a lot smaller (and thus affordable)
  • performance regression testing - not going to be identical, but *consistency* (and testing over time) is enough for most use-cases
• 2 GL
  • Overcharged[?] what beta cluster is in many ways
  • not the place where users a
  • data parity also came up
  • caching should be consistent - not 3 layers of madness in prod, but purging normalization, etc
  • performance testing is hard, but idea: branch hosting. Could show to Design people that way
  • Lack of production data
  • global sideeffects from multiple people testing things at the same time - e.g. someone testing compression. Could resolve by [?]. Isolated areas - expensive but impactful.
  • TLDR: Cannot have 1 environment that does all things.
• 3. DK
  • 3 areas of parity
    • scale of data, scale of traffic (perhaps unrealistic)
    • having similar set up (extensions, services)
    • community defined 'stuff' - e.g. gadgets templates
    • hardware paritty, at least same behaviour and scalability
    • security patches that are live
    • want to be able to test the deployment methods - what scap does (10 mins, except when it breaks...)
  • Problems
    • can use beta cluster to test that the code does what expected, but it is difficult to catch unwanted side effects (there is no comprehensive test suite that could be used for this)
  • 4. AK
    • group with many SREs...
    • Really important to have parity
      • end-to-end testing
      • integration tests, between services etc
      • deployment tooling (scap, helm, etc) and deploy testing
    • Need some parity
      • dev environments
      • schema changes (subset of the above)
      • data parity
      • performance with hardware, difficult one
      • QA
      • config management (e.g. puppet)
    • Not at all
      • unit tests
      • showcasing to PMs
      • branch testing

• - GL: 3 different envs we're talking about. They all came up when talking about beta cluster is [?] .
  • • end-to-end / integration testing
    • user-testing: branch hosting, experiments, etc. user-environment testing. Should be easy to spin up.
  • AK: Personal comment on staging, as there has been quite some history there. I think we should drop the idea of the staging environment, as we everyone would like to have one (would be expensive to maintain all of them)
  • AK: We have so much data that we can't create a staging environment. We should focus on testing everything that is possible to test.
    • GL: end-to-end integration testing. performance testing.
  • Summary of the let's ditch the term discussion: The ambiguity of the term "staging environment" is the issue that is causing some emotional response
  • PG: Would it help to draw a pipeline to illustrate the stages and boundaries that we need to serve?
    • That could be the step to do with the outcome of this session, to start to bucket these requirements
  • We're always focused on a monolithic single env to solve all our needs, but that's not really the desire

• [everyone place post-its on posters clustered into Musts and Wants]

• - TODO for doc sprint - transcribe Must and Want lists from posters to the notes
  • Now done?

• Musts
  • Deployment Tooling
  • Deployment of Services
  • Integration tests
  • edge to edge tests
  • browser tests
  • content complexity & variety are needed to get full coverage
  • actively monitored & maintained platform to create trust in results
  • mean time to test result needs to be low enough to maintain flow
  • isolation of tests running in parallel to reduce false positive failures
  • service integration needs config/complexity parity
  • Branch hosting
  • no significant downtime
  • Parity
    • services parity
    • data: pages, langlinks, templates...
    • partial on-demand pull (of content)
  • services testing
  • caching
    • purging behaviour
    • URL normalization
  • spin up test servers with different config, number of wikis, extensions, etc.
  • full user story automated tests (that aren't slow)
  • more rogue admins
  • impactful problems
    • lack of production data
    • global side effects from testing

• Wants
  • Dev env Schema changes
  • branch testing
  • QA showcase
  • Performance
  • Config Management
  • performance regression detection
  • full hardware and network parity for complete system integration testing
  • per patch QA environment with config, content, etc
  • data full import
  • load balancing