Page MenuHomePhabricator

Wikimedia Technical Conference 2019 Session: Quo Vadis Beta Cluster? Towards better testing and staging environments
Open, Needs TriagePublic

Subscribers
Tokens
"Like" token, awarded by brennen."Like" token, awarded by MusikAnimal."Like" token, awarded by Jrbranaa."Love" token, awarded by hashar."Like" token, awarded by kostajh.
Assigned To
Authored By
debt, Fri, Oct 4

Description

Session

  • Track: Deploying and Hosting
  • Topic: Quo Vadis Beta Cluster? Towards better testing and staging environments

Description

So called "beta" sites have been commonly used as testing and staging environment for code changes before these are shipped to production. There are several issues with these sites used like a pre-production environment

  • they are hosted on the de facto not maintained infrastructure
  • There is no easy automated way to set the configuration (in the sense of extensions and services enabled, features set enabled, and other config options) of the testing/staging site, not to mention being able to launch a pre-produiction environment with the particular configuration and software versions, e.g. when intending to test compatibility of the new feature with different versions of extensions, skins, etc.
  • They are actual "permanent" sites, i.e. they're not suited for use as mid-/short-term living staging environments

In this session we will look into what requirements we would have for the staging environments used in our work, and also see what possible solutions could we see that could replace "beta" sites as testing and staging sites (e.g. using new possibilities allowed with the adoption of container-based solutions).

Questions to answer and discuss

Question:
Significance:

Question:
Significance:

Related Issues

  • ...
  • ...

Pre-reading for all Participants

  • [add links here]

Notes document(s)

[link to notes document (gdoc and / or etherpad)]

Notes and Facilitation guidance

https://www.mediawiki.org/wiki/Wikimedia_Technical_Conference/2019/NotesandFacilitation


Session Leader(s)

  • [name]
  • [name]

Session Scribes

  • [name]
  • [name]

Session Facilitator

  • [name]

Session Style / Format

  • [what type of format will this session be?]

Session Leaders please:

  • Add more details to this task description.
  • Coordinate any pre-event discussions (here on Phab, IRC, email, hangout, etc).
  • Outline the plan for discussing this topic at the event.
  • Optionally, include what this session will not try to solve.
  • Update this task with summaries of any pre-event discussions.
  • Include ways for people not attending to be involved in discussions before the event and afterwards.

Post-event summary:

  • ...

Post-event action items:

  • ...

Event Timeline

debt created this task.Fri, Oct 4, 3:33 PM
hashar awarded a token.Mon, Oct 7, 8:18 PM

I'm very interested in this topic. Over the past couple of years, I've been involved in a number of discussions on the topic, especially that of making beta cluster more "production-like". However, I don't think we've defined what "production-like" means. I'm of the belief that we may need to approach the question differently. Right now the discussions seem to be focused on a single production-like staging environment. Perhaps viewing it as several testing environments that have characteristics of the production would yield some alternatives that are more achievable. I think coupling that discussion with adiscussion about testing approaches would also be beneficial.

kostajh added a subscriber: kostajh.Wed, Oct 9, 9:49 AM

Along the same lines as what @Jrbranaa said, it would also be interesting to think about "local-development-environment-like" in addition to "production-like", meaning is it possible to have a future where we can share configuration and tooling among local/QA/CI/production.

Perhaps viewing it as several testing environments that have characteristics of the production would yield some alternatives that are more achievable

Yes, agreed, and maybe also broadening this out a little bit to talk about Netlify-style on-demand QA environments that can be created on a per-patch basis, with predefined recipes for content / users / configuration / extensions in these environments.

Thanks @Jrbranaa and @kostajh for the input. The topic is indeed pretty broad, and, to actually make the work at the conference productive, its scope should probably be narrowed down, or split into multiple sessions.

I find the following ideas of yours very interesting:

Right now the discussions seem to be focused on a single production-like staging environment. Perhaps viewing it as several testing environments that have characteristics of the production would yield some alternatives that are more achievable.

Yes, agreed, and maybe also broadening this out a little bit to talk about Netlify-style on-demand QA environments that can be created on a per-patch basis, with predefined recipes for content / users / configuration / extensions in these environments.

I've recently discussed with a colleague of mine that what we miss from the existing staging environment (the fact it is a single one is also relevant/problematic!) is the ability to define and launch (in an automated way) an environment with particular production-like configuration (extensions, services, etc) but, what seems often overlooked, also the given set of feature flags. This would make development, testing and releasing new features to the complex systems we deal with significantly easier, less error-prone and simply nicer.

Does this blurry idea somehow corresponds, at least partly, to what you had in mind @Jrbranaa @kostajh. Do you think it would make sense to try to specify this session further in this direction?

Does this blurry idea somehow corresponds, at least partly, to what you had in mind @Jrbranaa @kostajh. Do you think it would make sense to try to specify this session further in this direction?

Yes, it does, especially the part about "also the given set of feature flags. This would make development, testing and releasing new features to the complex systems we deal with significantly easier, less error-prone and simply nicer."

It doesn't seem like we are that far away, I think what is missing is the ability to easily define CI settings (which could be done per-patch/branch), for example in extension.json the ability to have:

 json
"SomeVar": {
  "description": "Some feature flag.",
  "value": false,
  "ci": true
}

Would help get us part of the way. I haven't seen this done, but I imagine we could also standardize on writing maintenance scripts that run on install which could populate content or make further changes by checking to see if the wgWikimediaJenkinsCI variable is true.

These things would both be useful for CI, but they'd apply as well to creating a throwaway QA environment as well.

Maybe we could use Quibble for this since it already knows how to clone dependencies and such -- on toolforge, we could have an app that listens for particular comments coming from Gerrit ("make environment") which would then use Quibble to run a container with a publicly exposed port for a certain period of time, and report that URL back to Gerrit.

Do you think it would make sense to try to specify this session further in this direction?

@WMDE-leszek Yes, I think this direction makes sense. As @kostajh mentioned, I think there are things in place that we can build on already from a tech perspective. I do think that it will require some change in the way people see/plan/execute test though. In my experience, some of the most difficult discussions around this have been what "production-like" actually means and the testing context in which those production-like attributes matter (i.e., you don't need production-like scale if you're primarily interesting in compatibility with other extensions).

WMDE-leszek renamed this task from Wikimedia Technical Conference 2019 Session: Quo Vadis Beta Cluster? Towards production-like testing and staging environments to Wikimedia Technical Conference 2019 Session: Quo Vadis Beta Cluster? Towards better testing and staging environments.Fri, Oct 11, 10:34 AM
WMDE-leszek updated the task description. (Show Details)
TheDJ added a subscriber: TheDJ.Sat, Oct 12, 2:20 PM

Good points being raised. Personally I see beta as actually two things.

  • shadow service (use the latest software against existing data for verification), which it is not, since it uses separate data structures
  • pre-production staging of significant software configuration changes and features. Which it is not, since it doesn't really reproduce production

I think this sort of causes the -like descriptions that people have given. Ideally both cases are more separated and more true to their intended purpose. This was not really possible before due to resource limitations, but maybe now we can be closer ?