Experiment with hosted kubernetes solutions for Beta
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	thcipriani
	May 8 2019, 4:50 PM

Description

As we're moving more services through the Deployment Pipeline to production, beta is beginning to suffer.

There are several proposed solutions; Let's see if using an existing hosted k8s solution is viable.

Problems

The Deployment Pipeline is currently unable to perform system tests that incorporate both a change to a service and an existing MediaWiki installation; It is limited to e2e testing only the service itself.
A k8s cluster that integrates with Beta Cluster (as in has secure network ingress/egress between deployed pods/services and existing deployment-prep instances) would allow the Deployment Pipeline to perform this kind of testing. However, at this time neither SRE nor RelEng can commit to maintaining an in-house k8s cluster for this purpose.

Proposal

Experiment with [third party k8s provider] to evaluate its potential as a third-party hosted k8s cluster that can:

Provide a k8s cluster that the Deployment Pipeline can target as part of its graduated deployment/testing strategy.
Securely integrate with Beta Cluster at a network level.
Run e2e helm tests that exercise service changes and existing MediaWiki deployments in Beta Cluster together.

Evaluation

A very basic test for teasing out whether any third party k8s is viable could be:

Can our existing Mathoid helm chart be used to deploy there?
If not, how much refactoring would the chart(s) need? More precisely, can we make them work with both [third party k8s] and WMF k8s without too much divergence?

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		akosiaris	T220235 Migrate Beta cluster services to use Kubernetes
		Declined		None	T222820 Experiment with hosted kubernetes solutions for Beta

Event Timeline

thcipriani created this task.May 8 2019, 4:50 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 8 2019, 4:50 PM

assigning to @dduvall based on hangout discussion

Restricted Application edited projects, added Release-Engineering-Team (Kanban); removed Release-Engineering-Team. · View Herald TranscriptMay 8 2019, 4:52 PM

Krenair subscribed.May 8 2019, 4:56 PM

dduvall updated the task description. (Show Details)May 8 2019, 5:41 PM

• LarsWirzenius subscribed.May 8 2019, 5:48 PM

dduvall updated the task description. (Show Details)May 8 2019, 5:48 PM

dduvall updated the task description. (Show Details)May 8 2019, 6:01 PM

dduvall updated the task description. (Show Details)

jeena subscribed.May 8 2019, 6:05 PM

Jdforrester-WMF subscribed.May 8 2019, 6:24 PM

I was able to run Mathoid just fine on GKE using the latest chart. What remains of this experiment, however, is getting Beta Cluster's MediaWiki talking to a service deployed to GKE (or Amazon EKS if that makes sense for us policy/budget wise) and vice versa; Basically the networking part.

A couple of options:

VPN between the deployment-prep labs project and Google GKE using ipsec and Google VPC. This might involve more investment to set up initially, but once (if) it's working Beta instances should be able to freely communicate with anything deployed on GKE. There may be a DNS component as well.
Ingress for both ends and public communicate. This might be easier to set up initially but requires additional ingress configuration for each service. MediaWiki/service communication shouldn't include anything sensitive, but maybe?

Unassigning while on leave. Anyone should feel free to pick this up.

greg added a project: Release-Engineering-Team-TODO.Jul 1 2019, 9:28 PM

greg moved this task from Should be empty (use Release-Engineering-Team) to Soon-ish on the Release-Engineering-Team-TODO board.Jul 1 2019, 9:30 PM

greg removed a project: Release-Engineering-Team (Kanban).Jul 1 2019, 9:31 PM

Moving to our current kanban board so @jeena can chat with @dduvall when he's back next week.

greg moved this task from INBOX to Ready on the Release-Engineering-Team-TODO (201907) board.Jul 17 2019, 5:45 PM

greg assigned this task to jeena.Jul 29 2019, 4:32 PM

greg edited projects, added Release-Engineering-Team-TODO (201908); removed Release-Engineering-Team-TODO (201907).

greg moved this task from INBOX to Ready on the Release-Engineering-Team-TODO (201908) board.

jeena moved this task from Ready to Doing on the Release-Engineering-Team-TODO (201908) board.Aug 10 2019, 1:06 AM

jeena edited projects, added Release-Engineering-Team-TODO (201909); removed Release-Engineering-Team-TODO (201908).Sep 17 2019, 10:38 PM

jeena moved this task from INBOX to Doing on the Release-Engineering-Team-TODO (201909) board.Sep 17 2019, 10:41 PM

jeena removed jeena as the assignee of this task.Sep 18 2019, 6:00 PM

jeena moved this task from Doing to INBOX on the Release-Engineering-Team-TODO (201909) board.

• Bstorm subscribed.Sep 18 2019, 7:01 PM

If this project ends up integrating with WMCS-managed stuff at all (Beta cluster -- does that mean deployment-prep?), I'd at least be interested in being a fly on the wall. I'm kind of curious what people come up with in general for our use or understanding, but if we are doing any peering or VPN with things in Cloud, I definitely would like to know to see how it impacts things.

bd808 subscribed.Sep 18 2019, 7:19 PM

In T222820#5504304, @Bstorm wrote:

If this project ends up integrating with WMCS-managed stuff at all (Beta cluster -- does that mean deployment-prep?), I'd at least be interested in being a fly on the wall. I'm kind of curious what people come up with in general for our use or understanding, but if we are doing any peering or VPN with things in Cloud, I definitely would like to know to see how it impacts things.

Yep, that was the tentative plan here; however, at the moment the Deployment Pipeline project that was the reason we started down this path wasn't funded as part of the annual planning process.

I'm going to call this task stalled for the moment since we don't have the bandwidth to work on it :(

I would like to say that if we are considering external clouds to integrate into deployment-prep we should ensure we have access to those sorted out for existing deployment-prep members and new ones going forward, before committing to anything. I don't want to end up in a situation where part of deployment-prep is only administer-able from inside the wikimedia.org google domain or something.

In T222820#5504304, @Bstorm wrote:

Beta cluster -- does that mean deployment-prep?

Yes.

Addshore subscribed.Oct 15 2019, 10:31 PM

• Mholloway subscribed.Jun 1 2020, 3:06 PM

WMDE-leszek subscribed.Oct 20 2020, 8:23 AM

thcipriani edited projects, added Release-Engineering-Team (thcipriani-workboard-fiddling); removed Release-Engineering-Team-TODO.Apr 20 2021, 3:41 AM

thcipriani moved this task from thcipriani-workboard-fiddling to Seen (ARCHIVE) on the Release-Engineering-Team board.Apr 20 2021, 3:46 AM

thcipriani edited projects, added Release-Engineering-Team; removed Release-Engineering-Team (thcipriani-workboard-fiddling).

thcipriani edited projects, added Release-Engineering-Team (Seen); removed Release-Engineering-Team.Apr 20 2021, 3:23 PM

That was as part of supporting containerized MediaWiki services on beta (T220235: Migrate Beta cluster services to use Kubernetes ). Instead of a full Kubernetes, we resorted on using one of containers. That seems to be good enough in the Beta-Cluster-Infrastructure context and save us from having to maintain a whole Kubernetes cluster. So essentially nothing to do since a solution has been found to run the services.

Experiment with hosted kubernetes solutions for BetaClosed, DeclinedPublicActions