Page MenuHomePhabricator

Migrate EventStreams to k8s deployment pipeline
Open, MediumPublic

Details

Related Gerrit Patches:

Event Timeline

Ottomata created this task.Tue, Nov 19, 2:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Nov 19, 2:54 PM

Change 551838 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[integration/config@master] Setup eventstreams service-pipeline-test-and-publish

https://gerrit.wikimedia.org/r/551838

Change 551843 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] [WIP] eventstreams chart

https://gerrit.wikimedia.org/r/551843

Ottomata added subscribers: thcipriani, akosiaris, Joe.EditedTue, Nov 19, 4:42 PM

I can't seem to develop this locally due to a barrage of version problems.

From what I can tell, in prod we use.

k8s 1.12.9
helm 2.12.2
helmfile 0.66.0 2.

To use minikube with k8s 1.12.9, I tried installing minikube 1.1.0 (released at about the same time as 1.12.9).

I failed making this combo work locally (macOS). I think perhaps my Docker Desktop (with hyperkit?) has been keeping itself up to date, and no longer works with older kubectl? Not sure.

Anyway, ok so I tried updating the k8s components. Now I've got k8s v1.16.3, minikube v1.5.2, and helm v2.16.0. Helm 2.16.0 has k8s 1.16 support, but isn't Helm 3 which won't work at all with our Helm charts. Finally minikube is running. I don't know if these versions will ultimately let me use helm to deploy charts into my minikube, but at least i've got this far.

Agh!, but I guess I updated blubber in there somewhere and now my blubber v3 .pipeline files don't work? Ok fine whatever, I'm making a new k8s service here let's focus. I can use Blubber v4.

However, docker build now fails on apt-get update:

E: Failed to fetch http://mirrors.wikimedia.org/debian/dists/stretch/main/binary-amd64/Packages  404  Not Found

This seems just to be an issue with using the same hyperkit vm that minikube is using to build docker images. If I use my Docker Desktop install, it gets a bit farther. I think I'm stuck on a node binary build problem with node-gyp and node-rdkafka, but I'm not really sure yet.

So, yessome of this is a problem with my blubberfile / chart, but much is also wrong with deployment pipeline versions and docs. I've spent several hours trying to get versions in line just so I can develop this! I'm going to stop for now until I get some help or we update some wikitech Deployment Pipeline docs about how to develop charts. The existing documentation is really just under Alex's user page, and the links to install there don't work with our current version of helm/k8s in production.

Should I be worried about these?
https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/

Ping @thcipriani @akosiaris @Joe.

Change 551838 merged by jenkins-bot:
[integration/config@master] Setup eventstreams service-pipeline-test-and-publish

https://gerrit.wikimedia.org/r/551838

Change 551914 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[integration/config@master] Correct eventstreams repo name to mediawiki/services/eventstreams

https://gerrit.wikimedia.org/r/551914

Change 551914 merged by jenkins-bot:
[integration/config@master] Correct eventstreams repo name to mediawiki/services/eventstreams

https://gerrit.wikimedia.org/r/551914

Mentioned in SAL (#wikimedia-releng) [2019-11-20T16:37:18Z] <James_F> Zuul: Add CI for mediawiki/services/eventstreams T238658

fdans triaged this task as Medium priority.Thu, Nov 21, 5:48 PM
fdans moved this task from Incoming to Operational Excellence on the Analytics board.

I can't seem to develop this locally due to a barrage of version problems.
From what I can tell, in prod we use.
k8s 1.12.9
helm 2.12.2
helmfile 0.66.0 2.
To use minikube with k8s 1.12.9, I tried installing minikube 1.1.0 (released at about the same time as 1.12.9).
I failed making this combo work locally (macOS). I think perhaps my Docker Desktop (with hyperkit?) has been keeping itself up to date, and no longer works with older kubectl? Not sure.

Doubtful that you need minikube if you have Docker Desktop. It already provides kubernetes support, it's a couple of clicks to enable it. See https://docs.docker.com/docker-for-mac/#kubernetes. I have no idea if the 2 envs would interfere with each other, I sure hope not.

Anyway, ok so I tried updating the k8s components. Now I've got k8s v1.16.3, minikube v1.5.2, and helm v2.16.0. Helm 2.16.0 has k8s 1.16 support, but isn't Helm 3 which won't work at all with our Helm charts. Finally minikube is running. I don't know if these versions will ultimately let me use helm to deploy charts into my minikube, but at least i've got this far.

1.16.3 isn't currently supported though yet by our charts. There's a number of deprecations/removals of API versions/resources that we need first to address.

Agh!, but I guess I updated blubber in there somewhere and now my blubber v3 .pipeline files don't work? Ok fine whatever, I'm making a new k8s service here let's focus. I can use Blubber v4.
However, docker build now fails on apt-get update:

E: Failed to fetch http://mirrors.wikimedia.org/debian/dists/stretch/main/binary-amd64/Packages  404  Not Found

I don't think we publish Packages, but rather Packages.gz and Packages.xz. Weird that apt-get update required that.

This seems just to be an issue with using the same hyperkit vm that minikube is using to build docker images. If I use my Docker Desktop install, it gets a bit farther. I think I'm stuck on a node binary build problem with node-gyp and node-rdkafka, but I'm not really sure yet.
So, yessome of this is a problem with my blubberfile / chart, but much is also wrong with deployment pipeline versions and docs. I've spent several hours trying to get versions in line just so I can develop this! I'm going to stop for now until I get some help or we update some wikitech Deployment Pipeline docs about how to develop charts. The existing documentation is really just under Alex's user page, and the links to install there don't work with our current version of helm/k8s in production.

Yes, known problem that we 've been trying to address for some time unsuccessfully.

Should I be worried about these?
https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/

No not really, we are going to wrap them all up for all charts right after we upgrade to 1.13. But do avoid 1.16 for now. It should make your life easier.

Ping @thcipriani @akosiaris @Joe.