• imported calico 3.17.0 packages into component/calico-future for stretch-wikimedia
  • calico-cni
  • calicoctl
  • calico-images
• pushed docker images:
  • docker-registry.discovery.wmnet/calico/kube-controllers:v3.17.0
  • docker-registry.discovery.wmnet/calico/node:v3.17.0
  • docker-registry.discovery.wmnet/calico/typha:v3.17.0

In T268612#6644782, @akosiaris wrote:

Couple of reasons:

In T268612#6644662, @Joe wrote:

In T268612#6644659, @JMeybohm wrote:

Ouch.

Could we normalize everything to use the public image reference? That would also make local test more easy or straight forward.
Or do we gain a bit benefit by using the internal reference?

The point is consistency. We want to use the same registry when referencing images and saving them.

Sure. My question is more like: Why are did we start using both names in first place and can we stop doing so. :)

Could we normalize everything to use the public image reference? That would also make local test more easy or straight forward.
Or do we gain a bit benefit by using the internal reference?

I did a bit of testing and it looks as if it is totally possible to switch helmfile.d/admin to use helm3 and get rid of tiller there (e.g. catch-22) while keeping helm2 + tiller for helmfile.d/services for now (see T268434).

@akosiaris and me discussed this further and we initially decided to give the kubernetes addon-manager a try for rolling out calico components to a freshly bootstrapped cluster. I created a binary package from kubernetes source package that sets up the addon-manager and added corresponding puppet code.

After discussing with @akosiaris we decided to keep building the calico-images package but only use it as kind of artifact and a way to get the images out of the pbuilder environment. After building, the package can be extracted and the images imported to the registry via a script added to the package.
While not ideal, it's an okay solution for now and I've documented it at: https://wikitech.wikimedia.org/wiki/Calico

To solve the catch-22 we could deploy the to-be calico helm chart via helm3. Which would require us to invest into helm3 integration earlier than we hoped for (at least for the helmfile.d/admin part).

In T266893#6605404, @JMeybohm wrote:

When deployed as cluster addon, we can bypass all this and have mandatory components of our stack deployed/reconceiled directly when the cluster is set up (shipping manifests via debian packages or generate via puppet).
While that sounds appealing to me, I'm not sure this is still the "correct" way. The upstream addons are called legacy for a couple of years now (https://github.com/kubernetes/kubernetes/commit/43276035734ba8a5914977b13da86ec2548fa745) ....

I've crafted a debian package similar to how the k8s packages are not build (packages calico-cni and calicoctl as we currently have).

1.16.15 released to stretch-wikimedia component/kubernetes-future

In T266893#6605376, @Joe wrote:

As for the manifests, if we need them, they should be in hellfile.d/admin I guess?

[WIP] will continue shortly

In T244335#6600895, @akosiaris wrote:

Decide decide if we are going to be staying with direct access to etcd (version 3?) or try and switch to the kubernetes APIs

The big issues we currently have with the etcd backed datastore is that the information in there is not tracked anyway. It is backed up but fully unsearchable. So we definitely want to at least test using the API.

In T244335#6600895, @akosiaris wrote:

We are not able to go 1.19 because of calico only supporting 1.18

Looks like this isn't true. Judging from https://github.com/projectcalico/calico/commit/21a45a4a141fff03b251fde2f1ab77fbb0c903ee#diff-f386c272afd3d855bf9f1d3609d1782962951258a58e2b298df60c70b16517ee, the calico 3.16 requirements page will be updated soon.

In T266893#6594832, @Joe wrote:

I typically prefer if we rebuild images from dockerfiles, using our base images. That gives us a tad more control over upgrading in case of a disaster security hole in e.g. alpine linux.

An additional alternative could be to switch to a build without docker like the debian upspream does. That would maybe require us to backport golang-1.15 but would remove the build-dependency to docker with all it's space and network requirements.

Unfortunately, the current version of calico is not tested against kubernetes 1.19.x yet (https://docs.projectcalico.org/getting-started/kubernetes/requirements#kubernetes-requirements).
While it might still work, we should stick to a tested/supported version (at least for production) and there is no release timeline I found for k8s 1.19 support in calico. So no idea if we get a calico version supporting 1.19 anytime "soon".

In T266766#6591595, @akosiaris wrote:

Unfortunately there are no signed releases of k8s as of now (https://github.com/kubernetes/release/issues/914) so the best we could get is the binary tar balls plus sha512 from the official GCS bucket and package those.

So relying on e.g. https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#downloads-for-v11810 (from what I see the server binaries tar is enough as it contains whatever the node and client tars have) and some thin debian/rules to fetch/extract and put in the right place before building the deb so that we don't have to change our puppet code as well? That could work.

With Kubernetes 1.19.3 things changed a bit and we now need a docker version supporting the --platform flag for FROM.
The envoy builder host would support that but lacks enough space on /var/lib/docker to hold all the intermediate images used for build.

We're fine here.

Eventrouter is deployed to all clusters now.

Two more eventrouter patches in I must say I'm a bit disappointed by my decision to go with that one but I *think* it should be good now. Will revisit over the week to see if duplicate events are gone, re-check on resources etc.

I've changed the field names to be more specific so events are indexed now.

Looks way better now, even under higher load.

My current plan is to build a kubeval deb and add a git repo with the needed kubernetes api schema. I think we don't need fancy auto-upgrade stuff for the schema repo as we don't upgrade k8s that often. I will add a script like the above, so that we can just run that locally and commit the new kubernetes version schema to the repo once we plan to upgrade to a new version (or once we want to test for compatibility with a new version).

In T262675#6574444, @JMeybohm wrote:

Unfortunately it looks as if the logging pipeline does not parse the output of eventrouter by default:
https://logstash-next.wikimedia.org/goto/d8b98b06cbe6f8089e48c090f479bfc9

Unfortunately it looks as if the logging pipeline does not parse the output of eventrouter by default:
https://logstash-next.wikimedia.org/goto/d8b98b06cbe6f8089e48c090f479bfc9

In T266194#6570986, @akosiaris wrote:

That's pretty interesting, there shouldn't be so much throttling at so low CPU usage. user+system summed barely hit 1/5 of the limit.

+1 to bumping the limit to see if it would solve latency issues, but it might be indeed related to T262527

I think you are right, thanks for the heads up!

I fiddled with this a bit and I it is possible to use a local version/checkout of the schema which we can also generate ourselves with something like:

In T265324#6567213, @Joe wrote:

Oh, well. Sorry then. I guess I just misread the "one image configured to manage a php-fpm application." part as already containing fpm.

While it looks like kubeval basically works and we can easily replace kubeyaml with it, it has the disadvantage of relying on https://kubernetesjsonschema.dev (https://github.com/instrumenta/kubernetes-json-schema) to fetch the schema from so we would need to include a copy of that in the CI image and run like:
kubeval -v 1.11.0 -s file:///tmp/kubernetes-json-schema/

If I got this right you are purposing to put apache and php-fpm in the same container, correct (talking about vhosts in fpm context)?
I can think of reasons why that might make sense to do (sharing "static" assets for example, ease of MVP), but maybe you could outline them here?

Quick chat in IRC turned out that we don't have a "good for kubernetes" way to discover the kafka brokers (like DNS SRV records) producing directly to kafka-logging would require some coupling with puppet code/re-deployments on changes to kafka-logging brokers (which is obviously bad).

In T258572#6562150, @Ottomata wrote:

Ok! Done.