In T306165#7941085, @Joe wrote:

Frankly I'd say we can keep this relatively simple. Building a debian package doesn't relly give us any advantage as I doubt we'd use it anywhere but in the helm-linter image.

I would propose instead you create a local repo on either gitlab or gerrit, with a script allowing to checkout just the stuff we want from upstream, then we clone it inside the image.

kubeconform debian package is ready as well (needs gerrit repo etc.) but I'm not sure about the best way to deal with the kubernetes json schema (the repo is quite big, 14GB on disk).

In T306649#7931058, @akosiaris wrote:

Regarding the "fake nodes": I think that could be done with adding the leafs as GlobalNetworkSet to the K8s/Calico API. That should make them easily selectable via peerSelectors without creating the confusion fake nodes would create.

Reading that, I am under the impression it won't work cause it only applies for Network policies. Can't hurt to try though.

Yeah. Some other docs suggest they can be used in selector fields an general, though.

In T306165#7930435, @Joe wrote:

Looking at datree, it's very interesting it can be installed as an helm plugin, so that it can be just used on helm charts from helm itself; however I don't think it can be integrated with helmfile, so we should use it as a standalone cli tool on the generated yaml anyways.

We should be aware that changing the typology annotations also changes scheduling behavior. As of now, the scheduler will try to schedule Pods of the same Replicaset across different rows (as zone contains just the row). Changing zone do include the rack will make each rack unique, so we lower the possibility of Replicatsets Pods to span multiple rows. That might have unwanted implications in case of power or network issues on one row.

Do we really think we need a global/shared cache directory? AIUI it is used to:

In T305729#7917199, @elukey wrote:

Hi folks! In T307927 I am trying to figure out why ml-team deployers (not in the deployment group) are not able to use helmfile/helm when there are charts changes, is it possible that this is due to the HELM_CACHE_HOME perm changes? The ml-team can opt-in the deployment group without issues, we didn't need it since we only deploy to the ML team's k8s clusters.

In T303049#7902555, @BTullis wrote:

I understand that it's something to do with DNS Discovery - but it seems counter-intuitive to have to refer to a read-only DNS record if the service supports writing in both DCs. Have I misunderstood something, or is it just a quirk of the setup that I have to get used to?

This is done with miscweb being the first full Ingress service and datahub following up.
Docs at https://wikitech.wikimedia.org/wiki/Kubernetes/Ingress

I finally managed to verify and document the steps needed to put a service under Ingress. I did also update the general
https://wikitech.wikimedia.org/wiki/Kubernetes/Add_a_new_service documentation (which contains a link to the Ingress specific part).
@BTullis: I'd very much like you to go over the new docs to verify those are useful to others. From what I remember datahub still needs:

• most of the DNS CNAME records (currently only datahub-gms.discovery.wmnet exists)
• service::catalog entries for datahub-frontend and datahub-gms
• to make use of datahub-frontend.discovery.wmnet in hieradata/common/profile/trafficserver/backend.yaml

This is now used by miscweb and documented at https://wikitech.wikimedia.org/wiki/Kubernetes/Add_a_new_service (more specifically https://wikitech.wikimedia.org/wiki/Kubernetes/Ingress#Add_a_new_service_under_Ingress)

In T306797#7897118, @Ottomata wrote:

@JMeybohm I wonder if the main pain points were more around the fact that the WDQS Updater is stateful. If there is no state (other than Kafka consumer offsets, which are stored in Kafka), perhaps multi DC k8s deployment won't be as difficult.

I think the use cases we are targeting atm are stateless.

In T307252#7897176, @MattCleinman wrote:

Happy to help with this however we can. So you know, this APNs key rotation isn't something we anticipate happening frequently, so we may not want to over-engineer an automated check for it. There is no expiry of the key. In this case, we wanted to rotate the key we had been using for dev before it goes to production, but from here out we don't expect to update it even once per year.

I can reproduce that when running rake run_locally['default'] on https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/787058 but not on master (a2857b739e8a1578e1b124c45d02e51cd940c363).

In T297140#7886240, @Dzahn wrote:

It says to test if everything is ok, when adding a new namespace, with:

kube_env $YOUR-SERVICE-NAME staging-codfw
kubectl get ns

When literally doing that I run into:

Error from server (Forbidden): namespaces is forbidden: User "image-suggestion" cannot list resource "namespaces" in API group "" at the cluster scope

I _know_ this happened to me back when I added miscweb and it was not actually an issue but can you remind me what the corrext fix to the docs is?

This is bad docs. The deployer Kubernetes accounts don't have (and don't need) permission to "get ns", that's why you get this error.
I've changed the docs to:

kube_env admin staging-codfw
kubectl describe ns $YOUR-SERVICE-NAME

In T288546#7882754, @Dzahn wrote:

CCing @JMeybohm based on involvement in the original setup at T256973. Does that sound reasonable?

In T306827#7881534, @dancy wrote:

In T306827#7879622, @JMeybohm wrote:

Rolled out to canaries + deploy1002, still super slow as introduced with T305949

I created T306915 for further conversation on this topic.

Please keep this open as it is absolutely in a hacky state currently (DNS + service::catalog wise)

In T305729#7856071, @dancy wrote:

• WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/mwdebug-deploy-eqiad.config is display by helmfile. There are times when it seems like it is upgrading the warning to an error, but it's unclear .

I don't think this ever gets elevated to en error (but I do get that it is annoying).

Rolled out to canaries + deploy1002, still super slow as introduced with T305949

In T305358#7854870, @akosiaris wrote:

• The monitoring: stanza can't be added as having that without lvs: breaks icinga. Can potentially be ignored (T291946), see above.

I am not sure this is true. I see helm-charts not having an lvs: stanza and still having monitoring and icinga having those services just fine.

I think this is a lucky coincidence as for both services host config is created by via monitoring::service resource (in modules/profile/manifests/chartmuseum.pp and modules/profile/manifests/releases/common.pp. As there are no "real hosts" with case of ingress services, monitoring::service resources usually don't exist.

I've removed the tags from the registry (https://wikitech.wikimedia.org/wiki/Docker-registry#Deleting_images) and triggered docker-reporter-releng-images.service

In T305949#7851774, @jnuche wrote:

Would you be ok with going ahead with the deploy? We can take a deeper look in the future if the issue persists.

In T305949#7850762, @jnuche wrote:

@JMeybohm I'll take a look. What host(s) did you run scap deploy --environment dev-cluster on?

4.6.1 rolled out to canaries. scap pullworks as usual, cd /srv/deployment/restbase/deploy/; scap deploy --environment dev-cluster feels way(!) slower then the last times I did this.

I'll prepare the needed patches.

Rolled out everywhere

Something like curl -I https://miscweb.k8s-staging.discovery.wmnet:30443 now works by default.

Basic checks on mwdebug and restbase reploy looking fine. Will roll out to fleet wide tomorrow

We skipped buster with T300744

In T304891#7823127, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2022-03-31T20:40:52Z] <mutante> reserving port 4017 for new k8s service request 'image-suggestions' T304891

In T301147#7821813, @dcausse wrote:

The additional PODs won't be used as a flink job does not automatically scale so it would be a pure waste of resources (2.5G of reserved mem per additional POD). It would help I guess to improve redundancy in this scenario only if k8s assigns every POD to a distinct machine, in which case even with a single machine misbehaving flink would have enough redundancy to allocate the job to the spare POD. If k8s does do allocation randomly or that there are not enough k8s worker nodes (1 spare POD in our case would mean spreading the PODs over 8 different machines) then it's probably not worth the waste of resources.

To be discussed with service ops:

• Investigate and address the reasons why after a node failure k8s did not fulfill its promise of making sure that the rdf-streaming-updater deployment have 6 working replicas

The problem was more that the node did not really fail (to it's complete extend). It was heavily overloaded (for an unknown reason) and that's potentially why containers/processed running there seemed dead. But from K8s perspective the Pods where still running and a new pod was scheduled as soon as I power cycled the node (e.g. K8s was able to detect a mismatch in desired end existing replicas).