Page MenuHomePhabricator
Feed Advanced Search

Yesterday

JMeybohm added a comment to T179696: Homepage for https://docker-registry.wikimedia.org.

The job fails on registry2002, leading to icinga alerts

Mon, Jan 25, 9:42 AM · serviceops, Patch-For-Review, SRE, MediaWiki-Containers
JMeybohm added a comment to T238909: Proposal: simplify set up of a new load-balanced service on kubernetes.

Adding https://metallb.universe.tf/ as a potential solution as well.

Mon, Jan 25, 9:01 AM · SRE, Prod-Kubernetes, Pybal, Traffic, serviceops

Fri, Jan 22

JMeybohm added a comment to T272673: Puppet failling on deploy-1002.devtools.eqiad.wmflabs.

Not completely sure (limited puppet knowledge) but maybe that kafka_config does not exist on cloudvps for some reason?!
@Ottomata I guess you're not yet using the valued generated by that code, so we could simply revert for now, right?

Fri, Jan 22, 12:17 PM · Release-Engineering-Team (Development services), Beta-Cluster-Infrastructure
JMeybohm claimed T272633: ChartMuseum responses are cached in the CDN with default (24h) ttl.

Unfortunately, upstream was not very responsive on my question about adding Cache-Control (https://github.com/helm/chartmuseum/issues/368). I wonder why this problem arised only recently as we ran this configuration from the start.

Fri, Jan 22, 7:51 AM · SRE, Traffic, serviceops

Thu, Jan 21

JMeybohm closed T242200: Docker registry needs cache to vary on Accept header value as Resolved.

The registry now responds properly with vary: Accept

Thu, Jan 21, 2:04 PM · Traffic, SRE
JMeybohm closed T271702: kubestage200* change on every puppet run, a subtask of T244335: Upgrade kubernetes clusters to a security supported (LTS) version, as Resolved.
Thu, Jan 21, 9:10 AM · Patch-For-Review, Kubernetes, Prod-Kubernetes, serviceops
JMeybohm closed T271702: kubestage200* change on every puppet run as Resolved.

With that merged, this is fixed now. Thanks @jbond !

Thu, Jan 21, 9:10 AM · Patch-For-Review, Kubernetes, Prod-Kubernetes, serviceops
JMeybohm added a comment to T272550: Upgrade docker-registry servers to Debian Buster.

I don't see anything interesting in the 2.7.1 release (https://github.com/docker/distribution/releases/tag/v2.7.1, https://metadata.ftp-master.debian.org/changelogs//main/d/docker-registry/docker-registry_2.7.1+ds2-7_changelog) so I would vote for just copying the stretch package over to buster.

Thu, Jan 21, 9:07 AM · serviceops, SRE

Mon, Jan 18

JMeybohm triaged T269461: k8s_infrastructure_users: rsyslog and echostore share the same id as Medium priority.
Mon, Jan 18, 11:30 AM · Kubernetes, Prod-Kubernetes, serviceops

Mon, Jan 11

JMeybohm renamed T271702: kubestage200* change on every puppet run from kubestage200 to kubestage200* change on every puppet run.
Mon, Jan 11, 10:47 AM · Patch-For-Review, Kubernetes, Prod-Kubernetes, serviceops
JMeybohm created T271702: kubestage200* change on every puppet run.
Mon, Jan 11, 10:46 AM · Patch-For-Review, Kubernetes, Prod-Kubernetes, serviceops

Fri, Jan 8

JMeybohm added a project to T242604: Remove obsoleted docker images: Upstream.
Fri, Jan 8, 8:31 AM · Upstream, User-brennen, SRE, Release Pipeline, Release-Engineering-Team, serviceops
JMeybohm added a comment to T271381: "envoy-tls-local-proxy" image on docker-registry.wikimedia.org has no tags.

I prefer to dupe for more context. Hope that's fine with you.

Fri, Jan 8, 7:52 AM · serviceops
JMeybohm merged T271381: "envoy-tls-local-proxy" image on docker-registry.wikimedia.org has no tags into T242604: Remove obsoleted docker images.
Fri, Jan 8, 7:51 AM · Upstream, User-brennen, SRE, Release Pipeline, Release-Engineering-Team, serviceops
JMeybohm merged task T271381: "envoy-tls-local-proxy" image on docker-registry.wikimedia.org has no tags into T242604: Remove obsoleted docker images.
Fri, Jan 8, 7:51 AM · serviceops

Thu, Jan 7

JMeybohm triaged T271422: Calico 3.17.1 kube-controllers fail to reach apiserver at startup as Low priority.
Thu, Jan 7, 2:01 PM · serviceops, Kubernetes, Prod-Kubernetes
JMeybohm created T271422: Calico 3.17.1 kube-controllers fail to reach apiserver at startup.
Thu, Jan 7, 2:01 PM · serviceops, Kubernetes, Prod-Kubernetes
JMeybohm added a comment to T271381: "envoy-tls-local-proxy" image on docker-registry.wikimedia.org has no tags.

Yeah, the image was removed (as far as thats easily possible) as of T253396

Thu, Jan 7, 8:48 AM · serviceops

Tue, Jan 5

JMeybohm claimed T269461: k8s_infrastructure_users: rsyslog and echostore share the same id.
Tue, Jan 5, 10:22 AM · Kubernetes, Prod-Kubernetes, serviceops
JMeybohm added a comment to T271142: Some Service Operations clusters apparently do not support IPv6.

As I see it these hosts do have IPv6 in DNS (and netbox).

Tue, Jan 5, 9:56 AM · IPv6, serviceops, SRE-tools

Mon, Jan 4

JMeybohm closed T270302: Restart kubernetes services on package upgrades, a subtask of T244335: Upgrade kubernetes clusters to a security supported (LTS) version, as Resolved.
Mon, Jan 4, 5:39 PM · Patch-For-Review, Kubernetes, Prod-Kubernetes, serviceops
JMeybohm closed T270302: Restart kubernetes services on package upgrades as Resolved.
Mon, Jan 4, 5:39 PM · Kubernetes, Prod-Kubernetes, serviceops
JMeybohm closed T270298: kubernetes tmpfiles references path below legacy directory /var/run/, a subtask of T244335: Upgrade kubernetes clusters to a security supported (LTS) version, as Resolved.
Mon, Jan 4, 5:39 PM · Patch-For-Review, Kubernetes, Prod-Kubernetes, serviceops
JMeybohm closed T270298: kubernetes tmpfiles references path below legacy directory /var/run/ as Resolved.
Mon, Jan 4, 5:39 PM · Kubernetes, Prod-Kubernetes, serviceops

Dec 21 2020

zeljkofilipin awarded T270270: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` a Like token.
Dec 21 2020, 2:51 PM · Patch-For-Review, Traffic, serviceops, SRE, User-zeljkofilipin, MediaWiki-Docker

Dec 18 2020

JMeybohm claimed T270302: Restart kubernetes services on package upgrades.
Dec 18 2020, 3:51 PM · Kubernetes, Prod-Kubernetes, serviceops
kostajh awarded T270270: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` a Yellow Medal token.
Dec 18 2020, 2:45 PM · Patch-For-Review, Traffic, serviceops, SRE, User-zeljkofilipin, MediaWiki-Docker
JMeybohm claimed T270298: kubernetes tmpfiles references path below legacy directory /var/run/.

I don't see those directories being used anywhere so I suggest we just remove the tmpfile registration from master and node package.

Dec 18 2020, 2:17 PM · Kubernetes, Prod-Kubernetes, serviceops
JMeybohm renamed T270063: kube-apiserver flag --admission-control has been deprecated from Flag --admission-control has been deprecated to kube-apiserver flag --admission-control has been deprecated.
Dec 18 2020, 2:05 PM · Kubernetes, Prod-Kubernetes, serviceops
JMeybohm added a comment to T244335: Upgrade kubernetes clusters to a security supported (LTS) version.

As of https://gerrit.wikimedia.org/r/c/operations/puppet/+/648356 we're now running staging-codfw with docker 18.06.3 and it looks good so far.

Dec 18 2020, 2:03 PM · Patch-For-Review, Kubernetes, Prod-Kubernetes, serviceops
JMeybohm added a comment to T207804: Upgrade Calico.

We packaged and deployed calico 3.16 to staging-codfw but currently we lack full IPv6 support due to the fact that we now require kubernetes to dual-stack/IPv6 enabled as well (as we run calico with kubernetes datastore backend).

Dec 18 2020, 1:59 PM · Prod-Kubernetes, User-fsero, serviceops, Kubernetes, SRE
JMeybohm closed T267653: Refactor calico deploy strategy as Resolved.

This is done with calico deployed now via puppet (CNI plugins and calicoctl) as well as helm3 (helmfile.d/admin_ng).
Everything is under version control and there are no catch-22's anymore during cluster bootstrapping.

Dec 18 2020, 1:56 PM · Patch-For-Review, Prod-Kubernetes, serviceops, Kubernetes, SRE
JMeybohm closed T267653: Refactor calico deploy strategy, a subtask of T207804: Upgrade Calico, as Resolved.
Dec 18 2020, 1:56 PM · Prod-Kubernetes, User-fsero, serviceops, Kubernetes, SRE
JMeybohm committed rLPRI5bfa9ce3fd07: k8s_infrastructure_users: Add system:kube-controller-manager (authored by JMeybohm).
k8s_infrastructure_users: Add system:kube-controller-manager
Dec 18 2020, 11:12 AM

Dec 17 2020

JMeybohm added a comment to T270270: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header`.

Maybe Mac sends a different user-agent? That would be fun...

Dec 17 2020, 4:41 PM · Patch-For-Review, Traffic, serviceops, SRE, User-zeljkofilipin, MediaWiki-Docker
Jdforrester-WMF awarded T270270: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` a Yellow Medal token.
Dec 17 2020, 4:38 PM · Patch-For-Review, Traffic, serviceops, SRE, User-zeljkofilipin, MediaWiki-Docker
JMeybohm claimed T242200: Docker registry needs cache to vary on Accept header value.
Dec 17 2020, 4:37 PM · Traffic, SRE
JMeybohm closed T270270: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` as Resolved.

For the record:
We where sending "Content-Type: application/vnd.docker.distribution.manifest.v2+json" as a gzip stream to the clients (hence no content-length) and docker 20.10 seems to no longer accept that but instead fail.

Dec 17 2020, 4:31 PM · Patch-For-Review, Traffic, serviceops, SRE, User-zeljkofilipin, MediaWiki-Docker
JMeybohm raised the priority of T270270: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header` from Low to High.

Also rising priority as I guess this will affect more and more users/devs when they upgrade to docker 20.10.

Dec 17 2020, 10:51 AM · Patch-For-Review, Traffic, serviceops, SRE, User-zeljkofilipin, MediaWiki-Docker
JMeybohm edited P13565 (An Untitled Masterwork).
Dec 17 2020, 10:39 AM
JMeybohm added a project to T270270: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header`: Traffic.

I was able to reproduce as well now. Seems like we are missing the "content-length" header (at least on the manifest API endpoint) if the response is *not* served from cache. P13565 can reproduce that when using "docker-registry.wikimedia.org" but not when using "docker-registry.discovery.wmnet".

Dec 17 2020, 10:35 AM · Patch-For-Review, Traffic, serviceops, SRE, User-zeljkofilipin, MediaWiki-Docker
JMeybohm edited P13565 (An Untitled Masterwork).
Dec 17 2020, 10:30 AM
JMeybohm edited P13565 (An Untitled Masterwork).
Dec 17 2020, 10:15 AM
JMeybohm edited P13565 (An Untitled Masterwork).
Dec 17 2020, 10:03 AM
JMeybohm created P13565 (An Untitled Masterwork).
Dec 17 2020, 9:58 AM
JMeybohm added a comment to T270270: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header`.

Unfortunately still unable to reproduce, even with docker & docker-engine 20.10.1 (on linux)

Dec 17 2020, 9:24 AM · Patch-For-Review, Traffic, serviceops, SRE, User-zeljkofilipin, MediaWiki-Docker

Dec 16 2020

JMeybohm added a comment to T270270: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header`.

Could you please check if you see any additional errors/hints in the docker daemon logs? The image/layers are not actually pulled by the client so a hint might be in there. You could also enable debug there (https://docs.docker.com/config/daemon/#enable-debugging) which could lead us somewhere.

Dec 16 2020, 5:38 PM · Patch-For-Review, Traffic, serviceops, SRE, User-zeljkofilipin, MediaWiki-Docker
JMeybohm added a comment to T270270: docker pull from docker-registry fails with `ERROR: missing or empty Content-Length header`.

Are we using harbor (was scanning T209271)? If so, perhaps this is the same issue as https://github.com/goharbor/harbor/issues/13740

Dec 16 2020, 5:15 PM · Patch-For-Review, Traffic, serviceops, SRE, User-zeljkofilipin, MediaWiki-Docker
JMeybohm created T270302: Restart kubernetes services on package upgrades.
Dec 16 2020, 3:30 PM · Kubernetes, Prod-Kubernetes, serviceops
JMeybohm created T270298: kubernetes tmpfiles references path below legacy directory /var/run/.
Dec 16 2020, 3:18 PM · Kubernetes, Prod-Kubernetes, serviceops
JMeybohm edited P13555 (An Untitled Masterwork).
Dec 16 2020, 1:15 PM
JMeybohm created P13555 (An Untitled Masterwork).
Dec 16 2020, 1:04 PM
JMeybohm updated the task description for T270271: Target Sources (component/kubernetes-future/source/Sources) is configured multiple times.
Dec 16 2020, 12:29 PM · Kubernetes, Prod-Kubernetes, serviceops
JMeybohm created T270271: Target Sources (component/kubernetes-future/source/Sources) is configured multiple times.
Dec 16 2020, 12:28 PM · Kubernetes, Prod-Kubernetes, serviceops

Dec 15 2020

JMeybohm added a comment to T270191: Add kubernetes 1.17+ topology annotations.

Currently we're setting those per node in hiera. Would be nice to have that automated (T229397).

Dec 15 2020, 4:21 PM · Kubernetes, Prod-Kubernetes, serviceops
JMeybohm created T270191: Add kubernetes 1.17+ topology annotations.
Dec 15 2020, 3:53 PM · Kubernetes, Prod-Kubernetes, serviceops
JMeybohm claimed T228967: Set up PodSecurityPolicies in clusters.
Dec 15 2020, 12:34 PM · Patch-For-Review, User-fsero, serviceops, Prod-Kubernetes
JMeybohm committed rLPRI143247ee80a4: k8s_infrastructure_users: Fix typo in types (authored by JMeybohm).
k8s_infrastructure_users: Fix typo in types
Dec 15 2020, 12:32 PM
JMeybohm added a comment to T249745: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable".

https://grafana.wikimedia.org/d/g0GUXaJMk/t249745?orgId=1 says that instead it might make sense to increase the memory limit. There's 3 spikes above the limit in the last 30 days, leading to extra CPU usage and probably increased GC cycles

Dec 15 2020, 8:42 AM · MW-1.36-notes (1.36.0-wmf.28; 2021-01-26), Patch-For-Review, User-brennen, serviceops, Platform Team Workboards (Clinic Duty Team), WMF-JobQueue, Wikimedia-production-error

Dec 14 2020

JMeybohm closed T270081: kube-apiserver errors for calico CRDs, a subtask of T207804: Upgrade Calico, as Resolved.
Dec 14 2020, 10:55 AM · Prod-Kubernetes, User-fsero, serviceops, Kubernetes, SRE
JMeybohm closed T270081: kube-apiserver errors for calico CRDs as Resolved.

"[...] has invalid property: anyOf" seems like a bug in apiserver and can be ignored IIUC: https://github.com/kubernetes/kubernetes/issues/90902

The message is a no-op, and means server-side apply for that custom resource falls back to "no schema" mode.

Dec 14 2020, 10:55 AM · Kubernetes, Prod-Kubernetes, serviceops
JMeybohm added a subtask for T207804: Upgrade Calico: T270081: kube-apiserver errors for calico CRDs.
Dec 14 2020, 10:22 AM · Prod-Kubernetes, User-fsero, serviceops, Kubernetes, SRE
JMeybohm removed a subtask for T244335: Upgrade kubernetes clusters to a security supported (LTS) version: T270081: kube-apiserver errors for calico CRDs.
Dec 14 2020, 10:22 AM · Patch-For-Review, Kubernetes, Prod-Kubernetes, serviceops
JMeybohm edited parent tasks for T270081: kube-apiserver errors for calico CRDs, added: T207804: Upgrade Calico; removed: T244335: Upgrade kubernetes clusters to a security supported (LTS) version.
Dec 14 2020, 10:22 AM · Kubernetes, Prod-Kubernetes, serviceops
JMeybohm created T270081: kube-apiserver errors for calico CRDs.
Dec 14 2020, 10:21 AM · Kubernetes, Prod-Kubernetes, serviceops
JMeybohm created T270063: kube-apiserver flag --admission-control has been deprecated.
Dec 14 2020, 8:52 AM · Kubernetes, Prod-Kubernetes, serviceops

Dec 11 2020

JMeybohm added a comment to T244335: Upgrade kubernetes clusters to a security supported (LTS) version.

Discussion as of today: "We messed up looking at changelogs and figure out dependencies"

Dec 11 2020, 2:25 PM · Patch-For-Review, Kubernetes, Prod-Kubernetes, serviceops

Dec 10 2020

JMeybohm added a comment to T267653: Refactor calico deploy strategy.

[puppet-private] (487bdca0) (jayme) Add calicoctl and calico-cni kubernetes users

Dec 10 2020, 3:54 PM · Patch-For-Review, Prod-Kubernetes, serviceops, Kubernetes, SRE
JMeybohm triaged T269835: Implement switching of staging clusters as Medium priority.
Dec 10 2020, 10:01 AM · Kubernetes, Prod-Kubernetes, serviceops
JMeybohm created T269835: Implement switching of staging clusters.
Dec 10 2020, 10:01 AM · Kubernetes, Prod-Kubernetes, serviceops
JMeybohm added a comment to T269581: Add Link engineering: Allow external traffic to linkrecommendation service.

Can/should it maybe be integrated into https://wikitech.wikimedia.org/wiki/API_Gateway instead of going though MW?

Dec 10 2020, 9:25 AM · Patch-For-Review, Growth-Team, serviceops, Add-Link

Dec 9 2020

JMeybohm closed T269573: Add Link engineering: Puppetize DB credentials as Resolved.

Secrets have been planted with "[puppet-private] (d2082d1e) (jayme) Add linkrecommendation db credentials".
Generated YAML looks fine (deploy1001:/srv/deployment-charts/helmfile.d/services/linkrecommendation$ helmfile -e staging template)

Dec 9 2020, 6:14 PM · serviceops, Growth-Team, Add-Link
JMeybohm added a comment to T267653: Refactor calico deploy strategy.

The new calico chart is merged, thanks @akosiaris

What is missing currently is a proper RoleBinding for the calicoctl user as I was not sure yet what permissions he's going to need.
We should be not using the tool for changing calico config, that's to be done via the helm chart now. But we will want to keep the analyze functionality intact. Could not find any docs on that by know so we will maybe just have to figure it out when we have a node in staging-codfw

That's fine, but since the tool is also used to run some diagnostics and will only be run from the kubernetes nodes by an SRE, it's probably ok to use the network-admin role that is defined in https://docs.projectcalico.org/getting-started/kubernetes/hardway/end-user-rbac

Dec 9 2020, 6:08 PM · Patch-For-Review, Prod-Kubernetes, serviceops, Kubernetes, SRE
JMeybohm added a comment to T267653: Refactor calico deploy strategy.

The new calico chart is merged, thanks @akosiaris

Dec 9 2020, 11:14 AM · Patch-For-Review, Prod-Kubernetes, serviceops, Kubernetes, SRE

Dec 7 2020

JMeybohm reopened T236017: Move blubberoid to use TLS only. as "Open".
Dec 7 2020, 1:13 PM · Patch-For-Review, Release Pipeline (Blubber), Kubernetes, serviceops, SRE
JMeybohm reopened T254581: Move termbox to use TLS only as "Open".
Dec 7 2020, 1:13 PM · Traffic, HTTPS, Patch-For-Review, Prod-Kubernetes, Kubernetes, serviceops
JMeybohm reopened T254581: Move termbox to use TLS only, a subtask of T235411: Add TLS termination to services running on kubernetes, as Open.
Dec 7 2020, 1:12 PM · Prod-Kubernetes, Kubernetes, serviceops, SRE
JMeybohm reopened T236017: Move blubberoid to use TLS only., a subtask of T235411: Add TLS termination to services running on kubernetes, as Open.
Dec 7 2020, 1:12 PM · Prod-Kubernetes, Kubernetes, serviceops, SRE
JMeybohm closed T236017: Move blubberoid to use TLS only. as Resolved.
Dec 7 2020, 1:12 PM · Patch-For-Review, Release Pipeline (Blubber), Kubernetes, serviceops, SRE
JMeybohm closed T236017: Move blubberoid to use TLS only., a subtask of T235411: Add TLS termination to services running on kubernetes, as Resolved.
Dec 7 2020, 1:11 PM · Prod-Kubernetes, Kubernetes, serviceops, SRE
JMeybohm closed T254581: Move termbox to use TLS only, a subtask of T235411: Add TLS termination to services running on kubernetes, as Resolved.
Dec 7 2020, 1:07 PM · Prod-Kubernetes, Kubernetes, serviceops, SRE
JMeybohm closed T254581: Move termbox to use TLS only as Resolved.
Dec 7 2020, 1:07 PM · Traffic, HTTPS, Patch-For-Review, Prod-Kubernetes, Kubernetes, serviceops
JMeybohm updated the task description for T254581: Move termbox to use TLS only.
Dec 7 2020, 1:06 PM · Traffic, HTTPS, Patch-For-Review, Prod-Kubernetes, Kubernetes, serviceops

Dec 4 2020

JMeybohm committed rLPRI09d85e7162a1: Add tokens for calico::kubernetes cni and ctl (authored by JMeybohm).
Add tokens for calico::kubernetes cni and ctl
Dec 4 2020, 8:16 PM
JMeybohm committed rLPRI0e40d1ecdba0: k8s_infrastructure_users: add calicoctl (authored by JMeybohm).
k8s_infrastructure_users: add calicoctl
Dec 4 2020, 7:21 PM
JMeybohm committed rLPRI90861b7e4ea4: k8s_infrastructure_users: fix type of client-infrastructure (authored by JMeybohm).
k8s_infrastructure_users: fix type of client-infrastructure
Dec 4 2020, 6:33 PM
JMeybohm committed rLPRIb67fb0016b12: Order k8s_infrastructure_users by id (authored by JMeybohm).
Order k8s_infrastructure_users by id
Dec 4 2020, 6:33 PM
JMeybohm committed rLPRI621cfa62e8bb: k8s_infrastructure_users: add calico-cni (authored by JMeybohm).
k8s_infrastructure_users: add calico-cni
Dec 4 2020, 6:33 PM
JMeybohm created T269461: k8s_infrastructure_users: rsyslog and echostore share the same id.
Dec 4 2020, 6:01 PM · Kubernetes, Prod-Kubernetes, serviceops

Dec 3 2020

JMeybohm claimed T267653: Refactor calico deploy strategy.
Dec 3 2020, 2:15 PM · Patch-For-Review, Prod-Kubernetes, serviceops, Kubernetes, SRE

Dec 2 2020

JMeybohm closed T268743: Migrate Chartmuseum (python3-docker-report) to use helm3 as Resolved.

docker-report 0.0.9 builds helm charts with helm3 now. Rolled the fix out to chartmuseum hosts (and patched puppet to install helm3 there).

Dec 2 2020, 11:18 AM · Kubernetes, serviceops, SRE
JMeybohm closed T268743: Migrate Chartmuseum (python3-docker-report) to use helm3, a subtask of T251305: Migrate to helm v3, as Resolved.
Dec 2 2020, 11:18 AM · Patch-For-Review, Kubernetes, serviceops, SRE
JMeybohm claimed T268743: Migrate Chartmuseum (python3-docker-report) to use helm3.

helm2 is not capable of packaging helm apiVersion v2 charts ofc.

Dec 2 2020, 9:04 AM · Kubernetes, serviceops, SRE

Nov 27 2020

JMeybohm added a comment to T251305: Migrate to helm v3.

Thanks @hashar
I'll take a look if what we use from the old incubator has made it to the new one.

Nov 27 2020, 3:47 PM · Patch-For-Review, Kubernetes, serviceops, SRE
JMeybohm closed T268434: Refactor our helmfile.d dir structure for admin as Resolved.
Nov 27 2020, 12:58 PM · serviceops, Kubernetes, Prod-Kubernetes
JMeybohm updated the task description for T251305: Migrate to helm v3.
Nov 27 2020, 12:57 PM · Patch-For-Review, Kubernetes, serviceops, SRE
JMeybohm added a comment to T267214: Add a link engineering: Database for link recommendation service.

@kostajh - reminder we are still waiting on knowing from where this database will be accessed.
I could grant 10.64.% or whatever, but if there's something more concrete, that'd be useful.

I unfortunately cannot answer that, but perhaps @jijiki / @JMeybohm / @akosiaris know?

Nov 27 2020, 11:13 AM · DBA

Nov 26 2020

JMeybohm added a comment to T251305: Migrate to helm v3.

Need to update/fix helm (2) to satisfy helmfile's "helm version" parser (it panics again)

Nov 26 2020, 5:21 PM · Patch-For-Review, Kubernetes, serviceops, SRE
JMeybohm updated the task description for T251305: Migrate to helm v3.
Nov 26 2020, 5:19 PM · Patch-For-Review, Kubernetes, serviceops, SRE
JMeybohm updated the task description for T251305: Migrate to helm v3.
Nov 26 2020, 4:34 PM · Patch-For-Review, Kubernetes, serviceops, SRE