Page MenuHomePhabricator
Paste P58695

Staging cloud runners k8s cluster failure
ActivePublic

Authored by dancy on Mar 8 2024, 6:58 PM.
Tags
None
Referenced Files
F42472369: Staging cloud runners k8s cluster failure
Mar 8 2024, 6:58 PM
Subscribers
None
Terraform has been successfully initialized!
module.do.digitalocean_domain.cluster_domain: Creating...
module.do.digitalocean_spaces_bucket.runner_cache: Creating...
module.do.digitalocean_kubernetes_cluster.cloud_runner: Creating...
module.do.digitalocean_domain.cluster_domain: Creation complete after 1s [id=staging.cloud.releng.team]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [10s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [10s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [20s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [20s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [30s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [30s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [40s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [40s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [50s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [50s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [1m0s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [1m0s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [1m10s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [1m10s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [1m20s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [1m20s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [1m30s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [1m30s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [1m40s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [1m40s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [1m50s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [1m50s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [2m0s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [2m0s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [2m10s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [2m10s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [2m20s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [2m20s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [2m30s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [2m30s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [2m40s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [2m40s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [2m50s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [2m50s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [3m0s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [3m0s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Still creating... [3m10s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [3m10s elapsed]
module.do.digitalocean_spaces_bucket.runner_cache: Creation complete after 3m17s [id=org-wikimedia-releng-cloud-runner-staging-cache3]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [3m20s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [3m30s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [3m40s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [3m50s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [4m0s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [4m10s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [4m20s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [4m30s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [4m40s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [4m50s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Still creating... [5m0s elapsed]
module.do.digitalocean_kubernetes_cluster.cloud_runner: Creation complete after 5m3s [id=02382cb1-844f-4108-bbcb-d5fa55c357fd]
module.do.digitalocean_kubernetes_node_pool.memory_optimized_pool: Creating...
module.do.digitalocean_project.gitlab_cloud_runner: Creating...
module.gitlab.kubernetes_namespace.this: Creating...
module.certs.kubernetes_namespace.this: Creating...
module.ingress.kubernetes_namespace.this: Creating...
module.monitoring.kubernetes_namespace.this: Creating...
module.istio.kubernetes_namespace.this: Creating...
kubernetes_secret.s3access: Creating...
kubernetes_secret.buildkit_s3access: Creating...
module.certs.helm_release.issuers: Creating...
module.istio.kubernetes_namespace.this: Creation complete after 0s [id=istio-system]
module.certs.kubernetes_namespace.this: Creation complete after 0s [id=certs]
module.gitlab.kubernetes_namespace.this: Creation complete after 0s [id=gitlab-runner]
module.ingress.kubernetes_namespace.this: Creation complete after 0s [id=ingress]
module.monitoring.kubernetes_namespace.this: Creation complete after 0s [id=monitoring]
kubernetes_config_map.coredns-custom: Creating...
kubernetes_secret.s3access: Creation complete after 0s [id=gitlab-runner/s3access]
kubernetes_secret.buildkit_s3access: Creation complete after 0s [id=gitlab-runner/buildkit-s3access]
kubernetes_config_map.coredns-custom: Creation complete after 0s [id=kube-system/coredns-custom]
module.gitlab.kubernetes_deployment.gitlab-runner-pod-cleanup: Creating...
module.certs.helm_release.this: Creating...
module.gitlab.helm_release.runner_memopt: Creating...
module.gitlab.helm_release.this: Creating...
module.ingress.helm_release.this: Creating...
module.istio.helm_release.istio-base: Creating...
module.do.digitalocean_project.gitlab_cloud_runner: Creation complete after 3s [id=00ca9f24-b029-4600-8c31-df412c7d03cc]
module.monitoring.helm_release.prometheus-adapter: Creating...
module.monitoring.helm_release.kube-prometheus-stack: Creating...
module.buildkitd.helm_release.this: Creating...
module.do.digitalocean_kubernetes_node_pool.memory_optimized_pool: Still creating... [10s elapsed]
module.gitlab.kubernetes_deployment.gitlab-runner-pod-cleanup: Still creating... [10s elapsed]
module.certs.helm_release.this: Still creating... [10s elapsed]
module.gitlab.helm_release.runner_memopt: Still creating... [10s elapsed]
module.istio.helm_release.istio-base: Creation complete after 10s [id=istio-base]
module.gitlab.helm_release.this: Still creating... [10s elapsed]
module.ingress.helm_release.this: Still creating... [10s elapsed]
module.k8s-pvc-cleaner.helm_release.this: Creating...
module.monitoring.helm_release.prometheus-adapter: Still creating... [10s elapsed]
module.monitoring.helm_release.kube-prometheus-stack: Still creating... [10s elapsed]
module.gitlab.kubernetes_deployment.gitlab-runner-pod-cleanup: Creation complete after 16s [id=gitlab-runner/gitlab-runner-pod-cleanup]
module.reggie.helm_release.this: Creating...
module.buildkitd.helm_release.this: Still creating... [10s elapsed]
module.do.digitalocean_kubernetes_node_pool.memory_optimized_pool: Still creating... [20s elapsed]
module.certs.helm_release.this: Still creating... [20s elapsed]
module.gitlab.helm_release.runner_memopt: Still creating... [20s elapsed]
module.gitlab.helm_release.this: Still creating... [20s elapsed]
module.ingress.helm_release.this: Still creating... [20s elapsed]
module.k8s-pvc-cleaner.helm_release.this: Still creating... [10s elapsed]
module.external-dns.kubernetes_secret.external-dns: Creating...
module.external-dns.kubernetes_secret.external-dns: Creation complete after 0s [id=gitlab-runner/external-dns]
module.external-dns.helm_release.external-dns: Creating...
module.monitoring.helm_release.prometheus-adapter: Still creating... [20s elapsed]
module.k8s-pvc-cleaner.helm_release.this: Creation complete after 13s [id=pvc-cleaner]
module.docker-hub-mirror.kubernetes_secret.docker-hub-mirror-creds: Creating...
module.docker-hub-mirror.kubernetes_secret.docker-hub-mirror-creds: Creation complete after 1s [id=gitlab-runner/docker-hub-mirror-creds]
module.monitoring.helm_release.kube-prometheus-stack: Still creating... [20s elapsed]
module.certs.helm_release.this: Creation complete after 26s [id=cert-manager]
module.istio.helm_release.istiod: Creating...
module.docker-hub-mirror.helm_release.docker-hub-mirror: Creating...
module.buildkitd.helm_release.this: Still creating... [20s elapsed]
module.gitlab.helm_release.runner_memopt: Creation complete after 26s [id=gitlab-runner-memopt]
module.gitlab.helm_release.this: Creation complete after 27s [id=gitlab-runner]
module.do.digitalocean_kubernetes_node_pool.memory_optimized_pool: Still creating... [30s elapsed]
module.ingress.helm_release.this: Still creating... [30s elapsed]
module.external-dns.helm_release.external-dns: Still creating... [10s elapsed]
module.monitoring.helm_release.prometheus-adapter: Still creating... [30s elapsed]
module.external-dns.helm_release.external-dns: Creation complete after 13s [id=external-dns]
module.monitoring.helm_release.kube-prometheus-stack: Still creating... [30s elapsed]
module.istio.helm_release.istiod: Still creating... [10s elapsed]
module.buildkitd.helm_release.this: Still creating... [30s elapsed]
module.istio.helm_release.istiod: Creation complete after 11s [id=istiod]
module.do.digitalocean_kubernetes_node_pool.memory_optimized_pool: Still creating... [40s elapsed]
module.ingress.helm_release.this: Still creating... [40s elapsed]
module.monitoring.helm_release.prometheus-adapter: Still creating... [40s elapsed]
module.monitoring.helm_release.kube-prometheus-stack: Still creating... [40s elapsed]
module.buildkitd.helm_release.this: Still creating... [40s elapsed]
module.do.digitalocean_kubernetes_node_pool.memory_optimized_pool: Still creating... [50s elapsed]
module.monitoring.helm_release.prometheus-adapter: Creation complete after 48s [id=prometheus-adapter]
module.ingress.helm_release.this: Still creating... [50s elapsed]
module.monitoring.helm_release.kube-prometheus-stack: Still creating... [50s elapsed]
module.buildkitd.helm_release.this: Still creating... [50s elapsed]
module.do.digitalocean_kubernetes_node_pool.memory_optimized_pool: Still creating... [1m0s elapsed]
module.ingress.helm_release.this: Still creating... [1m0s elapsed]
module.buildkitd.helm_release.this: Creation complete after 58s [id=buildkitd]
module.monitoring.helm_release.kube-prometheus-stack: Still creating... [1m0s elapsed]
module.do.digitalocean_kubernetes_node_pool.memory_optimized_pool: Still creating... [1m10s elapsed]
module.ingress.helm_release.this: Still creating... [1m10s elapsed]
module.monitoring.helm_release.kube-prometheus-stack: Creation complete after 1m6s [id=kube-prometheus-stack]
module.do.digitalocean_kubernetes_node_pool.memory_optimized_pool: Still creating... [1m20s elapsed]
module.ingress.helm_release.this: Still creating... [1m20s elapsed]
module.do.digitalocean_kubernetes_node_pool.memory_optimized_pool: Still creating... [1m30s elapsed]
module.ingress.helm_release.this: Still creating... [1m30s elapsed]
module.do.digitalocean_kubernetes_node_pool.memory_optimized_pool: Still creating... [1m40s elapsed]
module.ingress.helm_release.this: Still creating... [1m40s elapsed]
module.do.digitalocean_kubernetes_node_pool.memory_optimized_pool: Still creating... [1m50s elapsed]
module.do.digitalocean_kubernetes_node_pool.memory_optimized_pool: Creation complete after 1m51s [id=8c7fbacd-6def-4b74-9fa5-e3b681ae55f8]
module.ingress.helm_release.this: Still creating... [1m50s elapsed]
module.ingress.helm_release.this: Still creating... [2m0s elapsed]
module.ingress.helm_release.this: Still creating... [2m10s elapsed]
module.ingress.helm_release.this: Still creating... [2m20s elapsed]
module.ingress.helm_release.this: Still creating... [2m30s elapsed]
module.ingress.helm_release.this: Still creating... [2m40s elapsed]
module.ingress.helm_release.this: Creation complete after 2m48s [id=ingress-nginx]
│ Warning: Helm release "docker-hub-mirror" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
│ with module.docker-hub-mirror.helm_release.docker-hub-mirror,
│ on docker-hub-mirror/main.tf line 1, in resource "helm_release" "docker-hub-mirror":
│ 1: resource "helm_release" "docker-hub-mirror" {
│ Warning: Helm release "reggie" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
│ with module.reggie.helm_release.this,
│ on reggie/main.tf line 1, in resource "helm_release" "this":
│ 1: resource "helm_release" "this" {
│ Error: unable to build kubernetes objects from release manifest: resource mapping not found for name: "letsencrypt-production" namespace: "" from "": no matches for kind "ClusterIssuer" in version "cert-manager.io/v1"
│ ensure CRDs are installed first
│ with module.certs.helm_release.issuers,
│ on certs/main.tf line 25, in resource "helm_release" "issuers":
│ 25: resource "helm_release" "issuers" {
│ Error: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress.svc:443/networking/v1/ingresses?timeout=10s": http: server gave HTTP response to HTTPS client
│ with module.docker-hub-mirror.helm_release.docker-hub-mirror,
│ on docker-hub-mirror/main.tf line 1, in resource "helm_release" "docker-hub-mirror":
│ 1: resource "helm_release" "docker-hub-mirror" {
│ Error: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress.svc:443/networking/v1/ingresses?timeout=10s": http: server gave HTTP response to HTTPS client
│ with module.reggie.helm_release.this,
│ on reggie/main.tf line 1, in resource "helm_release" "this":
│ 1: resource "helm_release" "this" {