Current status:

The staging pod seems to run without issue and there is no log output. (Once it receives requests, we should see some output.)

[kharlan@deploy1002 ~]$ kube_env ipoid staging
[kharlan@deploy1002 ~]$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
ipoid-staging-6ff65d7df4-6pkpk   1/1     Running   0          15h
[kharlan@deploy1002 ~]$ kubectl logs -f ipoid-staging-6ff65d7df4-6pkpk
^C

curl -I https://ipoid.k8s-staging.discovery.wmnet:30443/_info 
HTTP/2 503
date: Fri, 07 Jul 2023 08:30:12 GMT
server: istio-envoy

We need more information from istio-envoy about the 503 error.

Other notes:

• docker run --rm -it -p 6927:6927 docker-registry.wikimedia.org/repos/mediawiki/services/ipoid:2023-07-06-165132-production and then curl localhost:6927/_info works as expected
• Running the production image via Minikube locally works as expected:
  • minikube start
  • kubectl create deployment ipoid --image=docker-registry.wikimedia.org/repos/mediawiki/services/ipoid:2023-07-06-165132-production
  • kubectl expose deployment ipoid --type=NodePort --port=6927
  • kubectl get services ipoid
  • minikube service ipoid

From everything I can see, the problem is in the wiring for the staging environment in Kubernetes, rather than an issue with the ipoid application code or its production image.

When the chart was created, it was missing enabling a template, which was not evident until we were able to successfully start ipoid containers in the staging environment.

values.yaml: 

mesh:
  enabled: true

After including the missing stanzas in the chart, we get:

jiji@deploy1002:~$ curl -I https://ipoid.k8s-staging.discovery.wmnet:30443/_info

HTTP/2 200
access-control-allow-origin: *
access-control-allow-headers: accept, x-requested-with, content-type
access-control-expose-headers: etag
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
content-security-policy: default-src 'self'; object-src 'none'; media-src 'none'; img-src 'none'; style-src 'none'; base-uri 'self'; frame-ancestors 'self'
content-type: application/json; charset=utf-8
content-length: 138
vary: Accept-Encoding
date: Fri, 07 Jul 2023 10:14:30 GMT
x-envoy-upstream-service-time: 4
server: istio-envoy

I updated the documentation on Wikitech for how to deploy to staging and verify that the service is working: https://wikitech.wikimedia.org/wiki/Service/IPoid#staging