Today, we've seen an unusual number of canary event produce failures in the data-engineering-alerts mailing list. Investigation of logs shows that the failures happen occasionally only to eventgate-logging-external.
I can reproduce this my manually curling the /v1/_test/events endpoint. Usually this succeeds, but every 4 or 5th try hangs, and then fails with a HTTP 503:
$ curl -i https://eventgate-logging-external.svc.eqiad.wmnet:4392/v1/_test/events HTTP/1.1 503 Service Unavailable content-length: 95 content-type: text/plain date: Tue, 11 Apr 2023 18:15:32 GMT server: production-tls upstream connect error or disconnect/reset before headers. reset reason: connection termination
I thought that perhaps this was a single pod having an issue, so I curled each of the pod container ports individually:
$ kube_env eventgate-logging-external eqiad $ for ip in $(kubectl get pods -o wide | awk '{print $6}' | grep -v IP); do url=https://$ip:4392/v1/_test/events; echo "== $url == "; curl -k -i $url; echo " "; done
I couldn't find any pattern. Each time I ran this, they might all succeed, or I might encounter a hanging/failed request from any pod IP.
Ah, but I went to look at container logs:
kubectl logs -f -l app=eventgate -c eventgate-logging-external | jq .
I see a lot of errors like:
"msg": "event 1d4ad7f5-992a-41b3-a658-abc2d10dfdb3 of schema at /w3c/reportingapi/network_error/1.0.0 destined to stream w3c.reportingapi.network_error encountered an error: message timed out",
Something is fishy with eventgate-logging-external for sure.