I've seen it both in toolsbeta and tools, when the certs get renewed the pods for the builds-api and envvars-api don't get restarted.
That ends up on the gateway returing 502:
<html> <head><title>502 Bad Gateway</title></head> <body> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx/1.21.0</center> </body> </html>
As it's getting invalid certs from the upstream:
2023/09/12 08:54:04 [error] 23#23: *134120 upstream SSL certificate verify error: (10:certificate has expired) while SSL handshaking to upstream, client: 192.168.247.64, server: , request: "POST /builds/v1/build HTTP/1.1", upstream: "https://10.98.19.138:8443/v1/build", host: "api.svc.tools.eqiad1.wikimedia.cloud:30003"
There might be something wrong in the way we set the certs or the labels for the autorestart or something :/