Currently ats-be doesn't enforce a timeout for backend servers, it will wait indefinitely till the other side (usually Envoy) chooses to give up waiting (after 203 seconds, more details on https://wikitech.wikimedia.org/wiki/HTTP_timeouts#App_server):
vgutierrez@cp3064:~$ sudo -i traffic_ctl config match active_timeout_out proxy.config.http.transaction_active_timeout_out: 0
This is effectively triggering the default timeout as ATS 9 is reporting on the new proxy.process.net.default_inactivity_timeout_applied metric:
vgutierrez@cp3064:~$ sudo -i traffic_ctl config match default_inacti proxy.config.net.default_inactivity_timeout: 360 vgutierrez@cp3064:~$ sudo -i traffic_ctl metric match default_inactivity proxy.process.net.default_inactivity_timeout_applied 2438573 proxy.process.net.default_inactivity_timeout_count 6
Setting a proxy.config.http.transaction_active_timeout_out value aligned with the rest of timeouts in our stack should help under heavy load scenarios