Change Details

Looking at the request [[ https://grafana-rw.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater?forceLogin&from=now-3d&orgId=1&refresh=5m&to=now | durations ]], fetching responses for wikidata tends to be slow (> 5s for the 50% quartile). To avoid unnecessary retries, we could increase the timeout to 7s to allow slow responses, especially, since wikidata is responsible for 60% of the non-rerender fetches. Retry logic is spread across multiple levels: envoy, http-client and flink's async operator. For the http-client it has been disabled explicitly, however, envoy still retries on 5xx upstream responses. Since envoy's retries are not transparent to the application, we might run into timeouts and loose the actual cause (5xx error). Therefore, we should use `mw-api-int-async` (w/o retries) instead of `mw-api-int-async-ro`, [[ https://gerrit.wikimedia.org/g/operations/puppet/+/refs/heads/production/hieradata/common/profile/services_proxy/envoy.yaml#302 | see envoy config ]]. AC: * client: fetch timeout is 7s * envoy: `mw-api-int-async` listener

Looking at the request [[ https://grafana-rw.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater?forceLogin&from=now-3d&orgId=1&refresh=5m&to=now | durations ]], fetching responses for wikidata tends to be slow (> 5s for the 50% quartile). To avoid unnecessary retries, we could increase the timeout to 7s to allow slow responses, especially, since wikidata is responsible for 60% of the non-rerender fetches. Retry logic is spread across multiple levels: envoy, http-client and flink's async operator. For the http-client it has been disabled explicitly, however, envoy still retries on 5xx upstream responses. Since envoy's retries are not transparent to the application, we might run into timeouts and loose the actual cause (5xx error). By passing a header `x-envoy-max-retries: 0` ([[ https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/router_filter#x-envoy-max-retries | see docs ]]) envoy won't retry automatically. AC: * client: fetch timeout is 7s * client: pass `x-envoy-max-retries: 0` header

Looking at the request [[ https://grafana-rw.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater?forceLogin&from=now-3d&orgId=1&refresh=5m&to=now | durations ]], fetching responses for wikidata tends to be slow (> 5s for the 50% quartile). To avoid unnecessary retries, we could increase the timeout to 7s to allow slow responses, especially, since wikidata is responsible for 60% of the non-rerender fetches. Retry logic is spread across multiple levels: envoy, http-client and flink's async operator. For the http-client it has been disabled explicitly, however, envoy still retries on 5xx upstream responses. Since envoy's retries are not transparent to the application, we might run into timeouts and loose the actual cause (5xx error). Therefore, we should use `mw-api-int-async` (w/o retries) instead of `mw-api-int-async-ro`, [[ https://gerrit.wikimedia.org/g/operations/puppet/+/refs/heads/production/hieradata/common/profile/services_proxy/envoy.yaml#302 | see envoy config ]]. By passing a header `x-envoy-max-retries: 0` ([[ https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/router_filter#x-envoy-max-retries | see docs ]]) envoy won't retry automatically. AC: * client: fetch timeout is 7s * envoy: `mw-api-int-async` listener* client: pass `x-envoy-max-retries: 0` header