Thus far we can see an increase of 404s that lead to 504s

Please see: https://grafana.wikimedia.org/dashboard/db/aqs-elukey?orgId=1&from=1521543814143&to=1521544407591

In order to cache 404s in Varnish for a short period of time we'd need to add some config setting to hyperswitch and use that setting in AQS. RESTBase will just proxy whatever cache-control AQS will set.

I think we need to investigate the effects of 404 in api, caching those for a short time will not stop the effects of a storm of 404s but it will prevent re-running queries we know are going to return 404 if data is -for example- delayed.

Great analysis @elukey !

Would it make sense to increase on Restbase the ephemeral ports available and/or enable tw_reuse?

Yes, we should definitely do that. I am not sure how these two (especially the latter) escaped us earlier.

TCP tw-reuse has been rolled out to all the restbase hosts via https://gerrit.wikimedia.org/r/#/c/421901. Next week we'll enable connection re-use between rb <-> aqs as test.

https://github.com/wikimedia/restbase/pull/984 was created to enable TCP conn reuse between rb and aqs. Once enabled and tested that nothing is on fire, we can close the task.

After a chat with @JAllemandou, we agreed that it would be best also to apply the same config from AQS to Druid, so hyperswitch will try to re-use conns on every "level" (rb -> aqs -> druid).

In T190213#4133926, @elukey wrote:

After a chat with @JAllemandou, we agreed that it would be best also to apply the same config from AQS to Druid, so hyperswitch will try to re-use conns on every "level" (rb -> aqs -> druid).

+1. Note that for that you will need the latest version of preq.

@JAllemandou whenever you have time, let's discuss about this :)

After a chat with the Services team, preq needs to contain https://github.com/wikimedia/preq/pull/31, so as far as I can see 0.5.4+ (we are currently running 0.5.2).

After some patches we ended up with:

Merged and deployed in AQS labs. I configured the environment to use a "fake" localhost Druid cluster, and created a simple python http webserver to log POST queries. This is what I registered via tcpdump for the swagger edit tests:

POST /druid/v2/ HTTP/1.1
content-type: application/json
x-request-id: 983418b4-6ee4-11e8-a969-9f1bc9f00305
user-agent: HyperSwitch
x-client-ip: ::ffff:127.0.0.1
host: localhost:8082
content-length: 595
Connection: keep-alive

I did the same test on aqs1004, this time setting tcpdump to get all traffic to port 8082:

POST /druid/v2/ HTTP/1.1
content-type: application/json
x-request-id: 4c54dc04-585e-11e8-bfc7-b7b8a8526ccf
user-agent: HyperSwitch
x-client-ip: ::ffff:10.64.0.107
host: druid-public-broker.svc.eqiad.wmnet:8082
content-length: 595
Connection: keep-alive

So the keep alive seems already working! Is it a default setting for HyperSwitch ?

After a chat with Services we discovered that the request module sets keep alive by default for HTTP connections :)

Mentioned in SAL (#wikimedia-operations) [2018-06-13T12:57:28Z] <elukey@deploy1001> Started deploy [analytics/aqs/deploy@84fab89]: Update AQS for T190213

Mentioned in SAL (#wikimedia-operations) [2018-06-13T13:00:06Z] <elukey@deploy1001> Finished deploy [analytics/aqs/deploy@84fab89]: Update AQS for T190213 (duration: 02m 38s)

Change 440120 had a related patch set uploaded (by Elukey; owner: Elukey):
[analytics/aqs/deploy@master] Remove old/redundant git_server config

https://gerrit.wikimedia.org/r/440120

Change 440120 merged by Elukey:
[analytics/aqs/deploy@master] Remove old/redundant git_server config

https://gerrit.wikimedia.org/r/440120