Mentioned in SAL (#wikimedia-operations) [2021-02-17T10:13:23Z] <_joe_> depooling mw1331 to perform some tests for T266855

First observation I can make is that most requests are done by the math extension, and usually go in pairs:

• A POST request to /v1/media/math/check/tex
• A GET request for /v1/media/math/render/mml/<hash>

Large pages with a lot of math formulas can require up to ~ 100s of such urls in parallel (AIUI, will need to confirm from the code), so envoy allocates 100s of connections to accomodate for those.

Somehow though those connections never get closed because of idle timeout or anything.

A solution that surely works is to reduce the connections to be non-permanent, which is non-ideal, though.

Same effect was observed by adding an idle_timeout (which we don't usually add here, but has in practice more or less the same effect as removing persistent connections).

I am tempted to solve the issue "temporarily" by just adding an idle_timeout for now, and delegate the next level of fixes to future work introducing circuit breaking, or to let the problem solve itself if we finally manage to get away from using static resources and load-balancers.

Change 664791 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] P:services_proxy::envoy: add keepalive to restbase-https

https://gerrit.wikimedia.org/r/664791

Change 664655 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Revert "Revert "Switch restbase calls to be channeled via envoy""

https://gerrit.wikimedia.org/r/664655

Change 664791 merged by Giuseppe Lavagetto:
[operations/puppet@production] P:services_proxy::envoy: add keepalive to restbase-https

https://gerrit.wikimedia.org/r/664791

Change 664655 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Revert "Switch restbase calls to be channeled via envoy""

https://gerrit.wikimedia.org/r/664655