Phabricator got the HN/Twitter kiss of death this morning because of T273741 going viral. I noticed when accessing that article while logged out that our frontend caches don't cache tasks at all for logged-out users. This puts a lot unnecessary of pressure on PHP and MariaDB when a phabricator task goes viral.
Either Phabricator itself, Apache or our frontend caches should add a rule to have Phabricator tasks cached in our frontend caches for X minutes for logged-out users. People will still be able to see the latest version of the task and its comments by logging in.
According to @Krinkle this might only be solvable upstream, as Phabricator gives all viewers a session and CSRF tokens.
From 1/128 sampled requests, the traffic started at roughly midnight after https://twitter.com/chrisalbon/status/1358890731981611009 (which ironically refers to a different traffic surge). Most hits are for phab.wmfusercontent.org
The esams cache traffic got moved to eqiad which would have caused all hits from Europe to hit eqiad. Eventually Phabricator ran out of local sockets (30k limit):
Leading to issues such as can't reach its db m3-master.eqiad.wmnet failed with error #2002: Cannot assign requested address.