Phabricator got the HN/Twitter kiss of death this morning because of T273741 going viral. I noticed when accessing that article while logged out that our frontend caches don't cache tasks at all for logged-out users. This puts a lot unnecessary of pressure on PHP and MariaDB when a phabricator task goes viral.
Either Phabricator itself, Apache or our frontend caches should add a rule to have Phabricator tasks cached in our frontend caches for X minutes for logged-out users. People will still be able to see the latest version of the task and its comments by logging in.
According to @Krinkle this might only be solvable upstream, as Phabricator gives all viewers a session and CSRF tokens.
From 1/128 sampled requests, the traffic started at roughly midnight after https://twitter.com/chrisalbon/status/1358890731981611009 (which ironically refers to a different traffic surge). Most hits are for phab.wmfusercontent.org
{F34097833}
The esams cache traffic got moved to eqiad which would have caused all hits from Europe to hit eqiad. Eventually Phabricator ran out of local sockets (30k limit):
{F34097835}
Leading to issues such as `can't reach its db m3-master.eqiad.wmnet failed with error #2002: Cannot assign requested address.`