HomePhabricator

Reduced the DOS potential of 404 page floods

Description

Reduced the DOS potential of 404 page floods

  • showMissingArticle() will now only show the log excerpt if the page was deleted very recently or if the viewing user is logged in. Crawlers and poorly written bots tend to be logged out, and logged in users are better tracked. If many hits to the same recently deleted page are made, then the innoDB buffer pool should actually have the relevant index pages in cache (especially with query grouping).
  • There have been several outages or DB performance problems caused by crawlers or sloppy mirrors hitting many bogus pages (due to bad URL generation, ect...). Previously a redis bloom filter was used to handle this, but was removed due to high complexity (especially if it was to be supported in multi-DC setup). This is a simpler solution to keep 404 pages cheap, as they cannot really be cached (there are innumerable possible titles that never existed).

Change-Id: If948602a32deb16dba21d232d0c6128568a980d6