The idea is to develop a maintenance script that simulates a simple web request (by using FauxRequest and index.php's MediaWiki class) and we require that script to pass without notices, errors or exceptions before syncing files.
It could be run on tin, e.g. sync to local `/srv/mediawiki` on tin first, run the script and then continue sync if successful.
This would eliminate a large class of errors:
* Subtle errors in functional syntax (not strict parse error, but fatal on runtime, such as the infamous `arrray()` typo.
* PHP notices or warnings caused in unconditional code paths (e.g. a mistyped variable in wmf-config, or in global MediaWiki code).
* Exceptions that happen on every page view.
* Files synced in the wrong order.
The core idea is already implemented as <https://github.com/wikimedia/mediawiki-extensions-WikimediaMaintenance/blob/master/sanityCheck.php>, we can further develop it by also catching errors and exceptions. It'll also need updating since the sanity check "Served by" is probably outdated.
A basic set of checks, run once, against the staging server (e.g. tin) would catch 99% of cases where a PHP notice or fatal error happens on every page load. A more elaborate canary deploy process can be developed later to catch less common issues relating to specific pages or due to load (e.g. gradually roll out based on monitoring http responses and error logs).
Also note that the benefit of a pre-check script as proposed here, also helps catch when files are synced in the wrong order. When manually testing on a canary server, this is usually not spotted because one would use `scap pull` on the canary server (syncing all files). Whereas a pre-check script would be run on every sync operation (between syncing to `/srv/mediawiki` on staging host and app servers)