bd808 was told by many people when he was hired that he wouldn't be a "real" Sr Software Engineer at WMF until he had broken production and then figured out how to fix it. This seems to be a meme among those who have survived a site outage. There is often joking on irc about how the members of this cabal should get a t-shirt. So bd808 made one: http://oi61.tinypic.com/2ryj5kz.jpg
The first 30 shirts are on order and will be handed out as soon as the shirts and the recipients are located in the same room. Hopefully at least by all-hands if not sooner.
Shirts have arrived! <https://twitter.com/bd808/status/511661882634407937>
''Note:'' Writing bad code does not qualify for a shirt. :) You need to write bad code (or make bad decisions), get multiple people to sign off on the code/decisions, deploy, watch all hell break loose and then realize how to fix the problem.
1. bd808: changed scap such that wikiversions.cdb was missing from all MW servers; 6 minutes of 404 for all sites
2. andrewbogott: typo in variable name in InitialiseSettings.php that caused all MW servers to forget how to talk to commons
3. yuvipanda: debug=true in iOS app bits url that DOS attacked varnish caches
4. marktraceur: went to update prod and add new extension and settings. Step 1: update LocalSettings.php first to point to the new files. Step 2: OHSHIOHSHIOHSHI
5. mutante: merged Apache config with rewrite rules that rewrote a bit too much, also: nginx/SSL certs update on cluster proxies
6. chad: got brion to sync my syntax errors live (from before scap linted)
7. brion: <insert best story here>
8. tim: <insert best story here>
9. reedy: this one time... on wikipedia...<insert best story here>
10. ori: <insert best story here>
11. mark b: [suggestion: Made every request a 301 redirect to the Belgian chapter wiki, then Squid happily cached the redirects]
12. faidon: <insert best story here>
13. Roan: Accidentally wrote and deployed JavaScript code that worked fine in every browser, except in IE where it crashed the browser. I swear this was an accident :) https://bugzilla.wikimedia.org/show_bug.cgi?id=20668
14. Max: killed one little symlink that served /w/ - 404s everywhere
15. James: Got Roan to deploy a teeny VisualEditor change it was "important" to get out, with unfortunately disasterous results https://wikitech.wikimedia.org/wiki/Incident_documentation/20130628-Site
16. Alex: compiled Zend php5 with the bad libicu depedency. All projects lost sorting in categories.
17. Dereckson: prepared a CS change with a l/L case typo, got CR+1, deployed it, watched en.wikipedia error, fixed it https://wikitech.wikimedia.org/wiki/Incident_documentation/20160601-MediaWiki
18. ... add as needed ...
OK, I (Chris McMahon) didn't break Wikipedia, but this is the first instance of a bug reported in beta labs that subsequently went to production and made a big mess: https://bugzilla.wikimedia.org/show_bug.cgi?id=42452. The back story is funny also, some Europeans merged the buggy change on the Thanksgiving Day US holiday. I nominate MatmaRex for this one, and almost 2 years before MatmaRex became staff.
Not signed-off:
Antoine:
typo in index.php on the NFS share, causing blank pages on all sites almost instantly