Page MenuHomePhabricator
Paste P4469

IBrokeWikipediaList
ActivePublic

Authored by Krinkle on Nov 18 2016, 3:08 AM.
Tags
None
Referenced Files
F46189699: my-crash.png
Apr 12 2024, 11:02 PM
F46188840: I broke wikipedia sticker.png
Apr 12 2024, 10:43 PM
F46188591: i-broke-wikipedia.jpg
Apr 12 2024, 10:41 PM
F46185697: original-ibrokewikipedia-shirts.jpg
Apr 12 2024, 10:31 PM
F4738888: IBrokeWikipediaList
Nov 18 2016, 3:08 AM
## I broke Wikipedia... then I fixed it
bd808 was told by many people when he was hired that he wouldn't be a "real" Sr Software Engineer at WMF until he had broken production and then figured out how to fix it. This seems to be a meme among those who have survived a site outage. There is often joking on irc about how the members of this cabal should get a t-shirt. So bd808 made one: http://oi61.tinypic.com/2ryj5kz.jpg
The first 30 shirts are on order and will be handed out as soon as the shirts and the recipients are located in the same room. Hopefully at least by all-hands if not sooner.
Shirts have arrived! <https://twitter.com/bd808/status/511661882634407937>
''Note:'' Writing bad code does not qualify for a shirt. :) You need to write bad code (or make bad decisions), get multiple people to sign off on the code/decisions, deploy, watch all hell break loose and then realize how to fix the problem.
1. bd808: changed scap such that wikiversions.cdb was missing from all MW servers; 6 minutes of 404 for all sites
2. andrewbogott: typo in variable name in InitialiseSettings.php that caused all MW servers to forget how to talk to commons
3. yuvipanda: debug=true in iOS app bits url that DOS attacked varnish caches
4. marktraceur: went to update prod and add new extension and settings. Step 1: update LocalSettings.php first to point to the new files. Step 2: OHSHIOHSHIOHSHI
5. mutante: merged Apache config with rewrite rules that rewrote a bit too much, also: nginx/SSL certs update on cluster proxies
6. chad: got brion to sync my syntax errors live (from before scap linted)
7. brion: <insert best story here>
8. tim: <insert best story here>
9. reedy: this one time... on wikipedia...<insert best story here>
10. ori: <insert best story here>
11. mark b: [suggestion: Made every request a 301 redirect to the Belgian chapter wiki, then Squid happily cached the redirects]
12. faidon: <insert best story here>
13. Roan: Accidentally wrote and deployed JavaScript code that worked fine in every browser, except in IE where it crashed the browser. I swear this was an accident :) https://bugzilla.wikimedia.org/show_bug.cgi?id=20668
14. Max: killed one little symlink that served /w/ - 404s everywhere
15. James: Got Roan to deploy a teeny VisualEditor change it was "important" to get out, with unfortunately disasterous results https://wikitech.wikimedia.org/wiki/Incident_documentation/20130628-Site
16. Alex: compiled Zend php5 with the bad libicu depedency. All projects lost sorting in categories.
17. Dereckson: prepared a CS change with a l/L case typo, got CR+1, deployed it, watched en.wikipedia error, fixed it https://wikitech.wikimedia.org/wiki/Incident_documentation/20160601-MediaWiki
18. ... add as needed ...
OK, I (Chris McMahon) didn't break Wikipedia, but this is the first instance of a bug reported in beta labs that subsequently went to production and made a big mess: https://bugzilla.wikimedia.org/show_bug.cgi?id=42452. The back story is funny also, some Europeans merged the buggy change on the Thanksgiving Day US holiday. I nominate MatmaRex for this one, and almost 2 years before MatmaRex became staff.
Not signed-off:
Antoine:
typo in index.php on the NFS share, causing blank pages on all sites almost instantly
drop enwiki; # Albeit on a DB slave
Don't know if it justifies:
Matanya: https://wikitech.wikimedia.org/wiki/Incident_documentation/20140714-Lists
== First batch gone! ==
Bryan handed out the whole first batch of 30 shirts to deserving folks!
Maybe this should become an annual thing

Event Timeline

Copied from https://etherpad.wikimedia.org/p/IBrokeWikipediaList to Phabricator Paste so that changes may be easier to track and it won't get lost.

The photo of the just arrived shirts that I posted to twitter on 2014-09-15:

original-ibrokewikipedia-shirts.jpg (450×600 px, 48 KB)

I took these as luggage to the WMF all-hands meetup in San Francisco in January 2015 and handed them out to folks.

So bd808 made one: http://oi61.tinypic.com/2ryj5kz.jpg

tinypic.com has been lost to the sands of time. That photo was:

i-broke-wikipedia.jpg (287×300 px, 20 KB)

In 2016 I started making a sticker version because 1) easier to carry places, 2) no worries about not having the correct size, 3) cheaper.

I broke wikipedia sticker.png (196×196 px, 169 KB)

The 2014-04-08 scap triggered outage that "earned" me the shirt (and my Chaos Monkey badge):

my-crash.png (500×1 px, 27 KB)

The change to scap: https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/124627
The revert of the change to scap: https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/124658
The corrected version of the original change: https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/124762
See also:

Thanks to Wikipedia:Village stocks I found the link to that day's IRC logs. The interesting bits start at 2014-04-08 18:04:23. I suppose I should issue a "parental advisory, explicit content" warning to go with that link for the use of vulgar colloquialisms expressing dismay.

2014-04-08 18:12:57	<bd808|deploy>	There's my first crash all of the wikis

My favorite part of that stressful mistake was being reminded that the entire world had not ended:

2014-04-08 18:23:47	<manybubbles>	so I told my wife we broke the internet. she told me facebook was working....