Most Interesting Bugs

What makes a bug interesting? Any bug, by definition, has an element of a surprise (otherwise, it wouldn't be a bug). But not all bugs are equal in their ability to be surprising. Some bugs do stand out, presenting two essential elements of a surprise - unexpectedness and/or unusual manifestation. Based on my own classification criteria, I define a bug to be the most interesting if it is either (or both)
(1) difficult to find
(2) has a quite unexpected (mostly disruptive) impact

Let's take a look at some examples in those nominations. The first example of a bug that is "Difficult to find" - https://phabricator.wikimedia.org/T272103. The issue's has a long title "[regression - wmf.26] frwiki Homepage SE module has 'cirrussearch-query-too-long' for default filters". Even from the title two things look interesting - first, the issue is marked as a regression bug (indicating that previous testing did not find the issue) and second, it's very specific - mentions French wikipedia and a specific Cirrus error message.
To describe the issue briefly - a user visits Special:Homepage (to see the page you need to log in and to enable Homepage in User preferences), and with the default filters selection, the error would be displayed.

SE_module_initial_state2.gif (917×683 px, 349 KB)

The surprising elements of the issue are

  • the issue happens only on French wiki - which makes it classified as "Difficult to find"
  • the issue happens for default selection (no user input) - therefore, the impact on user experience is big
  • users are given false feedback that tasks are there (the tasks count is present) - again, the impact on user experience is big since users would see misleading info.

Screen Shot 2021-05-26 at 12.44.27 PM.png (1×1 px, 531 KB)

On the positive side, the error message is super specific, and points to the exact root cause of the issue:

Fetching task suggestions failed: cirrussearch-query-too-long cirrussearch-query-too-long

"Search request is longer than the maximum allowed length. (2122 > 2048)"

Although the issue possibly could be predicted, the chances that it would happen were low. And to answer the question "why it was difficult to find?" - the testing environment (the beta cluster) did not show the issue. Only when the Suggested edits module was deployed to seven wikis and post-deployment testing was done, the issue was found.

In the same nomination - "Difficult to find" - could be the issue for articles with the template {{Italic title}} (https://phabricator.wikimedia.org/T216044) that would break ContentTranslation workflows or https://phabricator.wikimedia.org/T274619 when enabling Homepage will remove "More" menu from User page, i.e.

User page with enabled HomepageUser page - Homepage disabled
More is not present
Screen Shot 2021-02-12 at 1.10.18 PM.png (394×1 px, 51 KB)
More is present
Screen Shot 2021-02-12 at 1.12.26 PM.png (397×1 px, 49 KB)

In fact, there is a not-so-small subgroup of "Difficult to find" bugs that could be titled "Regrettable regression bugs". The subgroup includes bugs that unxpectedly appear in places that should not be affected by new changes, or the bugs that could be found only in production.

The "Unexpected impact" group has issues, arguably, the most entertaining ones. The first example would be an issue ironically involving bug images (a pure coincidence!). The animated gif below shows that selecting a bug image (Special:MediaSearch with filters shows that there is only one of image returned) and clicking on it makes the image multiply:

Screen Shot 2021-01-25 at 3.33.56 PM.png (501×1 px, 318 KB)

hamar_daban_kafer2.gif (737×1 px, 3 MB)

Another example of a bug with "Unexpected impact"- https://phabricator.wikimedia.org/T280616. On an article with suggested Add links, a user clicks on 'Edit source'. The page starts reloading and keeps going with reloading, making a page disabled for any further interactions (click on the animated gif to see the action)

infinite_loop.gif (659×1 px, 481 KB)

The list of examples of bugs in "Difficult to find" and "Unexpected impact" nominations doesn't gets new items too often. But each issue on the list gives a sort of a lesson in exploratory testing, and, in fact, making the testing more efficient.

Written by Etonkovidova on May 27 2021, 4:20 PM.
dev.kadirselcuk, hashar, zeljkofilipin
"Like" token, awarded by NoblyDev."Love" token, awarded by mmodell."Hungry Hippo" token, awarded by zeljkofilipin.

Event Timeline

What tool are you using to record gif files?

What tool are you using to record gif files?

I'm using https://www.macupdate.com/app/mac/49461/licecap. It's been proven to be reliable and easy to use.

I have learned a lot more about the QA day to day activities, that was insightful. Thank you @Etonkovidova !

In J242#3754, @hashar wrote:

I have learned a lot more about the QA day to day activities, that was insightful. Thank you @Etonkovidova !

Thank you! It's really great to hear that my blog post is helpful.