Most Interesting Bugs

What makes a bug interesting? Any bug, by definition, has an element of a surprise (otherwise, it wouldn't be a bug). But not all bugs are equal in their ability to be surprising. Some bugs do stand out, presenting two essential elements of a surprise - unexpectedness and/or unusual manifestation. Based on my own classification criteria, I define a bug to be the most interesting if it is either (or both)
(1) difficult to find
(2) has a quite unexpected (mostly disruptive) impact

Let's take a look at some examples in those nominations. The first example of a bug that is "Difficult to find" - https://phabricator.wikimedia.org/T272103. The issue's has a long title "[regression - wmf.26] frwiki Homepage SE module has 'cirrussearch-query-too-long' for default filters". Even from the title two things look interesting - first, the issue is marked as a regression bug (indicating that previous testing did not find the issue) and second, it's very specific - mentions French wikipedia and a specific Cirrus error message.
To describe the issue briefly - a user visits Special:Homepage (to see the page you need to log in and to enable Homepage in User preferences), and with the default filters selection, the error would be displayed.

The surprising elements of the issue are

  • the issue happens only on French wiki - which makes it classified as "Difficult to find"
  • the issue happens for default selection (no user input) - therefore, the impact on user experience is big
  • users are given false feedback that tasks are there (the tasks count is present) - again, the impact on user experience is big since users would see misleading info.

On the positive side, the error message is super specific, and points to the exact root cause of the issue:

Fetching task suggestions failed: cirrussearch-query-too-long cirrussearch-query-too-long

"Search request is longer than the maximum allowed length. (2122 > 2048)"

Although the issue possibly could be predicted, the chances that it would happen were low. And to answer the question "why it was difficult to find?" - the testing environment (the beta cluster) did not show the issue. Only when the Suggested edits module was deployed to seven wikis and post-deployment testing was done, the issue was found.

In the same nomination - "Difficult to find" - could be the issue for articles with the template {{Italic title}} (https://phabricator.wikimedia.org/T216044) that would break ContentTranslation workflows or https://phabricator.wikimedia.org/T274619 when enabling Homepage will remove "More" menu from User page, i.e.

User page with enabled HomepageUser page - Homepage disabled
More is not present
More is present

In fact, there is a not-so-small subgroup of "Difficult to find" bugs that could be titled "Regrettable regression bugs". The subgroup includes bugs that unxpectedly appear in places that should not be affected by new changes, or the bugs that could be found only in production.

The "Unexpected impact" group has issues, arguably, the most entertaining ones. The first example would be an issue ironically involving bug images (a pure coincidence!). The animated gif below shows that selecting a bug image (Special:MediaSearch with filters shows that there is only one of image returned) and clicking on it makes the image multiply:

Another example of a bug with "Unexpected impact"- https://phabricator.wikimedia.org/T280616. On an article with suggested Add links, a user clicks on 'Edit source'. The page starts reloading and keeps going with reloading, making a page disabled for any further interactions (click on the animated gif to see the action)

The list of examples of bugs in "Difficult to find" and "Unexpected impact" nominations doesn't gets new items too often. But each issue on the list gives a sort of a lesson in exploratory testing, and, in fact, making the testing more efficient.

Written by Etonkovidova on Thu, May 27, 4:20 PM.
dev.kadirselcuk, hashar, zeljkofilipin
"Like" token, awarded by NoblyDev."Love" token, awarded by mmodell."Hungry Hippo" token, awarded by zeljkofilipin.

Event Timeline

What tool are you using to record gif files?

What tool are you using to record gif files?

I'm using https://www.macupdate.com/app/mac/49461/licecap. It's been proven to be reliable and easy to use.

I have learned a lot more about the QA day to day activities, that was insightful. Thank you @Etonkovidova !