The issue turned out to be a single line of code that was returning false, when it should be returning true:
https://github.com/wikimedia/apps-android-wikipedia/pull/6184/files#diff-686125b80591056ae47c1f9099ffaf0c9ac421c81148cbb62b4e4de1114c7a0eR778

This caused existing users of the app (actually users who've had it installed before ~June) to not see the CTA for year-in-review.

The problem

We have a preference in our code that's literally called isYearInReviewEnabled, which should default to true. The reason it was defaulting to false is that it was introduced many months ago, when the initial iteration of YiR was first built during our internship program, and it made perfect sense to keep the feature "disabled" for the time being, until we pick it back up.

The reason this default state was forgotten about is:
We have another place where the "defaults" for our preferences are applied, namely the XML file that lays out the preferences, as seen in the actual Settings screen in the app. This XML file contains a defaultValue for each preference, which has the correct value of true for the YiR preference.

These XML defaults become applied implicitly whenever the user visits PageActivity (an extremely legacy bit of code whose purpose is no longer even apparent), which gave us developers a false sense that the default for this preference is true.

Now, you might ask, what's the problem -- if a regular user visits PageActivity, wouldn't the XML defaults become applied for them, too? Actually.... no! Not if the user already had the app installed. And not just "already had it installed", but "had it installed anytime since June of this year", i.e. when YiR was "released" in our code, even though it wasn't yet surfaced to users. If the XML defaults were already applied from a previous install of the app, any new preferences do not get their defaults applied from the XML.

Therefore, effectively, this issue affects users who've already had the app installed prior to YiR (specifically prior to our release in June), which indeed accounts for a sizeable majority.

This is why, during the course of our development and our testing, this never surfaced as an issue, because we generally launch a fresh install of the app, in which this issue wouldn't be seen.

How could it have been caught

This would have been extremely difficult to catch using any kind of regular QA process. However, we should work towards having comprehensive upgrade testing, i.e. tests of functionality longitudinally through successive upgrades of the app version. But even then, this issue could have only been caught with an upgrade test that is 10+ versions deep (!), since that's how many releases have been made from June until today.

The most relevant actions going forward have to do with our development practices:

• Add very loud TODO! comments in the code where some default value should be flipped at some point in the future.
• Document such defaults and assumptions in the Epic phab task, as a checklist item to be completed before rollout.
• Carve out tech-debt time to look for legacy code that is no longer relevant, but might be impacting current functionality.
  • Investigate and remove the legacy code that sets the defaults from XML. There should not be two sources of truth for preference defaults.

One more learning from this:
The 5-day funnel check task T411886 was something new we added for Year in Review. It was really useful for this purpose of finding display bugs, and next year we should do it earlier, after 2 days.