Page MenuHomePhabricator

Fundraising banners still showing on iOS app?
Open, HighPublic

Description

I was checking the numbers of donations from the app banners, and found that there's still a few coming in from the iOS app. Banners were supposed to come down at the end of 2019. It does seem like donations have decreased, but not dropped to zero (contrast with Android)

Haven't been able to reproduce seeing a banner in the app myself.

Date (UTC)Android donationsiOS donations
2019-12-28131320
2019-12-29133289
2019-12-3093225
2019-12-31119225
2020-01-01079
2020-01-02250
2020-01-03057
2020-01-04048
2020-01-05051
2020-01-06038
2020-01-07037
2020-01-08034

Event Timeline

Pcoombe created this task.Jan 9 2020, 4:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 9 2020, 4:23 PM
MBeat33 added a subscriber: MBeat33.Jan 9 2020, 4:32 PM

Thank you, @Pcoombe this comports with what we're seeing in Donor Services, where we're still seeing new persistent banner complaints.

LGoto triaged this task as High priority.Jan 13 2020, 7:43 PM
LGoto moved this task from Needs Triage to Bug Backlog on the Wikipedia-iOS-App-Backlog board.
LGoto moved this task from Bug Backlog to Engineering Backlog on the Wikipedia-iOS-App-Backlog board.

Looks like this is still happening. Is there anything fundraising can do to help with this? It seems to be in all the targeted countries.

@Pcoombe Odd, we can't repro on our end but I'll dig deeper tomorrow. Do you know which context the donations are coming from? (in the feed or overlay on article)? Also knowing which app versions these donations are coming from could be useful as well for us. Thanks!

@Pcoombe wanted to double-check - what query items are you using to get these numbers? We noticed both the Settings link and Article link share utm_medium and utm_campaign values. Just want to make sure Settings numbers aren't being counted.

Settings:
https://donate.wikimedia.org/w/index.php?title=Special:LandingPage&country=US&uselang=en&utm_medium=WikipediaApp&utm_source=6.5.1.0&utm_campaign=iOS

Article:
https://donate.wikimedia.org/w/index.php?title=Special:LandingPage&country=US&uselang=en&utm_medium=WikipediaApp&utm_source=app_201912_6C_control&utm_campaign=iOS&appeal=JimmyQuote

This is donations with utm_source = app_201912_6C_control and utm_campaign = iOS, so it should be only the banners, not the settings menu. I'm not able to tell what app version since we aren't collecting that for the banners (this is something we can think about improving for next year)

I'm still having trouble reproducing unfortunately. As long as the endTime has consistently returned 12/31 from the announcements feed, this seems to consider that each time and hide the announcement once past. Here's some of my test cases:

Test cases:

  1. Fresh install 6.5.1, change device date to December, force announcements feed to return announcements with endTime 12/31. Shows in article. Goes away after dismissal. Change device time forward. Is still gone.
  2. Fresh install 6.5.1, change device date to December, force announcements feed to return announcements with endTime 12/31. Change device time forward. Disappears.
  3. Fresh install 6.4.1, change device date to December, force announcements feed to return announcements with endTime 12/31. Shows in feed. Goes away after dismissal. Change device time forward. Is still gone.
  4. Fresh install 6.4.1, change device date to December, force announcements feed to return announcements with endTime 12/31. Shows in feed. Change device time forward. Disappears.
  5. Fresh install 6.4.1, change device date to December, force announcements feed to return announcements with endTime 12/31. Shows in feed. Upgrade to 6.5.1. Disappears in feed, shows in article. Dismiss in feed or article. Relaunch, does not show. Repeat step 5 but instead of dismiss move device date to current. Does not show.
  6. Fresh install 6.5.1, change device date to 12/31, , force announcements feed to return announcements with endTime 12/31. Go to article, see announcement, term while in article, move device date to Jan 1st, no longer see announcement.

Unless the user is messing around with the device date to see the announcement in the same way I am, but that seems very unlikely. I'll move this back to Ready for Dev in case another dev wants to take a crack at it. If we can't figure it out maybe we can push out a hotfix that deletes any local WMFAnnouncements with an actionURL that contains app_201912_6C_control upon launch, but would be good to fix the core issue so we don't run into it again next year.

Tsevener removed Tsevener as the assignee of this task.Jan 24 2020, 6:14 PM
Tsevener added a comment.EditedJan 24 2020, 7:20 PM

One other note - after seeing the announcement while my device date was in December, then moving the date to current time, I confirmed that the announcement is still hanging around in the local database even though I don't see it anymore.

@Pcoombe are we still seeing donations come through for this? Are the numbers decreasing? Trying to gauge if we need to do a hotfix for it.

@Tsevener It has decreased, but we are still seeing donations every day. It would be great to get this fixed since our donor services team have seen complaints.

datedonations
2020-01-0250
2020-01-0357
2020-01-0448
2020-01-0551
2020-01-0638
2020-01-0737
2020-01-0835
2020-01-0933
2020-01-1035
2020-01-1145
2020-01-1234
2020-01-1334
2020-01-1420
2020-01-1527
2020-01-1615
2020-01-1715
2020-01-1819
2020-01-1918
2020-01-2016
2020-01-2110
2020-01-2214
2020-01-2310
2020-01-248
2020-01-2510
2020-01-2614
2020-01-2713
2020-01-288
2020-01-299
2020-01-3012
2020-01-319
2020-02-0112
2020-02-029
2020-02-0313
2020-02-048

@Pcoombe thanks! We will look into a hotfix right away. Question for @MBeat33 - on our side we didn't capture a way of knowing which context the announcement is displaying, but is it possible to glean from the donor services complaints where in the app these announcements are showing (in explore feed or on top of the article view)? Only reason I ask is if we are able to confirm that it's only via an overlay on the article view, we can do a lower-impact fix. But if we aren't certain we can do the higher-impact fix which should remove them from both the explore feed and article view.

Hi @Tsevener it is difficult to get feedback from donors after the fact about what they saw specifically. My team is having capacity issues this week - I could track a handful of donors to their CID records in CiviCRM and get transaction IDs for them, if that would be helpful, but I do not have capacity to search fredge for which banners they saw. This is one of the smaller issues on my plate at the moment - sorry! If some sample transactions would be helpful please let me know.

JMinor claimed this task.Feb 11 2020, 7:23 PM
JMinor lowered the priority of this task from High to Medium.

Hey, seems like this isn't a priority on your end, and we've received no reports or complaints to the app store or OTRS regarding this, so I'm inclined to close this issue. If you have any customer emails on your end, you could just forward or copy/paste (minus payment/personal info) and send them to me or post it here. Transactions might be useful if they show the device (model or iPad/iPhone), app version, or other info derived from user agent. Just the donations themselves won't tell us much about the app :(

Without more info, there's really not much we can do.

This is a priority for us. We're still seeing several of these donations every day, and are now at over 800 donations since the campaign supposedly ended. I don't know how many unintended banners that means we're showing.

I did check the donatewiki webrequests for these donors, and it looks like a mix of user agents very similar to what we saw during the actual campaign: mostly iPhone / about 4% iPad. Mostly on iOS 13 with a few on 12 or lower. We aren't collecting the app version unfortunately, that's definitely something to add for our tracking in the next campaign.

2020-02-106
2020-02-115
2020-02-125
2020-02-138
2020-02-146
2020-02-152
2020-02-1612
2020-02-177

Thanks, @Pcoombe this is a priority for Donor Services as well, and sorry if my last comment made it seem otherwise. Persistent banners are one of the top two complaints from donors, and it would be great to be able to turn these off.

@JMinor would Civi CIDs be helpful info to you?

Donor comments are not often specific enough to enable us to differentiate the ones who may be seeing the banner on the app versus who may be complaining about persistent banners from before they were taken down from elsewhere on Wikipedia. Samples:

ticketdatecomment
7292262/6The only thing I ask is, that once I have donated, please stop asking over and over. I don't mind donating at all. But once I have please stop asking.
7285812/5I donated once. And then when I logged onto Wikimedia, it asked me for a donation again and again. Think, I wasn't able to move forward at one point and I kept saying (to the computer screen) that I have already donated but you don't know that I have.
7239101/13Is it possible somehow not to receive messages asking for donations when I use Wiki?
7233231/9Recently I have not been able to use Wiki for information as I keep getting asked to donate.
7230831/8just fyi I finally donated... AND IT STILL KEPT BUGGING ME TO DONATE. EVEN ON COMPUTERS WHERE I HAD DISMISSED THE THING. like omfg I AM! Anyway I'd like to cancel my donation because of that.

Unfortunately our system are not really integrated in a way that anything other than user reports or that is user agent like (platform, OS version, app version, language/wiki) is much help. So @pcombe thanks for that info, although it sounds like nothing is jumping out.

Usually we'd follow-up with some specific questions for the users about their experience, but given the lag here and the game of "telephone" that would involve its probably not an efficient option.

So a few comments on what I see so far given the info we have:

    • Reports of seeing banners after donating are a platform wide issue, given our choices around login and privacy. We get these every year on the apps and I don't think its soemthing we can "fix", though maybe having a canned response about why they see a banner after donating when switching platforms (or even devices if not logged in) and its the price they pay for our valuing their privacy. Most of the examples given are indeed ambigious about apps being involved or seem outright not about apps ("when I switch computers...").
  • Reports of the banner or popover not being dismissable are generally simple user error on understanding they can tap outside the banner, use the close button, etc to dismiss it. We haven't been able in several rounds of testing before or after been able to reproduce this, and again, not received any app specific complaints via OTRS or email about this. Its possible this is happening, but again I don't think thats the core issue leading to 800 additional donations (ie. I doubt you'd donate just to dismiss the banner).
  • Finally, and here's where I think the core fix will be, which is for us to run a couple more tests using the real production announcement infrastructure and our beta channel. I'll circle back with the team to figure out how to set that testing up.
  • If all else fails and this continues, we can do a patch to delete all store messages one time, which is not a long term fix, but is the most forceful option.

cc @ABorbaWMF our QA guy since this might involve some unique testing needs...

Thanks for continuing to look at this @JMinor.

I agree that deleting the messages wouldn't be a long term fix, but if we do run another campaign it can definitely include some better tracking for app version number etc. So if the problem does recur then, we would at least be in a better position to debug.

Hi @JMinor, we're still seeing about 10 donations coming in through the iOS app every day. Could we please go with the option of a patch to delete all stored messages?

JMinor added a comment.Mar 4 2020, 7:07 PM

Ok, will circle back with the team to get ETA for that. Sorry about this!

JMinor raised the priority of this task from Medium to High.Mar 5 2020, 7:03 PM
JMinor added a subscriber: SNowick_WMF.

One more thing that would be helpful for us: can we get the impression count for those banners during this same time? Sorry I didn't ask this earlier.

ls cc'ing @SNowick_WMF our data analyst who might be able to help pull this number. The SQL query should be the same as from December.

Bump. We are still seeing donations from this.

Pcoombe closed this task as Resolved.May 26 2020, 1:57 PM

Thank you. These donations do appear to have dried up now.

Pcoombe reopened this task as Open.Tue, Jan 12, 3:55 PM

I'm afraid this is happening again with the 2020 banners continuing into 2021:

datedonationsimpression events
2021-01-0183490
2021-01-0286380
2021-01-0378377
2021-01-0453288
2021-01-0551219
2021-01-0638249
2021-01-0750210
2021-01-0840212
2021-01-0942225
2021-01-1046221
2021-01-1138155

Query for impression events

SELECT
  CONCAT(year, '-', LPAD(month, 2, '0'), '-', LPAD(day, 2, '0')) AS date,
  COUNT(*) AS event_count
FROM event.MobileWikiAppiOSFeed
WHERE year = 2021
  AND event.action = 'impression'
  AND ( event.label = 'announcement' OR event.label = 'article_announcement' )
GROUP BY year, month, day
ORDER BY date
LIMIT 50;

@Pcoombe darn, thanks for the heads up - I'll take a look today.

SHust added a subscriber: SHust.Tue, Jan 12, 4:05 PM

Note for QA: this fix is in the TestFlight build 6.7.4 (1791).