Fundraising banners still showing on iOS app?
Open, HighPublic
Actions

Description

I was checking the numbers of donations from the app banners, and found that there's still a few coming in from the iOS app. Banners were supposed to come down at the end of 2019. It does seem like donations have decreased, but not dropped to zero (contrast with Android)

Haven't been able to reproduce seeing a banner in the app myself.

Date (UTC)	Android donations	iOS donations
2019-12-28	131	320
2019-12-29	133	289
2019-12-30	93	225
2019-12-31	119	225
2020-01-01	0	79
2020-01-02	2	50
2020-01-03	0	57
2020-01-04	0	48
2020-01-05	0	51
2020-01-06	0	38
2020-01-07	0	37
2020-01-08	0	34

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		Tsevener	T242347 Fundraising banners still showing on iOS app?
Resolved		JoeWalsh	T247554 ensure 2019 fundraising messages are removed from announcement database
Resolved		Pcoombe	T247563 Get impressions for fundraising baners for 2019
Open	Spike	Tsevener	T247976 Investigate the root cause of the 2019 fundraising banner still showing in the app after the end date before implementing the 2020 banner

Event Timeline

Pcoombe created this task.Jan 9 2020, 4:23 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 9 2020, 4:23 PM

Thank you, @Pcoombe this comports with what we're seeing in Donor Services, where we're still seeing new persistent banner complaints.

LGoto triaged this task as High priority.Jan 13 2020, 7:43 PM

LGoto added a project: iOS-app-v6.6-Penguin-On-A-Sled.

LGoto moved this task from Needs Triage to Bug Backlog on the Wikipedia-iOS-App-Backlog board.

LGoto moved this task from Tasks from Product Backlog to Ready for Development on the iOS-app-v6.6-Penguin-On-A-Sled board.

LGoto moved this task from Bug Backlog to Engineering Backlog on the Wikipedia-iOS-App-Backlog board.

Looks like this is still happening. Is there anything fundraising can do to help with this? It seems to be in all the targeted countries.

@Pcoombe Odd, we can't repro on our end but I'll dig deeper tomorrow. Do you know which context the donations are coming from? (in the feed or overlay on article)? Also knowing which app versions these donations are coming from could be useful as well for us. Thanks!

Tsevener claimed this task.Jan 22 2020, 2:58 PM

Tsevener moved this task from Ready for Development to Doing on the iOS-app-v6.6-Penguin-On-A-Sled board.

@Pcoombe wanted to double-check - what query items are you using to get these numbers? We noticed both the Settings link and Article link share utm_medium and utm_campaign values. Just want to make sure Settings numbers aren't being counted.

Settings:
https://donate.wikimedia.org/w/index.php?title=Special:LandingPage&country=US&uselang=en&utm_medium=WikipediaApp&utm_source=6.5.1.0&utm_campaign=iOS

Article:
https://donate.wikimedia.org/w/index.php?title=Special:LandingPage&country=US&uselang=en&utm_medium=WikipediaApp&utm_source=app_201912_6C_control&utm_campaign=iOS&appeal=JimmyQuote

This is donations with utm_source = app_201912_6C_control and utm_campaign = iOS, so it should be only the banners, not the settings menu. I'm not able to tell what app version since we aren't collecting that for the banners (this is something we can think about improving for next year)

I'm still having trouble reproducing unfortunately. As long as the endTime has consistently returned 12/31 from the announcements feed, this seems to consider that each time and hide the announcement once past. Here's some of my test cases:

Test cases:

Fresh install 6.5.1, change device date to December, force announcements feed to return announcements with endTime 12/31. Shows in article. Goes away after dismissal. Change device time forward. Is still gone.
Fresh install 6.5.1, change device date to December, force announcements feed to return announcements with endTime 12/31. Change device time forward. Disappears.
Fresh install 6.4.1, change device date to December, force announcements feed to return announcements with endTime 12/31. Shows in feed. Goes away after dismissal. Change device time forward. Is still gone.
Fresh install 6.4.1, change device date to December, force announcements feed to return announcements with endTime 12/31. Shows in feed. Change device time forward. Disappears.
Fresh install 6.4.1, change device date to December, force announcements feed to return announcements with endTime 12/31. Shows in feed. Upgrade to 6.5.1. Disappears in feed, shows in article. Dismiss in feed or article. Relaunch, does not show. Repeat step 5 but instead of dismiss move device date to current. Does not show.
Fresh install 6.5.1, change device date to 12/31, , force announcements feed to return announcements with endTime 12/31. Go to article, see announcement, term while in article, move device date to Jan 1st, no longer see announcement.

Unless the user is messing around with the device date to see the announcement in the same way I am, but that seems very unlikely. I'll move this back to Ready for Dev in case another dev wants to take a crack at it. If we can't figure it out maybe we can push out a hotfix that deletes any local WMFAnnouncements with an actionURL that contains app_201912_6C_control upon launch, but would be good to fix the core issue so we don't run into it again next year.

Tsevener removed Tsevener as the assignee of this task.Jan 24 2020, 6:14 PM

Tsevener moved this task from Doing to Ready for Development on the iOS-app-v6.6-Penguin-On-A-Sled board.

One other note - after seeing the announcement while my device date was in December, then moving the date to current time, I confirmed that the announcement is still hanging around in the local database even though I don't see it anymore.

@Pcoombe are we still seeing donations come through for this? Are the numbers decreasing? Trying to gauge if we need to do a hotfix for it.

@Tsevener It has decreased, but we are still seeing donations every day. It would be great to get this fixed since our donor services team have seen complaints.

date	donations
2020-01-02	50
2020-01-03	57
2020-01-04	48
2020-01-05	51
2020-01-06	38
2020-01-07	37
2020-01-08	35
2020-01-09	33
2020-01-10	35
2020-01-11	45
2020-01-12	34
2020-01-13	34
2020-01-14	20
2020-01-15	27
2020-01-16	15
2020-01-17	15
2020-01-18	19
2020-01-19	18
2020-01-20	16
2020-01-21	10
2020-01-22	14
2020-01-23	10
2020-01-24	8
2020-01-25	10
2020-01-26	14
2020-01-27	13
2020-01-28	8
2020-01-29	9
2020-01-30	12
2020-01-31	9
2020-02-01	12
2020-02-02	9
2020-02-03	13
2020-02-04	8

@Pcoombe thanks! We will look into a hotfix right away. Question for @MBeat33 - on our side we didn't capture a way of knowing which context the announcement is displaying, but is it possible to glean from the donor services complaints where in the app these announcements are showing (in explore feed or on top of the article view)? Only reason I ask is if we are able to confirm that it's only via an overlay on the article view, we can do a lower-impact fix. But if we aren't certain we can do the higher-impact fix which should remove them from both the explore feed and article view.

Hi @Tsevener it is difficult to get feedback from donors after the fact about what they saw specifically. My team is having capacity issues this week - I could track a handful of donors to their CID records in CiviCRM and get transaction IDs for them, if that would be helpful, but I do not have capacity to search fredge for which banners they saw. This is one of the smaller issues on my plate at the moment - sorry! If some sample transactions would be helpful please let me know.

JoeWalsh moved this task from Ready for Development to Blocked or Waiting on the iOS-app-v6.6-Penguin-On-A-Sled board.Feb 11 2020, 7:18 PM

JMinor claimed this task.Feb 11 2020, 7:23 PM

JMinor lowered the priority of this task from High to Medium.

Hey, seems like this isn't a priority on your end, and we've received no reports or complaints to the app store or OTRS regarding this, so I'm inclined to close this issue. If you have any customer emails on your end, you could just forward or copy/paste (minus payment/personal info) and send them to me or post it here. Transactions might be useful if they show the device (model or iPad/iPhone), app version, or other info derived from user agent. Just the donations themselves won't tell us much about the app :(

Without more info, there's really not much we can do.

This is a priority for us. We're still seeing several of these donations every day, and are now at over 800 donations since the campaign supposedly ended. I don't know how many unintended banners that means we're showing.

I did check the donatewiki webrequests for these donors, and it looks like a mix of user agents very similar to what we saw during the actual campaign: mostly iPhone / about 4% iPad. Mostly on iOS 13 with a few on 12 or lower. We aren't collecting the app version unfortunately, that's definitely something to add for our tracking in the next campaign.

2020-02-10	6
2020-02-11	5
2020-02-12	5
2020-02-13	8
2020-02-14	6
2020-02-15	2
2020-02-16	12
2020-02-17	7

Thanks, @Pcoombe this is a priority for Donor Services as well, and sorry if my last comment made it seem otherwise. Persistent banners are one of the top two complaints from donors, and it would be great to be able to turn these off.

@JMinor would Civi CIDs be helpful info to you?

Donor comments are not often specific enough to enable us to differentiate the ones who may be seeing the banner on the app versus who may be complaining about persistent banners from before they were taken down from elsewhere on Wikipedia. Samples:

ticket	date	comment
729226	2/6	The only thing I ask is, that once I have donated, please stop asking over and over. I don't mind donating at all. But once I have please stop asking.
728581	2/5	I donated once. And then when I logged onto Wikimedia, it asked me for a donation again and again. Think, I wasn't able to move forward at one point and I kept saying (to the computer screen) that I have already donated but you don't know that I have.
723910	1/13	Is it possible somehow not to receive messages asking for donations when I use Wiki?
723323	1/9	Recently I have not been able to use Wiki for information as I keep getting asked to donate.
723083	1/8	just fyi I finally donated... AND IT STILL KEPT BUGGING ME TO DONATE. EVEN ON COMPUTERS WHERE I HAD DISMISSED THE THING. like omfg I AM! Anyway I'd like to cancel my donation because of that.

Unfortunately our system are not really integrated in a way that anything other than user reports or that is user agent like (platform, OS version, app version, language/wiki) is much help. So @pcombe thanks for that info, although it sounds like nothing is jumping out.

Usually we'd follow-up with some specific questions for the users about their experience, but given the lag here and the game of "telephone" that would involve its probably not an efficient option.

So a few comments on what I see so far given the info we have:

- Reports of seeing banners after donating are a platform wide issue, given our choices around login and privacy. We get these every year on the apps and I don't think its soemthing we can "fix", though maybe having a canned response about why they see a banner after donating when switching platforms (or even devices if not logged in) and its the price they pay for our valuing their privacy. Most of the examples given are indeed ambigious about apps being involved or seem outright not about apps ("when I switch computers...").
Reports of the banner or popover not being dismissable are generally simple user error on understanding they can tap outside the banner, use the close button, etc to dismiss it. We haven't been able in several rounds of testing before or after been able to reproduce this, and again, not received any app specific complaints via OTRS or email about this. Its possible this is happening, but again I don't think thats the core issue leading to 800 additional donations (ie. I doubt you'd donate just to dismiss the banner).
Finally, and here's where I think the core fix will be, which is for us to run a couple more tests using the real production announcement infrastructure and our beta channel. I'll circle back with the team to figure out how to set that testing up.
If all else fails and this continues, we can do a patch to delete all store messages one time, which is not a long term fix, but is the most forceful option.

cc @ABorbaWMF our QA guy since this might involve some unique testing needs...

Thanks for continuing to look at this @JMinor.

I agree that deleting the messages wouldn't be a long term fix, but if we do run another campaign it can definitely include some better tracking for app version number etc. So if the problem does recur then, we would at least be in a better position to debug.

JMinor removed a project: iOS-app-v6.6-Penguin-On-A-Sled.Mar 2 2020, 7:50 PM

Hi @JMinor, we're still seeing about 10 donations coming in through the iOS app every day. Could we please go with the option of a patch to delete all stored messages?

Ok, will circle back with the team to get ETA for that. Sorry about this!

One more thing that would be helpful for us: can we get the impression count for those banners during this same time? Sorry I didn't ask this earlier.

ls cc'ing @SNowick_WMF our data analyst who might be able to help pull this number. The SQL query should be the same as from December.

JoeWalsh added a subtask: T247976: Investigate the root cause of the 2019 fundraising banner still showing in the app after the end date before implementing the 2020 banner.Mar 18 2020, 1:58 PM

Bump. We are still seeing donations from this.

JMinor closed subtask T247554: ensure 2019 fundraising messages are removed from announcement database as Resolved.May 15 2020, 12:27 AM

Thank you. These donations do appear to have dried up now.

I'm afraid this is happening again with the 2020 banners continuing into 2021:

date	donations	impression events
2021-01-01	83	490
2021-01-02	86	380
2021-01-03	78	377
2021-01-04	53	288
2021-01-05	51	219
2021-01-06	38	249
2021-01-07	50	210
2021-01-08	40	212
2021-01-09	42	225
2021-01-10	46	221
2021-01-11	38	155

Query for impression events

SELECT
  CONCAT(year, '-', LPAD(month, 2, '0'), '-', LPAD(day, 2, '0')) AS date,
  COUNT(*) AS event_count
FROM event.MobileWikiAppiOSFeed
WHERE year = 2021
  AND event.action = 'impression'
  AND ( event.label = 'announcement' OR event.label = 'article_announcement' )
GROUP BY year, month, day
ORDER BY date
LIMIT 50;

MBeat33 added a project: fr-donorservices.Tue, Jan 12, 4:00 PM

@Pcoombe darn, thanks for the heads up - I'll take a look today.

SHust added a subscriber: SHust.Tue, Jan 12, 4:05 PM

Tsevener claimed this task.Thu, Jan 14, 1:06 AM

Tsevener added a project: iOS-app-v6.8-Manta-Ray-On-A-Riding-Mower.

Tsevener moved this task from Tasks from Product Backlog to Doing on the iOS-app-v6.8-Manta-Ray-On-A-Riding-Mower board.

Tsevener added a subscriber: JMinor.

PR in progress: https://github.com/wikimedia/wikipedia-ios/pull/3828

Tsevener moved this task from Doing to Needs Code Review on the iOS-app-v6.8-Manta-Ray-On-A-Riding-Mower board.Thu, Jan 14, 7:45 PM