Page MenuHomePhabricator

investigate: Record user agent in contribution_tracking
Closed, ResolvedPublic

Description

If this is possible it would be very helpful in solving browser issues. At the moment it takes time for donor services to follow up with users to get their browser details, but with this we could look it up from a contact or transaction id.

We could also monitor conversion rates across different browsers to try and discover issues pre-emptively.

Event Timeline

Hmm, unfortunately this table is so huge that adding a column would mean taking everything down for multiple hours. I don't think we can do this before January.

Oh yeah, I was definitely thinking after Big English for this! Just wanted to get it down while I remembered.

Hi @Ejegg, just wanted to flag this again. Would it be possible to look into this? It would be super useful data to have, and I think we could tolerate a few hours scheduled downtime for it.

My $0.02:

$0.01 - we should be careful with user agent and save some sort of fuzzy version because it is vulnerable to browser fingerprinting: https://en.wikipedia.org/wiki/Device_fingerprint

$0.01 - is there anywhere that would make sense to save it besides contribution_tracking? That table is a cross to bear at this point.

OK, I'll bring this up in sprint planning today

Looks like we DO have the full useragent in our fraud logs for anyone who gets far enough to run a minFraud check. That's not useful for the non-cc methods, though.

DStrine renamed this task from Record user agent in contribution_tracking to investigate: Record user agent in contribution_tracking.Feb 13 2018, 9:13 PM
DStrine moved this task from Triage to Completed in Q3 2017-18 on the Fundraising-Backlog board.

@Pcoombe and @MBeat33: Recarding @cwdent's point 1, if we were to parse the user-agent and store the info as a few discrete fields, how much of that info would you need? E.g. browser family, browser major version, OS family?

@Ejegg If we need to parse it, then family and major version for both browser and OS should be enough.

It looks like Analytics use this library for user agents in their webrequest data: https://github.com/ua-parser/uap-php

Thanks for the link @Pcoombe !
@cwdent, any ideas for a non-contribution-tracking place this might live? We wouldn't want to put it in civi contribution custom fields, since this data would help diagnose issues with people who can't successfully donate.

Maybe a new table with a key to c_t? I'm not real familiar with schema best practices there, but it seems worth trying not to centralize on that table even more.

We are adding the fields os os_version,browser, browser version to the new civicrm_contribution_tracking table, but they will only be populated when this task is brought into the sprint to pipe the data into that table

@Cstone suggested this task was relevant to the current conversation we're having around attribution of spontaneous donations, @ERoden-WMF , @JMando .

@Ejegg offered that additional info could be added to the contribution_tracking table like OS, and browser versions - perhaps some exploration around what could be proxies for source here.

That said - don't know if there should be a separate task investigating why there has been an increase in volume on these YoY.

Edit: ah, @Cstone found an existing task T161539: NULL referrers in contribution_tracking - I'll pull this into triage for discussion.

Change 967266 had a related patch set uploaded (by Ejegg; author: Ejegg):

[mediawiki/extensions/DonationInterface@master] Add user-agent info to contribution_tracking

https://gerrit.wikimedia.org/r/967266

Change 967271 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Save user-agent info to contribution_tracking table

https://gerrit.wikimedia.org/r/967271

Change 967266 merged by jenkins-bot:

[mediawiki/extensions/DonationInterface@master] Add user-agent info to contribution_tracking

https://gerrit.wikimedia.org/r/967266

Change 967271 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Save user-agent info to contribution_tracking table

https://gerrit.wikimedia.org/r/967271

This is deployed! We can do one follow-up patch to show the columns in the two places that contribution_tracking data is displayed in the Civi UI.

XenoRyet set Final Story Points to 2.

Change 968009 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Add OS and browser to contribution_tracking displays

https://gerrit.wikimedia.org/r/968009

Change 968009 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Add OS and browser to contribution_tracking displays

https://gerrit.wikimedia.org/r/968009