Problem
We have run two external QuickSurveys recently (T217080 and T217576) and for both surveys, about 10% of the responses recorded (i.e. surveys filled out via Qualtrics and Google Forms) do not have associated EventLogging data from QuickSurveyInitiation or QuickSurveysResponses. Some of the responses missing QuickSurveysResponses EL do have associated QuickSurveyInitiation EL.
What have we ruled out:
- Issue with the queries we are using to gather the EL data. The surveys missing EL data are scattered through the whole survey period so it is not related to incorrectly filtering the data.
- Issues w/ the URL-encoded EL being too long. I only saw six total instances in the event.eventerror table related to the surveys.
- Respondents tampering with the pageviewToken that we use to connect EL with the survey responses. In Qualtrics, this is not possible and we still see the missing EL.
Hypotheses
- Missing all EL is possibly related to browser agent. We have observed that certain browsers are being undersampled by QuickSurveys (T218243#5086923). It could be that these browsers are in fact seeing QuickSurveys, they just are not appropriately logging that.
- It seems other EventLogging schema have run into related challenges (T204143)
- Just missing QuickSurveysResponses EL (but not QuickSurveyInitiation) is likely related to this bug: T217171#4992112
- Essentially, right-clicking and opening a quicksurvey link in a new tab is not registered by the extension. Presumably the 91 responses for Reader Trust and 38 for Demographics Pilot that had QuickSurveyInitiation but not QuickSurveysReponses (despite completing the survey) could be the result of this behavior.
Survey Overviews
Reader Trust (T217576):
- Out of the 1971 survey responses recorded by Qualtrics:
- 1702 (86%) of those have corresponding QuickSurveysResponses data
- Another 92 (for a total of 1793 or 91%) can be matched to QuickSurveyInitation data.
Demographics Pilot (T217080):
- Out of the 626 survey responses recorded by Google Forms:
- 514 (82%) of those have corresponding QuickSurveysResponses data
- Another 38 (for a total of 552 or 88%) can be matched to QuickSurveyInitation data.
For these analyses, I fully skipped EventLogging and instead used webrequest logs, using a query like that below to gather the EL (and then attempting to join it to the survey responses provided by Qualtrics/Google Forms):
SELECT *,
REFLECT(‘java.net.URLDecoder', 'decode', SUBSTR(uri_query, 2)) AS json_event
FROM wmf.webrequest
WHERE uri_path LIKE '%beacon/event'
AND uri_query LIKE '%QuickSurvey%'
AND uri_query LIKE ‘%<survey-name>%’
AND year = 2019 AND month = 3 AND day >= 18 AND day < 23