Page MenuHomePhabricator

WikimediaEvents do not track logged in beta users on Special:MobileOptions
Closed, ResolvedPublic

Description

When a user is logged and in opts into beta mode in Special:MobileOptions we set a preference, but this is not logged in Schema:PrefUpdate. It should be.

Problem

Currently WikimediaEvents Hook listens to UserSaveOptions hook, and when the user changes preferences it tracks the change in PrefUpdate schema. The Mobile Beta Mode (do not mix with Beta Features) preferences is stored in UserOptions, so it supposedly, all Mobile Beta Mode switches should be tracked by PrefUpdate schema. This doesn't happen because WikimediaEventsHooks::onUserSaveOptions logic tracks the Analytics Events only when User::saveOptions was called on:

  • Special:Preferences
  • via API action=options call

All other events are just ignored. This is expected as most probably we don't want to track user preferences when user object is created (user creation) and during maintenance scripts.

Why?

We're planning to store the Advanced Mobile Contributions preference as UserOption and track user retention. The PrefUpdate schema looks like the best candidate but because of WikimediaEventsHooks:onUserSaveOptions logic we cannot use it.

Open questions

Why WikimediaEvents tracks PrefUpdate only on Special:Preferences page submits and action=options API calls?

Answer: It's done this way to limit all analytics events only to user actions. We don't want to track PrefUpdate when maitenance script or custom logic enables features for users.

Can we remove that logic and track PrefUpdate on any page? There are not that many places that call User::saveSettings, definitely we have to not track events on maitenanceScripts, what about other places?

Answer: No, there are pages that update user options and we don't want to track those (because that are not user initiated actions), for example, we don't want to track:

Most probably there are more situations like two mentioned above.

Developer notes:

The safest way is to just to add $title->isSpecial('Preferences') || $title->isSpecial('MobileOptions'), but maybe there is something we can improve.

There is a TODO in code

 		// TODO (mattflaschen, 2013-06-13): Ideally this would be done more cleanly without
		// looking explicitly at page names and URL parameters.
		// Maybe a userInitiated flag passed to saveSettings would work.

Event Timeline

Jdlrobson renamed this task from WikimediaEvents do not track user option changes on Special:MobileOptions to WikimediaEvents do not track logged in beta users on Special:MobileOptions.Dec 21 2018, 6:30 PM
Jdlrobson triaged this task as Medium priority.

We're gonna do an async estimation before considering pulling this in.

Change 481925 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[mediawiki/extensions/WikimediaEvents@master] Track Shema:Pref changes on Special:MobileOptions page

https://gerrit.wikimedia.org/r/481925

Jdlrobson changed the task status from Open to Stalled.Jan 2 2019, 11:45 PM

We attempted an async estimation but estimations were varied between 2-5. It seems to me the lower estimates came from the people close to this problem and the associated spike and those less close who felt this was a context switch.

My recommendation is thus for us to let this task stew a little more and resestimate as part of enabling the amc mode.

Jdlrobson changed the task status from Stalled to Open.Jan 7 2019, 7:40 PM

With the above patch merged do we want to repurpose this to be about fixing the FIXME?

Change 481925 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Track Shema:Pref changes on Special:MobileOptions page

https://gerrit.wikimedia.org/r/481925

To fix this issue properly I created T213840, for now let's fix only handling for Special:MobileOptions page.

Let's check that events from beta changes are showing up in the PrefUpdate schema. Provided that's working I think this can be resolved.

Yesterday afternoon, I opted in and out of mobile beta mode whilst logged in with my Phuedx (WMF) account (ID: 22867088, see [1]).

I see the following in the event.prefupdate table in Hive:

[0]
select
  event
from
  event.prefupdate
where
  year = 2019
  and month = 1
  and day = 16
  and event.userId = 22867088
;

{"isDefault":false,"property":"mfMode","saveTimestamp":"20190116174920","userId":22867088,"value":"\"beta\"","version":"1"}
{"isDefault":true,"property":"mfMode","saveTimestamp":"20190116175123","userId":22867088,"value":"\"\"","version":"1"}
[1]
console.log( mw.config.get( 'wgUserId' ) ); // 22867088
This comment was removed by Tbayer.
phuedx claimed this task.

We appear to be capturing the appropriate information in the PrefUpdate schema (see below) though it's a shame we don't capture the source of the update as well.

[0]
+---------+-------+
|  value  |   n   |
+---------+-------+
| ""      | 277   |
| "beta"  | 1192  |
+---------+-------+

select
    event.value as value,
    count(*) as n
from
    event.prefupdate
where
    year = 2019
    and month = 1
    and day >= 13
    and event.property = "mfMode"
group by
    event.value
;

Thanks! Out of paranoia an abundance of caution, I double-checked that the "value" values are consistent with the "isDefault" values, which seems to be the case:

SELECT 
SUM(IF(event_value = '"beta"' AND event_isDefault != 1,1,0)) AS beta_and_not_default, 
SUM(IF(event_value = '""' AND event_isdefault = 1,1,0)) AS default_and_not_beta, 
SUM(1) AS all_events 
FROM log.PrefUpdate_5563398 
WHERE timestamp LIKE '201902%' 
AND event_property = 'mfMode';
+----------------------+----------------------+------------+
| beta_and_not_default | default_and_not_beta | all_events |
+----------------------+----------------------+------------+
|                 5321 |                 1256 |       6577 |
+----------------------+----------------------+------------+
1 row in set (5.01 sec)