Page MenuHomePhabricator

Investigate April-May 2016 drop in mobile app edits
Closed, ResolvedPublic

Description

The December 2015–March 2016 level of mobile app edits was fairly stable around 100 K to 110 K. However, it dropped to 87 K in April and 67 K in May. The number so far in June (50 K on 21 June) extrapolates to 71 K.

What's going on?

Event Timeline

Hmm, the decline seems to be evenly split between the Android and iOS apps, which isn't consistent with some sort of regression preventing users from editing. Perhaps they switched to sampling the editing events?

SELECT LEFT(timestamp, 6) as "month", COUNT(*) as "mobile app edits",
SUM(IF(userAgent LIKE "%(iPhone%", 1, 0)) as "iOS edits",
SUM(IF(userAgent LIKE "%(Android%", 1, 0)) as "Android edits"
FROM log.MobileWikiAppEdit_9003125
WHERE
event_action = "saved" AND
wiki LIKE "%wiki" AND
wiki NOT IN (
"commonswiki", "wikidatawiki", "metawiki", "loginwiki",
"mediawikiwiki", "sourceswiki", "specieswiki", "testwiki",
"test2wiki", "testwikidatawiki", "incubatorwiki", "advisorywiki",
"outreachwiki", "qualitywiki", "strategywiki", "usabilitywiki",
"collabwiki", "donatewiki", "execwiki", "officewiki",
"legalteamwiki", "movementroleswiki", "internalwiki",
"auditcomwiki", "boardgovcomwiki", "boardwiki", "chairwiki",
"chapcomwiki", "checkuser_wiki", "foundationwiki", "grantswiki",
"iegcomwiki", "nostalgiawiki", "ombudsmenwiki", "otrs_wikiwiki",
"searchcomwiki", "spcomwiki", "stewardwiki", "tenwiki",
"transitionteamwiki", "votewiki", "wg_enwiki", "zerowiki",
"checkuserwiki", "fdcwiki" ) AND
wiki NOT LIKE "wikimania%" AND
wiki NOT LIKE "arbcom%" AND
event_pageNS IS NULL AND
timestamp BETWEEN "201601" AND "201607"
GROUP BY month;

month	mobile app edits	iOS edits	Android edits
201601	111921	19632	92287
201602	102102	19779	82323
201603	101795	11995	89799
201604	87315	5538	81776
201605	66800	5015	61785
201606	50020	2919	47101

Perhaps the data for April - May spans more than one table? See T138010#2390668.

@ori: They haven't deployed a new version of the same schema, but yeah, it's possible that they've done something more drastic like removing the logging of edit events altogether.

@JMinor , @Dbrant, do you know of any changes to the logging of edit events from the mobile apps in the past couple of months?

I just talked to @JMinor. There hasn't been any intentional change to the logging; the only other thing he could think of was an Nginx bug two months ago which made it appear that edits hadn't saved properly and affected both apps. However, it was promptly fixed, so it doesn't seem a likely candidate.

@Dbrant, I couldn't find the task for that bug. Could you point me to it?

It also looks like the logging code hasn't been touched either in the iOS app or in the Android app.

Here is the task that was related to the nginx issue that prevented users from saving edits: T134758 However, this issue only surfaced in May, and wouldn't explain the drop starting in April or even earlier.
My best theory so far would be as follows:
It's possible (but unlikely) that the issue started before April, but we weren't made *aware* of it until May. And even though we patched the issue quickly, the rate of users updating to the latest version of the app can be a bit sluggish. This would explain the edit rate continuing to fall in May, but picking back up (by extrapolation) in June.

On the iOS side, T134758 had the effect of showing a false error message "Could not connect to network." but edits did go through. So the impact was not as direct, but obviously users seeing an error message during save may have reduced editing or future motivation to edit.

I've pulled daily edit numbers, broken down by platform, and graphed them in this spreadsheet.

@Dbrant, the only noticeable event in the Android numbers is a significant drop on 4 May, which we've mostly recovered from since. That's clearly the Nginx bug.

@JMinor, the iOS numbers look considerably stranger. There's no noticeable effect from the Nginx bug, but on 11 March, edits abruptly drop from 600-700 a day to 100-200 a day, and have held at that lower level ever since. Any ideas?

@JMinor, I made a graph in that spreadsheet of iOS edits broken down by version. It seems like the drop happened when people largely stopped editing with 4.1.7.171, but didn't largely start editing with 5.0.0.775. Perhaps there was a design change that deemphasized the edit function?

Didn't v.5 of the iOS app bring in opt-in stats (instead of opt-out)? If that covers this, the numbers will be totally useless for tracking.

I just talked with @JMinor and he confirmed that version 5 of the iOS app included opt-in stats and that these event logs were included in that.

So it looks like the drop was due to the Nginx bug on the Android side and the switch to opt-in stats in the iOS app.

nshahquinn-wmf raised the priority of this task from Medium to Needs Triage.Mar 30 2018, 10:21 AM
nshahquinn-wmf moved this task from Neil's in progress to Done on the Contributors-Analysis board.