Page MenuHomePhabricator

As a PM I'd like to know how many iOS app user are blocked from editing and registration
Closed, ResolvedPublic

Description

We've seen an uptick in user reports on iOS and Android of users being blocked at registration or edit time on the apps. After some discussion my theory is that this is growing because of the heavy use of IP and range block techniques, which are no longer as relevant on mobile but being used in an increasingly broad manner anyway.

To determine if this user-reported trend is legitimate and the scale of the impact of these administrator choices on mobile users, Id like to have some basic data about the amount of blocking occcuring in the app.

So, for a given period (say last 30 days), lets start with the basics:

  • How many users are blocked from registering?
  • How many users are blocked from editing (both anon/IP and registered)?

Note that this should not include blocks based on pages being protected only on the user being prevented from registering or editing system-wide.

For these questions some additional dimensions would be ideal:

  • Geographic or regional
  • Language (or at least top N biggest wikipedias)
  • Connection type (wifi vs cellular)

Since we've not looked at blocking data before I realize some of these may not currently be instrumented. The first ask for this task is just to help determine what data we do have, and for data we don't have where/how to best instrument it.

cc Android PM and Analyst as FYI as this is likely a "platform wide" issue affecting both apps.

Event Timeline

JMinor triaged this task as Medium priority.Jan 24 2019, 7:19 PM
JMinor created this task.
JMinor moved this task from Needs Triage to Product Backlog on the Wikipedia-iOS-App-Backlog board.

The Anti-Harassment Tools team recently gathered some rough data in T189724: 👩‍👧 Track how often blocked users attempt to edit which I've documented on Meta Wiki.

Every block has a block ID (table) which could also be recorded. This could then be compared to compute if the block was a username, IP, or IP range block.

chelsyx moved this task from Triage to Next Up on the Product-Analytics board.Feb 14 2019, 7:29 PM
chelsyx moved this task from Next Up to Doing on the Product-Analytics board.Feb 21 2019, 9:34 PM

Thanks @TBolliger for the information and helping me creating a blocked account for test!

chelsyx added a comment.EditedFeb 21 2019, 9:45 PM

Test case 1: As a ID-blocked logged in user, I'm trying to edit a page on testwiki via the iOS app
Outcome: Error message shown on the screen:

Our existing edit funnel (MobileWikiAppEdit) captured the error as below:

{
    event =     {
        action = error;
        anon = 0;
        "app_install_id" = "BB34E533-F82A-43D4-847B-AF2E78D27ABA";
        "client_dt" = "2019-02-21T13:19:03-08:00";
        errorText = other;
        "session_token" = "84BEE7CA-0F10-4647-8934-386A83D66B51";
    };
    revision = 18115551;
    schema = MobileWikiAppEdit;
    wiki = testwiki;
}

@JoeWalsh You mentioned in T210970#4934751 that we are sending the unlocalized error code string returned by the server as the value for the errorText, I'm wondering the kinds of error this other value includes--does it only show up when users are blocked? Also, is there a mapping between the unlocalized error code string and their actual meaning?

chelsyx added a comment.EditedFeb 21 2019, 10:21 PM

Test case 2: As a IP-blocked user (logged out), I'm trying to edit a page on testwiki via the iOS app
Outcome: Error message shown on the screen:
ID block but user is logged out:


IP block:

The events captured by our edit funnel is the same as test case 1:

{
    event =     {
        action = error;
        anon = 1;
        "app_install_id" = "BB34E533-F82A-43D4-847B-AF2E78D27ABA";
        "client_dt" = "2019-02-21T14:02:20-08:00";
        errorText = other;
        "session_token" = "8D2090CD-9DCB-4246-A88A-35BD8820EB29";
    };
    revision = 18115551;
    schema = MobileWikiAppEdit;
    wiki = testwiki;
}
chelsyx added a comment.EditedFeb 21 2019, 10:44 PM

Test case 3: As a IP-blocked user (logged out), I'm trying to register a new account on testwiki via the iOS app
Outcome: Error message shown on the screen:


Our existing MobileWikiAppCreateAccount captured the error as below:

{
    event =     {
        action = error;
        appInstallID = "ED0D972D-9F7E-44CB-97FF-E073C1331F6E";
        errorText = "Account creation from this IP address (<strong>198.73.209.241</strong>) has been blocked by [[User:TBolliger (WMF)|TBolliger (WMF)]].\n\nThe reason given by TBolliger (WMF) is <em>Testing block notices on iOS app</em>";
        sessionToken = "87F7FF16-79AC-4EFF-B3C4-54A9B6415E00";
        ts = "2019-02-21T14:34:24-08:00";
    };
    revision = 17836914;
    schema = MobileWikiAppCreateAccount;
    wiki = testwiki;
}

@JoeWalsh I assume the errorText here in MobileWikiAppCreateAccount is localized, correct?

Test case 4: As a ID-blocked user, I logged out and try to register a new account on testwiki via the iOS app
Outcome: I'm able to register a new account but cannot log in on the app. I can log in this account and make edit on desktop (see user log). However, after the block is expired, I still can't log in this account on the app. Additionally, I saw two different behaviors when I tried to log in on my phone and on Xcode simulator:

On Xcode simulator (No event sent to eventlogging):

On my phone (No event sent to eventlogging):

This looks a weird to me... @JMinor and @TBolliger , do you have any insight?

TBolliger added a comment.EditedFeb 22 2019, 5:46 PM

There is only one tool (to my knowledge) that prohibits a user from logging-in to an existing accounts: the WMF and steward tool called "account lock" which is used very infrequently that scrambles the password and disallows password resets.

On web, we allowed blocked usernames and usernames within a blocked IP to log-in and modify preferences, which I would expect to be the behavior on the apps.

@JoeWalsh I assume the errorText here in MobileWikiAppCreateAccount is localized, correct?

@chelsyx correct, that one looks to be localized. I can update it to match the unlocalized behavior we implemented for https://phabricator.wikimedia.org/T210970#4934751 if you like.

@JoeWalsh I assume the errorText here in MobileWikiAppCreateAccount is localized, correct?

@chelsyx correct, that one looks to be localized. I can update it to match the unlocalized behavior we implemented for https://phabricator.wikimedia.org/T210970#4934751 if you like.

Thanks @JoeWalsh ! But no rush :) Do you have a mapping between the unlocalized error code string and their actual meaning? Specifically, do we know what errors are we actually encountering when errorText = other?

There are a couple of options in the block that the admin can configure for IP blocks.

  • Account creation — Default checked. Prohibits people at that IP address from creating new accounts
  • Prevent logged-in users from editing from this IP address — Default unchecked. Prohibits existing user accounts from editing from that IP range.

These two preference are similar but not identical. Here are the behaviors on their combinations:

#Account creationPrevent logged-in [...]Behavior
1☑️☑️Logged-out users cannot edit, new accounts cannot be created, and existing accounts can log in but not edit.
2☑️🔲Logged-out users cannot edit, new accounts cannot be created, and existing accounts can log in and edit
3🔲🔲Logged-out users cannot edit, new accounts can be created, and existing accounts can log in and edit
4🔲☑️Logged-out users cannot edit, new accounts can be created, and existing accounts can log in but not edit

#1 is the most common, #2 and #3 happen regularly. #4 is rare.

This comment was removed by chelsyx.

The large number of IP block looks very suspicious to me. It's not even IP range block if my interpretation is correct...

I'm not surprised by that difference between ID and IP. It seems within a realistic realm of possibility to me. But if you're not confident, more logging/data could be useful.

The large number of IP block looks very suspicious to me. It's not even IP range block if my interpretation is correct...

I'm not surprised by that difference between ID and IP. It seems within a realistic realm of possibility to me. But if you're not confident, more logging/data could be useful.

@TBolliger Thanks for the insight! That was just my guess and I honestly don't have any idea about the reasonable difference between ID and IP block counts...

Given that we don't have block ID/wiki user ID in the eventlogging table, the next step of this investigation will be:
1, Join the IP addresses from these eventlogging table with the ipblocks table to make sure they really are IP block.
2, iOS engineers check with Android engineers to make sure there's nothing wrong with the API, or error message localization, or eventlogging on the iOS app. The fact that we didn't see any error messages relating to block on Android eventlogging is very suspicious.

chelsyx moved this task from Doing to Tracking on the Product-Analytics board.Feb 26 2019, 8:39 PM

I deleted the previous comment and redo the analysis. I get the counts from table MobileWikiAppEdit and MobileWikiAppCreateAccount, then join the IP addresses from these eventlogging table with the ipblocks table to make sure they really are IP block. See the full analysis here: https://people.wikimedia.org/~chelsyx/iOS%20app%20edit%20and%20registration%20block.html. The summary of it is shown below:

Summary

This analysis use the editing (MobileWikiAppEdit) and account creating (MobileWikiAppCreateAccount) event logging table to find out how many users got block error messages, thus we are only counting users who opt in to share their usage report with us -- the real number should be larger. We didn't see any error messages relating to block for Android on both tables, which may be a data bug on Android.

Editing Block

  • From Nov 2018 to now, on the iOS app, 71 unique IP addresses got IP block, 81 unique IP addresses got IP range block, at most 51 unique app install ID (83 unique IP) got ID block. For the rest of 137 blocked unique app install ID (275 unique IP), we cannot confirm their block type.
  • Among all 137 unknown block, 59 from the US, 19 from Japan. When breaking down by wiki, 84 from enwiki, 19 from jawiki, 8 from dewiki, 6 from nlwiki.
  • Among all 51 ID block, 19 from the US. When breaking down by wiki, 25 from enwiki, 4 from dewiki.
  • Among all 81 IP range block, 31 from the US, 23 from the UK. When breaking down by wiki, 59 from enwiki.
  • Among all 71 IP block, 17 from the US. When breaking down by wiki, 31 from enwiki, 10 from dewiki.

Caveat: Since we didn't send block ID/wiki user ID to the eventlogging table, we can only join the eventlogging data with Ipblocks table and Logging table by IP addresses to check whether an error message is an IP block. Therefore, the block type we're talking about here are defined as:

  • IP range block: Any users who see "Your IP address is in a range which has been blocked on all wikis. The block was made by User:XXX ..."
  • IP block: Users who see the message "You have been blocked from editing." and their IP addresses was in the Ipblocks table or the logging table (log type is block). Please note that this count is not very accurate but should be in the ballpark, since some tables in the queries are almost real-time, while others were only updated once a month.
  • ID block: Any users who see "Your IP address has been blocked automatically, because it was used by a blocked user.", or any logged in users who see the message "You have been blocked from editing.". This is an upper limit of the ID block counts, not an accurate number.
  • Unknown: Any other users who see the message "You have been blocked from editing."

Registration Block

  • From Nov 2018 to now, on the iOS app, 41 unique IP addresses (38 unique app install ID) got IP block when trying to register an account. Additionally, there are 347 unique IP addresses (332 unique app install ID) got error messages that contain the word "IP" but can't be confirmed as IP block -- we call them "Unconfirmed IP block" in this analysis.
  • Among the 347 unconfirmed IP block, 101 from the US, 43 from Japan, 32 from Germany, 32 from UK, 19 from India, 12 from Netherlands. When breaking down by wiki, 177 from enwiki, 41 from jawiki, 39 from trwiki, 19 from zhwiki, 17 from ruwiki, 15 from eswiki, 10 from dewiki.
  • Among the 41 IP block, 15 from the US. When breaking down by wiki, 25 from enwiki.

Caveat: Since the the error messages in MobileWikiAppCreateAccount table are translated, any message that contain the word "IP" will be taken as a IP block error message. Also, since we didn't send block ID/wiki user ID to the eventlogging table, we can only join the eventlogging data with Ipblocks table and Logging table by IP addresses to check whether an error message is an IP block.

JMinor claimed this task.Apr 9 2019, 5:25 PM
JMinor moved this task from Product Backlog to PM-Backlog on the Wikipedia-iOS-App-Backlog board.
JMinor closed this task as Resolved.May 31 2019, 5:54 PM
JMinor awarded a token.