Instrument blocked account registration and edit attempts
Open, HighPublic
Actions

Assigned To

Authored By

	nettrom_WMF
	Mar 16 2022, 6:20 PM

Description

We've recently done work to blocks and their impacts, for example T298273 to understand block activity, and T292103 to estimate relay blocks impact on editing. We know that blocks might be set to restrict editing, but they can also limit account creation. As we're doing work on increasing account creation (see T293695), the latter can potentially have a negative impact. Our current instrumentation lets us know when account creation or editing succeeds, it does not allow us to understand that someone attempted an action but was stopped because a block is in effect.

More specifically, we're looking to instrument:

Account creation attempts from Special:CreateAccount that fail due to a block.
Edit attempts that fail due to a block.

We'll have to define a schema for this and might want to reuse parts of the user blocks change schema.

Details

Other Assignee: kzimmerman

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		mpopov	T303995 Instrument blocked account registration and edit attempts
Resolved	Feature	Sgs	T306018 Instrument blocked account registration
Resolved		• ppelberg	T310390 Instrument blocked edit attempts
Resolved	PRODUCTION ERROR	DLynch	T320938 mediawiki.editattempt_block: '.country_code' should be string
Resolved		phuedx	T344486 Restrict interface parameter of wikimediaeventsblockededit API query module

Event Timeline

nettrom_WMF created this task.Mar 16 2022, 6:20 PM

At the same time we have seen large increases in blocks in English Wikipedia, we have also started to see declines in new active editors. See, for example, https://phabricator.wikimedia.org/T298273#7713338 for a graph of new active editors on English Wikipedia. We have also seen decreases in our high-level metrics; in January we noted that new active editors dropped below FY19-20 levels (https://www.mediawiki.org/w/index.php?title=File:January_2022_Wikimedia_movement_metrics.pdf&page=8).

@MMiller_WMF @Niharika I'm not sure where Product ownership around registration would be for this, but I see this instrumentation as being high priority so we can understand better the extent to which blocks play a role in preventing potentially productive editors from registering.

• mewoph moved this task from Inbox to Needs Discussion on the Growth-Team board.Mar 16 2022, 9:55 PM

Iflorez subscribed.Mar 21 2022, 9:36 PM

cc @DMburugu @MShilova_WMF for help with scheduling and prioritization.

If we're doing this instrumentation, it would also be nice to instrument account creation & success in staying logged-in. Per T267273: [arwiki] Submitting a POST on a form redirected to immediately after account creation sometimes logs user out and T299193: MediaWiki login failure due to race condition with session cookie, it appears that a sizable percentage of users who create accounts are immediately logged-out. We have instrumented the GrowthExperiments' WelcomeSurvey to capture this, but would be better to have the instrumentation directly in MediaWiki core, IMHO.

mpopov moved this task from Triage to Tracking on the Product-Analytics board.Apr 5 2022, 5:19 PM

There isn't really such a thing as an account creation attempt failing due to a block, strictly speaking; when your IP is blocked, Special:CreateAccount shows a block message and you can't even attempt account creation. I assume we want to instrument that error message. Is any extra information needed, to deduplicate (since this is something happening on a GET request, so the user might reload etc)?

The same goes for edit attempts, mostly, although it depends on the editor - some don't let you save if you are blocked, some only detect the block when trying to save.

Instrumenting account creation and editing is technically unrelated and both are nontrivial so they merit separate tasks. (Probably multiple ones for editing - VE, EditPage form-based, and the mobile wikitext editor? - but I'm not the best authority on that.) Filed T306018: Instrument blocked account registration for account creation.

In T303995#7803235, @kostajh wrote:

If we're doing this instrumentation, it would also be nice to instrument account creation & success in staying logged-in.

We do track account creation errors in Grafana, but we missed blocks, I think, since technically you can't even start account creation when blocked. We should probably fix that.

There's also a BlockNotices Grafana panel (belongs to Editing, maybe?), adding account creation blocks there would make sense as well.

Staying logged in after account creation is a corner case and pretty tricky to track so I don't think there's much overlap there.

MNeisler subscribed.Apr 14 2022, 10:21 AM

• ppelberg subscribed.Apr 14 2022, 10:23 PM

MMiller_WMF added a subtask: T306018: Instrument blocked account registration.Apr 19 2022, 8:55 PM

In T303995#7851103, @Tgr wrote:

In T303995#7803235, @kostajh wrote:

If we're doing this instrumentation, it would also be nice to instrument account creation & success in staying logged-in.

We do track account creation errors in Grafana, but we missed blocks, I think, since technically you can't even start account creation when blocked. We should probably fix that.

There's also a BlockNotices Grafana panel (belongs to Editing, maybe?), adding account creation blocks there would make sense as well.

Staying logged in after account creation is a corner case and pretty tricky to track so I don't think there's much overlap there.

Hmm, sorry I guess I misread "blocked" as "prevented" or "failed" and not the specific connatation that "block" has in MW software. You're right, we should do the instrumentation for T299193: MediaWiki login failure due to race condition with session cookie in that task or in some other task, but not here.

For editing, I guess we'd want to settle on a definition of "edit attempt" for this. If we're okay with "viewed an edit page while blocked", it's probably doable. If we want it to be comparable to other logging of edit attempts, we'd probably want to do all the logs from the client-side.

On desktop, you get shown the edit form in read-only mode in WikiEditor and also in VisualEditor modes along with an editnotice explaining that you're blocked, but you don't get any sign before you click the edit tab.

On mobile your edit button has a little padlock to show you're blocked, and then if you tap if you don't even see the edit form, just a popup saying you're blocked. This might make it difficult to compare to the rate on desktop, since it gives you a sign you're blocked before you interact in any way signaling your intentions.

Not sure if DiscussionTools is included in this? It does a similar "click the reply link, you get a popup saying you're blocked" thing.

I'm not aware of an editing interface that'd wait until the save attempt to tell you you're blocked, but I can't claim to have gone looking. Probably e.g. any gadgets which would attempt edits via the API, or maybe the GrowthExperiments which don't go through the edit page.

This could maybe be extracted from existing EditAttemptStep logging, insofar as it logs with a user_id field and presumably some creative query-writing could turn up whether a given user_id was blocked at the time of an edit attempt (@MNeisler could maybe comment on whether the tables exist to do this). EditAttemptStep is sampled, of course, and I don't know if this would be occurring at sufficient rates to give us useful data. It also wouldn't turn up edit attempts from IP-blocked users.

• ppelberg added projects: Editing-team, VisualEditor.May 2 2022, 11:39 PM

• ppelberg edited projects, added Editing-team (Tracking); removed Editing-team.

• ppelberg moved this task from Backlog to Dependent on others on the Editing-team (Tracking) board.

• ppelberg moved this task from To Triage to Triaged on the VisualEditor board.

SCherukuwada subscribed.May 19 2022, 6:41 AM

Next Step

The Editing Team will be discussing this ticket and the questions @DLynch posed in T303995#7886145 during the team meeting we have scheduled for this Wednesday 8 June.

• ppelberg claimed this task.Jun 7 2022, 11:07 PM

In T303995#7984557, @ppelberg wrote:

Next Step

The Editing Team will be discussing this ticket and the questions @DLynch posed in T303995#7886145 during the team meeting we have scheduled for this Wednesday 8 June.

Today, the Editing Team discussed how we might go about tracking edit attempts that fail due to a block.

The outcomes of that discussion are below.

Next step
I think the next step on this task is for the person/people who will be using this data to make decisions (I think that might be Kate?) to answer the "Open questions" below.

With the above in mind, I'm boldly assigning this task over to @kzimmerman. Although, please let me know if there is someone else who we ought to direct these questions to.

8 June Editing Team Discussion

Open Questions

1. What – if any – concerns and/or questions do you think we ought to address before moving forward with what is described in the "Editing Team Recommendation(s)" section below?
2. Assuming y'all are okay with doing what the "Editing Team Recommendation(s)" section proposes, when would you value this instrumentation being implemented?

Editing Team Recommendation(s)

Add the instrumentation required to log whenever someone clicks/taps an affordance to open an editing interface and they are prevented from progressing because the account/IP address they are using to access Wikipedia has been blocked.

Do NOT implement page view logging considering the unreliability of this data
- "Unreliable" in the sense that: 1) many page views are served to people who are logged out, 2) people who are logged out are served a cached version of the page, 3) cached page versions are not aware whether the person viewing the page is blocked or not...these three points sum into this logging to produce data that is misleading and therefor unreliable..

Understandings

In all editing interfaces, people who have been blocked are prevented from doing anything other than opening the editing interface at which point they are shown a notice of some kind that informs them they are blocked from editing.
There are two potential ways we could track edit attempts that fail due to a block:
- 1) We could log an event whenever someone clicks/taps an affordance to open an editing interface and they are prevented from progressing because the account/IP address they are using to access Wikipedia has been blocked.
  - "Editing interface" in this context refers to: the source and visual editors people access via MobileFrontend, 2010 wikitext editor (desktop), VE (desktop), New Wikitext Editor (mobile), Reply Tool (mobile and desktop), New Topic Tool (mobile and desktop).
- 2) We could hypothetically log when MobileFrontend serves a page to someone who has been blocked from editing.
  - Note: we're naming MobileFrontend here because it is the only interface that changes the way pages in read mode are rendered to inform people who are blocked that they are not able to edit Wikipedia.
Any analyses we do ought to name and consider the fact that many people who are blocked from editing who are accessing the site through MobileFrontend may NOT attempt to initiate an edit because the interface suggests to them – by way of showing a "🔒" next to the top-most edit pencil – that they will be prevented from doing so.

• ppelberg mentioned this in T310390: Instrument blocked edit attempts.Jun 10 2022, 7:18 PM

One more thing I remembered about edit attempts, that doesn't seem to have been mentioned yet – it is possible for a user to be blocked while they're editing a page (after they opened the editor), in which case they'll only see a message about this when they attempt publishing, and we do log when this happens in the EditAttemptStep schema (save_failure_message containing blocked, there are several different error codes depending on the type of the block). This is uncommon, and probably very rare for the wide VPN blocks that (I assume) prompted this research, but it does happen.