Page MenuHomePhabricator

Monitor registration rates to make sure captcha changes have no negative effects
Closed, ResolvedPublic

Description

The AICaptcha project involves various changes to the registration page (initially a popup, as shown in T183991#3904344, also a size increase of ~50K gzipped due to OOUI being pulled in; later the captcha mechanism itself might be changed). We want to make sure that does not have a significant negative effect to new user registrations.

Things to watch:

For T186244: Deploy AICaptcha data collection, we don't want registration numbers to be significantly negatively impacted (people getting scared of the warning? some kind of JS error in the collection? some small effect due to the extra page size is probably unavoidable), and it would be nice to get more information about the performance impact (will be useful for T85853).

Later, we might experiment with modified captcha behavior, and want to track captcha and spambot stats.

Event Timeline

Tgr added a comment.Jan 29 2018, 6:04 AM
This comment was removed by Tgr.
Tgr updated the task description. (Show Details)Jan 29 2018, 6:04 AM

Per @Neil_P._Quinn_WMF the ServerSideAccountCreation schema stats are the ones to watch. Also maybe some manual NavigationTiming queries for the registration page specifically.

Tgr updated the task description. (Show Details)Feb 1 2018, 7:17 PM
Tgr updated the task description. (Show Details)
Tgr updated the task description. (Show Details)Feb 1 2018, 7:25 PM
Tgr updated the task description. (Show Details)Feb 1 2018, 7:50 PM
Tgr added a comment.EditedFeb 8 2018, 6:36 AM

It seems like we broke the ServerSideAccountCreation dashboard: https://grafana.wikimedia.org/dashboard/db/eventlogging-schema?orgId=1&var-schema=ServerSideAccountCreation&from=now-7d&to=now
Web registrations via authevent are unchanged so I don't think there is any real effect on registrations.

Tgr added a comment.Feb 8 2018, 6:56 AM

Somehow related to https://gerrit.wikimedia.org/r/#/c/407989/ ; the old schema has ten times the event count of the new one.

Tgr added a subscriber: Halfak.Feb 8 2018, 7:02 AM

Caused by this schema change not being reflected in the logging code. @Neil_P._Quinn_WMF @Halfak would you prefer rolling back that schema change or updating the code to match it?

Web page test statistics before and after data collection code deployment:
Settings: Chrome on a Cable connection from Dulles, VA
Before:


After:

Observations:

  1. Load time: To load all content objects (CSS, JavaScript, images etc) it takes 0.137s extra post-deployment
  2. Document complete time: The time when browser finishes loading all content objects except object content that are triggered by JavaScript execution -It takes 0.137s more after deployment.
  3. Fully Loaded time (time after which there is no more data transfer between server and browser): Takes 0.136s more after deployment
Halfak added a comment.Feb 8 2018, 4:51 PM

Re. rolling back, code should be using the specific revision ID that corresponds to the EventLogging data -- not the most recent revision of the schema page.

Tgr added a comment.Feb 8 2018, 4:57 PM

Re. rolling back, code should be using the specific revision ID that corresponds to the EventLogging data -- not the most recent revision of the schema page.

The problem was caused when we updated the schema page and the logging code (added an isApi field to both), and did not check whether parent revision of the schema matched the logic in the parent revision of the commit.

I'll just undo that schema change for now.

Change 409079 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/Campaigns@master] Fix broken returnTo handling

https://gerrit.wikimedia.org/r/409079

Change 409079 merged by jenkins-bot:
[mediawiki/extensions/Campaigns@master] Fix broken returnTo handling

https://gerrit.wikimedia.org/r/409079

Change 409083 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/Campaigns@wmf/1.31.0-wmf.20] Fix broken returnTo handling

https://gerrit.wikimedia.org/r/409083

Change 409083 merged by jenkins-bot:
[mediawiki/extensions/Campaigns@wmf/1.31.0-wmf.20] Fix broken returnTo handling

https://gerrit.wikimedia.org/r/409083

Mentioned in SAL (#wikimedia-operations) [2018-02-08T19:17:22Z] <catrope@tin> Synchronized php-1.31.0-wmf.20/extensions/Campaigns/CampaignsSecondaryAuthenticationProvider.php: T185870 (duration: 01m 13s)

Tgr added a comment.Feb 10 2018, 8:05 AM

ServerSideAccountCreation stats have recovered, and the other stats never showed any problem. So it seems like there was no negative impact of deploying the data collection code.

Imarlier moved this task from Inbox to Radar on the Performance-Team board.Feb 12 2018, 9:33 PM
Imarlier edited projects, added Performance-Team (Radar); removed Performance-Team.

ServerSideAccountCreation stats have recovered, and the other stats never showed any problem. So it seems like there was no negative impact of deploying the data collection code.

Good to know! I'm sorry that I wasn't around to help out.

Groovier moved this task from Backlog to Done on the AICaptcha board.Feb 25 2018, 11:36 AM
Neil_P._Quinn_WMF closed this task as Resolved.Mar 6 2018, 8:19 AM
Neil_P._Quinn_WMF reassigned this task from Neil_P._Quinn_WMF to Tgr.