Page MenuHomePhabricator
Create Task
Maniphest T185870

Monitor registration rates to make sure captcha changes have no negative effects
Closed, ResolvedPublic
Actions

Description

The AICaptcha project involves various changes to the registration page (initially a popup, as shown in T183991#3904344, also a size increase of ~50K gzipped due to OOUI being pulled in; later the captcha mechanism itself might be changed). We want to make sure that does not have a significant negative effect to new user registrations.

Things to watch:

For T186244: Deploy AICaptcha data collection, we don't want registration numbers to be significantly negatively impacted (people getting scared of the warning? some kind of JS error in the collection? some small effect due to the extra page size is probably unavoidable), and it would be nice to get more information about the performance impact (will be useful for T85853).

Later, we might experiment with modified captcha behavior, and want to track captcha and spambot stats.

Details

SubjectRepoBranchLines +/-
Fix broken returnTo handlingmediawiki/extensions/Campaignswmf/1.31.0-wmf.20+1 -1
Fix broken returnTo handlingmediawiki/extensions/Campaignsmaster+1 -1
Customize query in gerrit

Related Objects

Event Timeline

Tgr created this task.Jan 29 2018, 5:53 AM
Tgr mentioned this in T183869: Collect AICaptcha data from WikimediaEvents extension.
Tgr added a comment.Jan 29 2018, 6:04 AM
This comment was removed by Tgr.
Tgr updated the task description. (Show Details)Jan 29 2018, 6:04 AM
Tgr added a subscriber: nshahquinn-wmf.Jan 29 2018, 6:12 AM

Per @Neil_P._Quinn_WMF the ServerSideAccountCreation schema stats are the ones to watch. Also maybe some manual NavigationTiming queries for the registration page specifically.

Tgr added a comment.Jan 29 2018, 6:34 AM

See also T152219: Statistics on Captcha success/failure rate.

nshahquinn-wmf claimed this task.Jan 30 2018, 8:32 PM
nshahquinn-wmf added a project: Contributors-Analysis.Jan 30 2018, 10:47 PM
Tgr mentioned this in T186244: Deploy AICaptcha data collection.Feb 1 2018, 6:23 PM
Tgr updated the task description. (Show Details)Feb 1 2018, 7:17 PM
Tgr updated the task description. (Show Details)
Tgr updated the task description. (Show Details)Feb 1 2018, 7:25 PM
Tgr updated the task description. (Show Details)Feb 1 2018, 7:50 PM
Peter added a project: Performance-Team.Feb 6 2018, 12:13 PM
Tgr added a comment.EditedFeb 8 2018, 6:36 AM

It seems like we broke the ServerSideAccountCreation dashboard: https://grafana.wikimedia.org/dashboard/db/eventlogging-schema?orgId=1&var-schema=ServerSideAccountCreation&from=now-7d&to=now
Web registrations via authevent are unchanged so I don't think there is any real effect on registrations.

ssac.png (367×695 px, 68 KB)

Tgr added a comment.Feb 8 2018, 6:56 AM

Somehow related to https://gerrit.wikimedia.org/r/#/c/407989/ ; the old schema has ten times the event count of the new one.

Tgr added a subscriber: Halfak.Feb 8 2018, 7:02 AM

Caused by this schema change not being reflected in the logging code. @Neil_P._Quinn_WMF @Halfak would you prefer rolling back that schema change or updating the code to match it?

Groovier subscribed.Feb 8 2018, 2:46 PM

Web page test statistics before and after data collection code deployment:
Settings: Chrome on a Cable connection from Dulles, VA
Before:

before.png (137×823 px, 34 KB)

After:
after.png (112×834 px, 29 KB)

Observations:

  1. Load time: To load all content objects (CSS, JavaScript, images etc) it takes 0.137s extra post-deployment
  2. Document complete time: The time when browser finishes loading all content objects except object content that are triggered by JavaScript execution -It takes 0.137s more after deployment.
  3. Fully Loaded time (time after which there is no more data transfer between server and browser): Takes 0.136s more after deployment
Halfak added a comment.Feb 8 2018, 4:51 PM

Re. rolling back, code should be using the specific revision ID that corresponds to the EventLogging data -- not the most recent revision of the schema page.

Tgr added a comment.Feb 8 2018, 4:57 PM
In T185870#3956033, @Halfak wrote:

Re. rolling back, code should be using the specific revision ID that corresponds to the EventLogging data -- not the most recent revision of the schema page.

The problem was caused when we updated the schema page and the logging code (added an isApi field to both), and did not check whether parent revision of the schema matched the logic in the parent revision of the commit.

I'll just undo that schema change for now.

gerritbot subscribed.Feb 8 2018, 5:03 PM

Change 409079 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/Campaigns@master] Fix broken returnTo handling

https://gerrit.wikimedia.org/r/409079

gerritbot added a project: Patch-For-Review.Feb 8 2018, 5:03 PM
gerritbot added a comment.Feb 8 2018, 5:23 PM

Change 409079 merged by jenkins-bot:
[mediawiki/extensions/Campaigns@master] Fix broken returnTo handling

https://gerrit.wikimedia.org/r/409079

gerritbot added a comment.Feb 8 2018, 5:25 PM

Change 409083 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/Campaigns@wmf/1.31.0-wmf.20] Fix broken returnTo handling

https://gerrit.wikimedia.org/r/409083

ReleaseTaggerBot added a project: MW-1.31-release-notes (WMF-deploy-2018-02-13 (1.31.0-wmf.21)).Feb 8 2018, 6:00 PM
gerritbot added a comment.Feb 8 2018, 7:05 PM

Change 409083 merged by jenkins-bot:
[mediawiki/extensions/Campaigns@wmf/1.31.0-wmf.20] Fix broken returnTo handling

https://gerrit.wikimedia.org/r/409083

Stashbot subscribed.Feb 8 2018, 7:17 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-08T19:17:22Z] <catrope@tin> Synchronized php-1.31.0-wmf.20/extensions/Campaigns/CampaignsSecondaryAuthenticationProvider.php: T185870 (duration: 01m 13s)

ReleaseTaggerBot edited projects, added MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)); removed MW-1.31-release-notes (WMF-deploy-2018-02-13 (1.31.0-wmf.21)).Feb 8 2018, 8:00 PM
Tgr added a comment.Feb 10 2018, 8:05 AM

ServerSideAccountCreation stats have recovered, and the other stats never showed any problem. So it seems like there was no negative impact of deploying the data collection code.

• Imarlier moved this task from Inbox, needs triage to Radar on the Performance-Team board.Feb 12 2018, 9:33 PM
• Imarlier edited projects, added Performance-Team (Radar); removed Performance-Team.
Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.Feb 13 2018, 1:15 AM
nshahquinn-wmf added a comment.Feb 23 2018, 11:52 AM
In T185870#3960184, @Tgr wrote:

ServerSideAccountCreation stats have recovered, and the other stats never showed any problem. So it seems like there was no negative impact of deploying the data collection code.

Good to know! I'm sorry that I wasn't around to help out.

Groovier moved this task from Backlog to Done on the AICaptcha board.Feb 25 2018, 11:36 AM
nshahquinn-wmf closed this task as Resolved.Mar 6 2018, 8:19 AM
nshahquinn-wmf reassigned this task from nshahquinn-wmf to Tgr.
nshahquinn-wmf moved this task from Backlog to Radar on the Contributors-Analysis board.Mar 30 2018, 10:30 AM
sbassett mentioned this in T255208: Catalog and evaluate methods of analysis for Wikimedia captcha performance.Jun 11 2020, 9:45 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 11 2020, 10:10 PM
Restricted Application added a project: Product-Analytics. · View Herald TranscriptJun 11 2020, 10:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL