Page MenuHomePhabricator

Newcomer tasks: edit tag not applying to all edits
Closed, ResolvedPublic


Starting on 2020-10-21, we noticed that in arwiki, cswiki, and testwiki that the "Newcomer task" edit tag is not being applied to all suggested edits. Though this was first observed on 2020-10-21, we don't know if that's when the issue started. According to tests so far, the tag may be being applied to half or fewer edits.

This spreadsheet contains a list of diffs made to investigate this, along with whether the tag was applied:

These diffs are all edits originating from clicking tasks on the newcomer homepage or in the post-edit dialog after completing an edit from the newcomer homepage.

Once we fix this, we'll want to know:

  • when it began
    • In production on September 15, 2020
  • whether it affected all wikis equally
    • yes
  • whether it affects both desktop and mobile
    • both
  • what share of edits it probably affects
    • we would have to analyze pre-September 15 data to figure out which proportion of edits originate from clicks to the first card seen when a user arrives at Special:Homepage.
  • whether those edits are recoverable so we can count them in our reporting
    • we can't retroactively apply edit tags but maybe we can reconstruct from other instrumentation data we have. For example, we could use is_oversample from, to find edits where is_oversample is true, where the newcomer task edit tag isn't present, and where the user ID belongs to a user who has suggested edits enabled. That wouldn't be 100% accurate but might be a good place to start.

Event Timeline

I just experienced this myself on testwiki with this edit. Looking from mwmaint1002:

kharlan@mwmaint1002:~$ mwscript eval.php testwiki
> $trackerFactory = MediaWiki\MediaWikiServices::getInstance()->get( 'GrowthExperimentsNewcomerTaskTrackerFactory' );

> $tracker = $trackerFactory->getTracker( User::newFromID( 106361 ) );

> $tracker->getTrackedPageIds();

> print_r( $tracker->getTrackedPageIds() );

Oh. As @Tgr suspected, the code changed for constructing the URL. Instead of a link to Special:Homepage/newcomertask/{pageID} I see a link like If you advance the pager, then the correct URL is constructed in the module.

Change 636746 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Suggested edits: Include page ID with task preview data

The problem originated in code that exported the data from the server-side to render in the suggested edits module (the first task that shows up). That data includes things like page title, task type, difficulty, etc but it did not include page ID. On the client-side, when we construct the edit cards that are shown in the suggested edits module, our code checks to see if there is a page ID in the data passed to that component. If there is one, the URL for the card is rewritten to use Special:Homepage/newcomertask/{pageID}. That URL is how we can then track that a user visited the page in the suggested edits flow so we can later apply the edit tag.

Other cards in the suggested edits module are not affected, just the first one that the user sees. So in practical terms this means that from the time that 314397c73af7 was in production, clicks to the first card in the suggested edits module do not have the edit tag applied. It appears that this change went live with 1.36.0-wmf.9 on 2020-09-15 per the tag on T258024.

Thanks for looking into this quickly, @kostajh. I listed some other questions in the task description that we will want to know.

Ugh, sorry about this :( One thing we could maybe improve (beyond tests, although instrumentation seems hard to test) is have some sort of production flag. (Or maybe not even a proper flag, just export to the javascript whether we use local search or not.) We have a bunch of hacks for using a different wiki's search engine, pretending pages exist when they don't, etc, and those are useful but also make it harder to be strict about data formats.

We can probably recover the edits by following the same logic the extension would have followed if it weren't for the error: collect all the task-impression events with position = 0, and query all edits made by the same user on the same article within 7 days. There is no way to account for JS/PHP differences (e.g. users with an ad blocker that blocks all frontend instrumentation), but apart from that this would be fairly accurate.

I don't think there's any technical reason for not adding a tag retroactively (for manual tags there's even an API for it) but we'd have to write the code for it.

Change 636746 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Suggested edits: Include page ID with task preview data

Change 636787 had a related patch set uploaded (by Gergő Tisza; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.14] Suggested edits: Include page ID with task preview data

Change 636787 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.14] Suggested edits: Include page ID with task preview data

Mentioned in SAL (#wikimedia-operations) [2020-10-28T18:40:02Z] <tgr@deploy1001> Synchronized php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/HomepageModules/SuggestedEdits.php: Backport: [[gerrit:636787|Suggested edits: Include page ID with task preview data (T266600)]] (duration: 00m 59s)

@Dyolf77_WMF confirmed that he did eight edits and got eight tags on arwiki.

@Dyolf77_WMF confirmed that he did eight edits and got eight tags on arwiki.

3 more edits today on ar wikipedia and all are tagged.

Checked betalabs and testwiki - did not find any issues.

MMiller_WMF closed this task as Resolved.EditedNov 24 2020, 5:47 AM

Thank you! This was fixed on Oct 28.