I just experienced this myself on testwiki with this edit. Looking from mwmaint1002:

kharlan@mwmaint1002:~$ mwscript eval.php testwiki
> $trackerFactory = MediaWiki\MediaWikiServices::getInstance()->get( 'GrowthExperimentsNewcomerTaskTrackerFactory' );

> $tracker = $trackerFactory->getTracker( User::newFromID( 106361 ) );

> $tracker->getTrackedPageIds();

> print_r( $tracker->getTrackedPageIds() );
Array
(
)

Oh. As @Tgr suspected, the code changed for constructing the URL. Instead of a link to Special:Homepage/newcomertask/{pageID} I see a link like https://test.wikipedia.org/wiki/Donna_Ida?geclickid=2e9c8290q0gof1efgm1f01l6s7uqrh0u&getasktype=copyedit. If you advance the pager, then the correct URL is constructed in the module.

Change 636746 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Suggested edits: Include page ID with task preview data

https://gerrit.wikimedia.org/r/636746

The problem originated in code that exported the data from the server-side to render in the suggested edits module (the first task that shows up). That data includes things like page title, task type, difficulty, etc but it did not include page ID. On the client-side, when we construct the edit cards that are shown in the suggested edits module, our code checks to see if there is a page ID in the data passed to that component. If there is one, the URL for the card is rewritten to use Special:Homepage/newcomertask/{pageID}. That URL is how we can then track that a user visited the page in the suggested edits flow so we can later apply the edit tag.

Other cards in the suggested edits module are not affected, just the first one that the user sees. So in practical terms this means that from the time that 314397c73af7 was in production, clicks to the first card in the suggested edits module do not have the edit tag applied. It appears that this change went live with 1.36.0-wmf.9 on 2020-09-15 per the tag on T258024.

Thanks for looking into this quickly, @kostajh. I listed some other questions in the task description that we will want to know.

Ugh, sorry about this :( One thing we could maybe improve (beyond tests, although instrumentation seems hard to test) is have some sort of production flag. (Or maybe not even a proper flag, just export to the javascript whether we use local search or not.) We have a bunch of hacks for using a different wiki's search engine, pretending pages exist when they don't, etc, and those are useful but also make it harder to be strict about data formats.

We can probably recover the edits by following the same logic the extension would have followed if it weren't for the error: collect all the task-impression events with position = 0, and query all edits made by the same user on the same article within 7 days. There is no way to account for JS/PHP differences (e.g. users with an ad blocker that blocks all frontend instrumentation), but apart from that this would be fairly accurate.

I don't think there's any technical reason for not adding a tag retroactively (for manual tags there's even an API for it) but we'd have to write the code for it.

Change 636746 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Suggested edits: Include page ID with task preview data

https://gerrit.wikimedia.org/r/636746

Change 636787 had a related patch set uploaded (by Gergő Tisza; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.14] Suggested edits: Include page ID with task preview data

https://gerrit.wikimedia.org/r/636787

Scheduled for the Wednesday morning backport window.

Change 636787 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.14] Suggested edits: Include page ID with task preview data

https://gerrit.wikimedia.org/r/636787

Mentioned in SAL (#wikimedia-operations) [2020-10-28T18:40:02Z] <tgr@deploy1001> Synchronized php-1.36.0-wmf.14/extensions/GrowthExperiments/includes/HomepageModules/SuggestedEdits.php: Backport: [[gerrit:636787|Suggested edits: Include page ID with task preview data (T266600)]] (duration: 00m 59s)

@Dyolf77_WMF confirmed that he did eight edits and got eight tags on arwiki.

@Dyolf77_WMF confirmed that he did eight edits and got eight tags on arwiki.

3 more edits today on ar wikipedia and all are tagged.

Checked betalabs and testwiki - did not find any issues.