Background
I just revisited T313273 because I kept suspecting that the problem I found might not fully explain the weird data.
It turns out we still see the weirdly imbalanced data even when we factor out diff.wikimedia.org and wikimediafoundation.org, which are the only ones that would suffer significantly with the bug in tracking non-mainspace previews. The implied clickthrough rates are much more plausible than before, but it's still strange that they're so different for touch and non-touch devices.
| device_type | previews | pageviews | implied_clickthrough_rate |
|---|---|---|---|
| touch | 741 | 120 | 16% |
| non-touch | 12792 | 102 | 0.80% |
(Numbers are from the past 4 weeks, filtering out the Wikimedia sites and known test sites.)
This effect persists even when you look at individual sites:
| device_type ➡ | non-touch | non-touch | touch | touch |
|---|---|---|---|---|
| website ⬇ | previews | pageviews | previews | pageviews |
| framablog.org | 1058 | 7 | 31 | 6 |
| lumion.pl | 652 | 0 | 46 | 24 |
| stehn-online.de | 2071 | 3 | 8 | 3 |
| swa.co.id | 225 | 4 | 14 | 2 |
| xpressenglish.com | 687 | 4 | 55 | 6 |
Problem
I'm pretty sure I know why this is happening. When you encounter a Wikipedia Preview–enabled link on mobile, you essentially have only one option for clicking through to Wikipedia: clicking on the "read more on Wikipedia" link in the preview, which results in a pageview properly tagged with our wprov tag.
However, when you encounter a Wikipedia Preview–enabled link on desktop, you have two options for clicking through: click on the link itself or click on the "read more on Wikipedia" link after hovering. The first is much easier, even if you've opened and read the preview—your cursor is already hovered on that link. But that route means you follow a URL without the wprov tag!
So the source of the weird data is probably that we are failing to record most of the pageviews on non-touch devices.
Solution
The best solution is probably for the library to add the wprov tag to the in-page link once a pop-up opens (this ensures that we don't count clicks on Wikipedia links that don't have Wikipedia Preview enabled or when the user clicks the link without waiting for the pop-up to open, although these are probably pretty minor issues).
Alongside this, I recommend we create a new "version" of the tag (e.g. wppw2) so we can track which sites have rolled out this and previous instrumentation fixes. Note, however, that this will require updating the ETL job to capture the instrumentation version, which requires an annoying amount of work.