When we original set up preview tracking for the KaiOS app in 2020, we were told that the pageview X-Analytics header was no longer used for pageview counting (T244547#5858544). Is there reason to think that has changed?

I don't know what Nuria & Francisco were talking about because 3 years later isTaggedPageview is alive and well:

private boolean isAppPageview(WebrequestData data) {
final String rawXAnalyticsHeader = data.getRawXAnalyticsHeader();
final String uriPath = data.getUriPath();
final boolean isTaggedPageview = Utilities.getValueForKey(rawXAnalyticsHeader, "pageview").trim().equalsIgnoreCase("1");

// See analytics request for KAiOS pageviews to be turned into events here:
// https://phabricator.wikimedia.org/T244547
final boolean isKAiOSPageview = Utilities.stringContains(uriPath, PageviewDefinition.URI_PATH_REST_API)
    && Utilities.stringContains(uriPath, "/page/mobile-sections/");

return (data.isAppUserAgent()) && (isTaggedPageview || isKAiOSPageview);

}

@Milimetric @mforns: can you please provide current day guidance about using pageview=1 in X-Analytics header for tagging pageviews in apps? Seems there was conflicting guidance 3 years ago.

A few of points of clarification:

• The app's codebase was actually updated to use the new API a full year ago (T272091)
• The KaiStore currently has version 1.6.0
• The JioStore currently has version 1.8.1.0

It's not totally clear to me if those versions include the code for the new API. According to the version tags in Git, they don't. However, from what @SBisson and @hueitan are saying, it sounds like those versions of the apps are still loading pages correctly, so maybe they do.

Then I don't know what's going on because on August 11, 2023 there were 3.5K 403-status requests made to /page/mobile-sections endpoint (consistent with our pageview counts up the endpoint getting decommissioned in July) and 0 requests made to /page/mobile-html endpoint.

In T344071#9096849, @mpopov wrote:

Then I don't know what's going on because on August 11, 2023 there were 3.5K 403-status requests made to /page/mobile-sections endpoint (consistent with our pageview counts up the endpoint getting decommissioned in July) and 0 requests made to /page/mobile-html endpoint.

Okay, we discussed this on Slack and it was probably that they had pre-release versions installed on their devices. Last year, an updated version 1.8.1.0 with the new API code was submitted to the JioStore, but it seems like they never released it.

So, the situation is that the code for the new API was written long ago, but hasn't been released on either store due to the difficulty of both partnerships 😞

Bummer :\

Thanks for talking with the team about it and documenting the situation, @nshahquinn-wmf! Okay, so the best thing we can do on our end at this point is update the pageview definition so that when (if ever?) the updated version of the app is finally released on either store, we'll be counting those pageviews.

@VirginiaPoundstone: So it sounds like there's no particular rush to get the pageview definition updated but it should still be done soon in case the app update finally goes live one of these days and that way we're not rushing to update the pageview definition while losing pageview counts.

@mpopov and @nshahquinn-wmf thank you for the research. I will reach out to partnerships to learn more about when and how to request the version updates to the stores.

This will remain a low priority until we get deadline and assurance the changes will be live on the store(s).

Question from @mpopov

can someone from your team [Data Products] please advise on pageview=1 tagging? Nuria said it was being phased out but as I demonstrated it’s still very much active in the pageview definition. If KaiOS app is updated to use that mechanism (which Inuka could have done in the first place but were advised against it by then-Analytics Engineering) and THAT’s the update that goes live (endpoint switch + pageview x-analytics tagging), then your team wouldn’t even need to touch pageview definition code (I think it would be counted automatically because it prioritizes pageview tag check over URI check

As @JAllemandou reminded me: T257860: Update PageviewDefinition to only include /api/rest_v1/page/mobile-html requests with X-Analytics: pageview=1 in pageviews

We discussed it and agreed that since KaiOS app has been switched to using the same endpoint as Android & iOS apps do and the definition/pipeline checks has been using the pageview=1 X-Analytics header thing for a while, the KaiOS app should be updated to also use this system.

If the app is updated and the version that releases includes both the endpoing switch AND the x-analytics header thing, there is no need to update the pageview definition code or make any kind of changes to the pipeline. @VirginiaPoundstone I will update the task description accordingly. You're welcome to either remove your team's tag or just keep this ticket on your team's radar/watchlist.

Although I think we'll still want to remove isKAiOSPageview from the definition at some point because eventually it will just be useless code that doesn't add any value, but that can be done separately.

Thanks @mpopov!

Change 953198 had a related patch set uploaded (by Joal; author: Joal):

[analytics/refinery/source@master] Remove special KaiOS App checks from pageview def

https://gerrit.wikimedia.org/r/953198

Added the patch above to clean-up pageview code - the lines of code I removed were checking deprecated behavior.

Change 953198 merged by jenkins-bot:

[analytics/refinery/source@master] Remove special KaiOS App checks from pageview def

https://gerrit.wikimedia.org/r/953198

joal opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/489

Update analytics webrequest refine job jar version

jebe merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/489

Update analytics webrequest refine job jar version