Page MenuHomePhabricator

Investigate malformed x_analytics_map[wprov] values
Closed, ResolvedPublic

Description

Description

Some observations were made on the quality of x_analytics_map[wprov]
Data shows many malformed/extended variants of wprov values such as:

yicw1%D8...
rarw1/static/...
sfla1https://...
sfti1%23…

This indicates wprov is often contaminated by appended text/URL fragments, which likely causes undercounting when matching exact values only (especially for external codes).

In this ticket, we need to :

  • investigate how much contaminated wprov data we have.
  • perform sanitization on the data.

Event Timeline

Snwachukwu claimed this task.
Snwachukwu added a subscriber: Miriam.

Change #1287508 had a related patch set uploaded (by Snwachukwu; author: Snwachukwu):

[analytics/refinery/source@master] Add Sanitizer to clean up wprov value of x-analytics.

https://gerrit.wikimedia.org/r/1287508

Change #1287909 had a related patch set uploaded (by Snwachukwu; author: Snwachukwu):

[analytics/refinery@master] Use SanitizeXAnalyticsWprovUDF to normalize x_analytics[wprov] values

https://gerrit.wikimedia.org/r/1287909

Change #1287508 merged by Snwachukwu:

[analytics/refinery/source@master] Add Sanitizer to clean up wprov value of x-analytics.

https://gerrit.wikimedia.org/r/1287508

Change #1287909 merged by Snwachukwu:

[analytics/refinery@master] Use SanitizeXAnalyticsWprovUDF to normalize x_analytics[wprov] values

https://gerrit.wikimedia.org/r/1287909

The changes to fix wprov values are merged and ready to be deployed. It would follow the DE deployment train.

Thank you @Snwachukwu just double checking, will this sanitization make the wprov data clean enough for metrics calculation purposes?

@Miriam Yes it would. It would keep them in the format defined on the Provenance page . It would remove any URL fragments, percent sign, paths etc that is added to the original wprov value.

Amazing, thank you so much!
@Khantstop you can add this to the documentation for IR2. Thanks so much everyone!

Ahoelzl triaged this task as High priority.