Description
Instrumentation is what turns Frodo's KR targets into something we can actually verify. The PP3.4 KR calls for a 2x increase in human views within 4 weeks of the Q3 release, measured against the existing developer portal baseline, with a follow-on KR (PP3.4b) tracking organic growth in new API users over time. Both depend on counting human traffic accurately. This means both having an accurate baseline, and ensuring that we are able to compare future performance against that baseline in and apples-to-apples manner. The later is particularly challenging, given a potential changing landscape for supported instrumentation tooling and the desire for unified data products.
Part of that complication is that Frodo is a standalone site, outside the on-wiki pageview pipeline the rest of Wikimedia's properties rely on. Matomo, the tool currently used for portal analytics, is being retired due to lack of internal maintenance support and data retention that conflicts with WMF's privacy policy. That gap is why this document specifies a DIY approach built specifically for Frodo, which utilizes data pipelines connected to Test Kitchen for event capture.
Conditions of acceptance
- page_visit events from the current developer portal website are written to the web base pipeline in Test Kitchen
- Contextual attributes are appropriately set
- Because content_language will be set as a contextual attribute, the uri_path provided in the event itself should have the language stripped out for normalization purposes. For example: https://developer.wikimedia.org/es/build-tools/ becomes https://developer.wikimedia.org/build-tools/ and the 'es' language is stored in content_language
- Deploy the site after the change is made. Recommend doing this via a pair programming session with @bd808 so we can learn the ropes. Deploy instructions are here: https://wikitech.wikimedia.org/wiki/Developer.wikimedia.org
Implementation details
The developer portal is currently a static site. This means that the scripts will need to be created, then the site rebuilt, so that the JS triggers on every page.
The events themselves are javascript triggers. This, by default, excludes the majority of bot traffic as most bots do not run javascript. Additional post-processing will occur within the data lake to also remove known bot user-agents and other heuristics.
Implementation spec: https://docs.google.com/spreadsheets/u/0/d/14f3KPhOtRlDpJe2CcXCMwir8gX17n6wSv7fEHAzKjbQ/edit
- Note: only page_visit instrumentation should be implemented in the current portal, rest are not needed.
For the current version of the developer portal, we only care about tracking basic page views so that we may have an apples-to-apples comparison.
Wikimedia25 site implementation for context/comparison:
- wikipedia25-years-of-wikipedia...useAnalytics.ts?ref_type=heads#L42 (instrumentation spec)
- https://gitlab.wikimedia.org/bearloga/new-external-path-experiment-analysis/-/merge_requests/1
Web base schema context (aka: Test Kitchen events)