Page MenuHomePhabricator

Capture experiment version in all experiment-related analytics events
Closed, ResolvedPublic5 Estimated Story Points

Description

Background

We generate an experiment version in order to invalidate the browser-local exposure event registry. The experiment version could also help us understand the flow of an experiment config through the system and/or when a new version of the same experiment starts sending events. This is particularly important for an investigation like T421152: Could not hoist data into experiment.subject_id for event, where it's possible that an old/bad config has been cached somewhere.

AC

  • Capture experiment version in all experiment-related analytics events

Notes

Here's the set of steps that you're likely going to take to get this Done™:

  1. Update the product_metrics/experiment fragment to include the optional experiment.version field
  2. Update ConfigsFetcher to read the version field from the TK UI API response if it's present, otherwise calculate the version internally
  3. Update ResourceLoaderHooks::getExperimentConfigs() to use the version from ConfigsFetcher
  4. Update the PHP SDK to fill the experiment.version field
  5. Update the JS SDK to fill the experiment.version field

Event Timeline

Thank you for tagging this task with good first task for Wikimedia newcomers!

Newcomers often may not be aware of things that may seem obvious to seasoned contributors, so please take a moment to reflect on how this task might look to somebody who has never contributed to Wikimedia projects.

A good first task is a self-contained, non-controversial task with a clear approach. It should be well-described with pointers to help a completely new contributor, for example it should clearly point to the codebase URL and provide clear steps to help a contributor get set up for success. We've included some guidelines at https://phabricator.wikimedia.org/tag/good_first_task/ !

Thank you for helping us drive new contributions to our projects <3

phuedx set the point value for this task to 5.

While investigating this task, I noticed that exposure_version is currently computed in ResourceLoader\Hooks::getExposureVersion().

For the fallback case ("calculate the version internally"), should experiment.version reuse this existing calculation, or is a different versioning strategy intended?

For the fallback case ("calculate the version internally"), should experiment.version reuse this existing calculation, or is a different versioning strategy intended?

Good spot. Yes. exposure_version can be used as the fallback value.

Change #1298112 had a related patch set uploaded (by Vanshika11; author: Vanshika11):

[mediawiki/extensions/TestKitchen@master] TestKitchen: Capture experiment version in analytics events

https://gerrit.wikimedia.org/r/1298112

@Vanshika: Would you like to have a go at updating the product_metrics/experiment schema fragment?

@phuedx Yes, I'd be happy to take a look at updating the product_metrics/experiment schema fragment.

@Vanshika: Have you had a chance to take a look at updating the schema fragment?

@phuedx Yes, I have. I took some time to explore the schemas-event-secondary repository and identified the product_metrics/experiment schema fragment. I'm working on updating it to include experiment.version and will prepare a patch.

I've updated the product_metrics/experiment schema fragment by adding experiment.version and bumping the fragment version to 2.1.0. Before I open the MR, should I keep the change limited to the fragment, or would you also like me to update the base schemas (web/base, app/base, and base_with_ip) to reference the new fragment version?

I've updated the product_metrics/experiment schema fragment by adding experiment.version and bumping the fragment version to 2.1.0. Before I open the MR, should I keep the change limited to the fragment, or would you also like me to update the base schemas (web/base, app/base, and base_with_ip) to reference the new fragment version?

Yes please!

@Vanshika I'm precisely working on adding another new field to the base schemas (phase_index) via https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary/-/merge_requests/127 and I have seen you are working on adding version to the same ones. Would you want to unify efforts and add also phase_index as a new field to your MR? Then, happy to close mine and keep only yours to add both new fields

Thanks! @Sfaci Sure, I'm happy to combine both changes into this MR. I'll add phase_index as well and update the MR accordingly.

Now that we have a new version of the web/base schema, 2.1.0, the corresponding TestKitchen MR should be updated accordingly. The 2.0.0 version will be still there, and we should use now 2.1.0 where version has been included. Also keep in mind that https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TestKitchen/+/1302171 (where phasex_index is being added also as a new field) will be doing the same

I've updated the corresponding TestKitchen Gerrit change to use schema version 2.1.0. I've rebased it on the latest master and uploaded a new patchset. It's ready for review.

Change #1298112 merged by jenkins-bot:

[mediawiki/extensions/TestKitchen@master] TestKitchen: Capture experiment version in analytics events

https://gerrit.wikimedia.org/r/1298112