The recent refactor of the Java MPC to Kotlin (T401023: [XL] xLab Client Library: Convert to Kotlin and bring into app repo.) was a direct port wherein it functions as a core event client (i.e. the client submits an interaction). It has the event submission, config, context, and sampling components but it does not provide an experiment-shaped wrapper for events, nor instrument retrieval/event-send capabilities.

The task is to bring the Test Kitchen Kotlin library to functional parity with the Test Kitchen JavaScript + PHP (note that imminent changes introduced in T367034: TestKitchen Extension: Merge PHP MetricsPlatform client library will inform suggested Kotlin updates below) SDKs and current platform requirements (including future work to provide exposure logging T414729: Add Experiment#logExposure method to SDKs + curated contextual attributes T414726: Include curated set of contextual attributes with exposure events). Per discussion with team, this work will NOT be included at this time. We will port over exposure logging after it has been successfully implemented on web.

Requirements

Ideally all the SDKs should:

• expose core fundamental conceptual components:
  • entry point into the SDK (initialization, configuration)
  • event - a structured, atomic unit that defines/includes an action, payload, context, schema/stream
  • context - metadata, contextual attributes to decorate the event
  • instrument - emit events for measurement
  • experiment - represents variation and assignment
  • assignment - deterministic mapping of a subject (user, device, session) to a variant in an experiment
  • exposure - the moment a subject is exposed to a variant
  • submission - how events reach EventGate (buffering, queuing, batching, retrying, dropping)
    • implicit in this is state management (lifecycle) of events (when they can be created, queued, flushed, dropped)
  • validation - events conform as expected to specified schemas
• have a public set of APIs that lets a caller:
  • create a valid event
  • attach the required context to the event
  • reliably submit the event
• know how to fetch remote configs for instruments and experiments from the Test Kitchen UI

The JS/PHP SDKs in their current incarnation sit in between experiment enrollment and event logging by taking standardized enrollment results and turning them into correctly-shaped events. They model an Experiment-type object that knows an experiment name and assigned variant, how/where to log events, and how to attach context to events. They also (to be implemented in PHP) provide basic functionality for getting instruments and sending valid events for guardrail and product health metrics.

Kotlin Updates

In order for Kotlin to become more about event construction and contextual attributes, it needs:

1. Experiment equivalent for getting assignments, checking groups, sending events, overrides, etc
   • Experiment abstraction/wrapper/manager layer (i.e. given an experiment state and config, produce correctly-shaped experiment event and send it):
     • Experiment types (Experiment interface, Experiment object, UnenrolledExperiment, OverriddenExperiment)
       • Require dependencies - EventSubmitter, EventFactory, StatsFactory (or equivalent?), experiment config
       • OverriddenExperiments are a no-op - do not send events - OverriddenExperiment.send() should log message instead
       • Experiment config provides:
         • stream_name
         • schema_id
         • contextual_attributes
       • Experiment event payload contains:
         • enrolled
         • assigned
         • subject_id
         • sampling_unit
         • coordinator
     • ExperimentManager
       • maps enrollment results to experiment objects
       • sets experiment payload contract
       • returns UnenrolledExperiment, OverriddenExperiment, or Experiment object
       • enrich experiment config with stream_name, schema_id, contextual_attributes
     • EnrollmentRequest - match the Coordination builder output
     • EnrollmentResultBuilder - standardized experiment assignments/enrollments/overrides/information about active experiments during experiment enrollment sampling:
       • active_experiments
       • overrides
       • enrolled
       • assigned
       • subject_ids
       • sampling_units
       • coordinator
     • EventFactory - event creation is its own component
       • creates a correctly-shaped event with schemaId, contextualAttributes, action, interactionData
       • Current Kotlin's TestKitchenClient.submitMetricsEvent() constructs an Event
     • ContextualAttributesFactory - gets context from the application
       • caching considerations?
       • Current Kotlin's ContextController does not fetch values, it selects and filters based on ClientData provided at event creation time
     • EventSubmitter interface - submit event to stream
       • Current Kotlin's EventProcessor and EventSender acts similar to EventSubmitter - need to expose submit method
     • StreamConfigs - lookup contextual attributes for a stream from static stream config
       • Current Kotlin's SourceConfig and StreamConfig can serve as StreamConfigs
       • Implement a way to bind stream to schema_id - currently schema_id is required

2. Instrument equivalent for instrument creation and event submission
   • getInstrument()
   • send()

3. Exposure logging event

• Expose logExposure() on the Experiment object (T414729: Add Experiment#logExposure method to SDKs)
• Produces event with action = 'experiment_exposure'
• Log once per exposure
• include curated set of attributes defined by T414726: Include curated set of contextual attributes with exposure events

(exposure logging N/A for now)

4. Fetch instrument and experiment configs from Test Kitchen UI
   • Add a TestKitchenConfigsFetcher (HTTP client)
   • Add a TestKitchenConfigsCache?
   • Add a class to resolve the JSON responses into the correct format needed by Kotlin
   • See T398422: MetricsPlatform: InstrumentConfigFetcher: Make fetching asynchronous and compare how Kotlin fetches static stream configs and how it should fetch instrument and experiment configs

5. Synthetic experiment and end-to-end testing for Kotlin SDK
   • Implement a simple experiment that runs in production-like conditions for Android:
     • assignment is stable
     • exposure logging is emitted << N/A, hopefully later
     • instrumented metric events are emitted
   • Query data to confirm:
     • expected enrollments/assignments correspond to experiment config
     • exposure events appear after assignment and matches expected counts/order N/A

6. Deprecation/Clean up
   • Remove unneeded classes/objects:
     • ContextController - replace with ContextualAttributesFactory
     • StreamConfigCollection - replace with suggested StreamConfigResolver
     • CurationController - fold into EventFactory
     • Collapse EventProcessor and EventSenderDefault into a single EventSubmitter?
     • Remove event construction from TestKitchenClient.submitMetricsEvent()
     • Remove the Java MPC from the MP library T413855: Remove Java Metrics Platform/Test Kitchen code - can be done now - no production code is using this client; can also be omitted if the MP library is being renamed to the TK lib
     • Remove android stream configs leftover from Java MPC:
       • android.product_metrics.article_link_preview_interaction and related references
       • android.product_metrics.article_toc_interaction and related references
       • android.product_metrics.article_toolbar_interaction and related references
       • android.product_metrics.find_in_page_interaction and related references

6. Documentation
   • Add examples for instrument + experiment workflows and exposure logging

Questions

1. Apps do not serve HTML cached pages through Varnish like MediaWiki - how do apps run A/B tests?
   • https://github.com/wikimedia/apps-android-wikipedia/blob/main/AGENTS.md states:

When setting up an A/B test for any feature, subclass from ABTest.kt, which automatically assigns the current user into a test bucket.

• See https://github.com/wikimedia/apps-android-wikipedia/blob/main/app/src/main/java/org/wikipedia/analytics/ABTest.kt
• Presumably it works like: device/app identity decides deterministic bucketing?
• How do apps do remote configuration?
• Do apps need to be able to do different bucketing based on logged-in status?
• How should sampling rates in the app context be handled?

2. Is cache-splitting still relevant in the app context for experiments?
   • Presumably not for client-side or UI-only experiments?
   • Do apps run server-side experiments?
   • Do apps ever run hybrid (client-side + server-side) experiments?

3. Are more endpoints needed in Test Kitchen UI?
   • /api/v1/experiments?format=config&authority=android?
   • /api/v1/experiments?format=config&authority=ios?
   • If yes, what differences in the instrument and experiment forms might be needed to capture app-specific fields?
   • If yes, how does the output of each response (for instruments per app-platform, for experiments per app-platform) change?

Next Steps

• Get questions answered, updates plan reviewed, mistakes corrected, missing pieces and known unknowns added
• If the Kotlin Updates section gets to a sound place, we can start building an epic with subtasks mapping to chunks of work with more fleshed-out details and ACs

Some other questions/notes coming out of Slack/Phab:

• We're essentially moving out of the MetricsClient library model in JS/PHP with T415579: Migrate JS and PHP client libraries to TestKitchen extension. For Kotlin then and for parity, it seems like we also want to move away from a TestKitchenClient - or do we? We need a way to initialize the SDK - in PHP we provide the ExperimentManager as a service to get experiments. And for instruments, we will be creating an InstrumentManager (T415579: Migrate JS and PHP client libraries to TestKitchen extension). I'm assuming we want to replicate this kind of structure in Kotlin

• Presumably we're still working out how we want to instruments and experiments to inherit either from each other or from a parent abstract class - pending this resolution, I'll update/add tickets accordingly

• Apps may need per-wiki traffic allocations and language/geography targeting needs - pending discussion

• Apps will need device added as a sampling unit in the TK UI

• Create API endpoint in TK UI to serve experiments by platform (MW, Varnish, Devices)