☝️ per T350495#9333001.

Change 974674 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[schemas/event/secondary@master] Add custom schemas for 2 Android article instruments

https://gerrit.wikimedia.org/r/974674

Updated PR in the Android repo to be reviewed:
https://github.com/wikimedia/apps-android-wikipedia/pull/4286

This PR depends on schema changes above - https://gerrit.wikimedia.org/r/974674

Instructions for end-to-end testing

This is a very tedious way to test but until we have something more automated, fool-proof, and less flaky, I suppose it will have to do.

The basic steps I go thru to test validation of an event sent via the MP Java client on my local EventGate:

• Because the Java client fetches stream configs from the meta api, submitting a test event requires a stream with producer config - android.metrics_platform.find_in_page_interaction is the example stream I use for this purpose since its config is on prod. This gives me the confidence that stream configs are being fetched correctly and my event is being validated using production config.
• Add the following test to EndToEndIT which sends an event via MetricsClient::submitInteraction() using the aforementioned stream and test ClientData and test InteractionData as parameters:

@Test void submitEventTimerStreamConfig() throws IOException, InterruptedException {
        // Create the metrics client.
        MetricsClient testMetricsClient = MetricsClient.builder(testClientData).build();
        await().atMost(10, SECONDS).until(testMetricsClient::isFullyInitialized);

        testMetricsClient.submitInteraction(
                "android.metrics_platform.find_in_page_interaction",
                DataFixtures.getTestClientData(getExpectedEventClick()),
                DataFixtures.getTestInteractionData("TestClick")
        );
        Thread.sleep(10_000);
        await().atMost(30, SECONDS).until(() -> testMetricsClient.getEventQueue().size() == 0);
    }

• Comment out the following lines:
  • in EndToEndIT because my local EventGate runs on port 8192 which breaks WireMock:
    • the fetchStreamConfigs() method (annotated with @BeforeEach)
    • @WireMockTest(httpPort = 8192)
  • filteredData + setting client data in ContextController:enrichEvent() because in this case the example prod stream only requests a few contextual values (agent data) and I would like to see all the data that is submitted with the test event. Also, in case one does testing with other prod streams, these likely have requested values that are incompatible with the latest Java data objects (i.e. edit properties) in 2.0. The net result is that all available contextual values are submitted with the event instead of just the few specified by prod config.
• Update StreamConfig::getDestinationEventService() to return DestinationEventService.LOCAL
• Start Docker and local eventlogging:

docker compose up --force-recreate eventlogging

• Add the following method to the MetricsClient:

public BlockingQueue<EventProcessed> getEventQueue() {
    return this.eventQueue;
}

• Run the test and see the event validate in a local EventGate instance:

For expediency in testing, I pushed the above changes to a branch that can be pulled down locally and tested with a local EventGate instance:
T351292/test-java-2.0
The diffs can be viewed here:
https://gitlab.wikimedia.org/repos/data-engineering/metrics-platform/-/merge_requests/18/diffs

Troubleshooting:

• You may have to run the test more than once in case your local MetricsClient hasn't fetched production stream configs before processing the submitted event.

Deployment instructions have been added to Wikitech:
https://wikitech.wikimedia.org/wiki/Metrics_Platform/How_To/Deploy_a_New_Version#Java

@VirginiaPoundstone @WDoranWMF I checked in with the Android team about the pending MR -- their next release is scheduled for Monday 11/27 but with the short holiday week, it appears there won't be enough time to review and merge in the requested changes.

This could be their last release of the year. There is a chance that they might do one more release in mid-December in which case the change set could got out then. They will know definitively if another release can be scheduled and will let me know for certain by 11/27. If there is not another release on 12/11, it will likely be early January when the MR can be reviewed/merged and some time thereafter before we can verify data using the core interaction schemas in the Java client.

Per discussion with @VirginiaPoundstone and @WDoranWMF, we will set up a more formal collab effort with the Android team starting in early January similar to how we are starting to work with the Web team for MP adoption.

Best case scenario: there is still a chance these changes could be merged before year end in which case we could have beta release data to verify in early January.

Moving this to blocked until if/when the Android team has bandwidth to review/merge requested changes.

Change 974674 merged by jenkins-bot:

[schemas/event/secondary@master] Add custom schemas for 2 Android article instruments

https://gerrit.wikimedia.org/r/974674

Change 980963 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[operations/mediawiki-config@master] Add stream config for Android article instruments

https://gerrit.wikimedia.org/r/980963

custom schemas patch and Android MR were merged \o/

So one last config patch and then this can be marked done

@WDoranWMF sorry to drop this on you - can you find someone to deploy the final config patch during one of the windows tomorrow 12/7? If not, it can go out Monday 12/11 (not sure when the deployment freeze kicks in - probably soon).

I don't think there's really a way to test this that I know of (during deployments, we're asked to test patches on a debug server before syncing to prod) until/if we see beta release data in Hive so whoever does the deployment can just tell the deployer it's a no-op.

thanks so much!

Change 980963 merged by jenkins-bot:

[operations/mediawiki-config@master] Add stream config for Android article instruments

https://gerrit.wikimedia.org/r/980963

Mentioned in SAL (#wikimedia-operations) [2023-12-12T21:06:02Z] <samtar@deploy2002> Started scap: Backport for [[gerrit:980963|Add stream config for Android article instruments (T351292)]]

Mentioned in SAL (#wikimedia-operations) [2023-12-12T21:07:26Z] <samtar@deploy2002> cjming and samtar: Backport for [[gerrit:980963|Add stream config for Android article instruments (T351292)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-12-12T21:18:02Z] <samtar@deploy2002> Finished scap: Backport for [[gerrit:980963|Add stream config for Android article instruments (T351292)]] (duration: 11m 59s)