This analysis will start following confirmation of data in T295756.

Quick update:
The content_translation_event_schema was updated to include the new event_source values instrumented in T301222 . Once those updates are deployed, we should start to see events logged in content_translation_event and I can begin reviewing aggregate data to determine the basic usage of each entry point. Note: I'd recommend waiting a week or two from deployment to obtain a sufficient number of events for review.

Currently, we have only logged the following entry points events in August 2022:

Data via:

SELECT 
event_type,
event_source,
access_method,
COUNT(*) AS n_events
FROM 
event.mediawiki_content_translation_event
WHERE
YEAR = 2022 
AND MONTH = 08 
GROUP BY
access_method,
event_source,
event_type

In T295757#8194882, @MNeisler wrote:

event_type	event_source	access_method	n_events
dashboard_open	content_language_selector	desktop	3074
dashboard_open	direct	desktop	2328
dashboard_open	direct_preselect	desktop	10067
dashboard_open	invite_new_article_creation	desktop	67

It was surprising to see the access method being "desktop" for all these entry points.
Should we double-check that the events from mobile are captured correctly?

Hi @ngkountas I took a look into why the new section entry points are not yet being logged in the content_translation_event schema and identified a few possible issues to be resolved:

• The instrument needs to be updated to send events to schema version 1.2.0 (The version number was updated in T287403). We're currently seeing validation errors because the events are being sent to schema 1.0.0 which does not include the new event_source names.
• Per @Pginer-WMF's comment above, it looks like we are logging all of the new section entry points from mobile incorrectly as access_method: desktop. These should be logged as access_method: mobile web.

Here's a screenshot of a validation error we're receiving for one of the event_source: frequent_languages events:

Let me know if there are any other details or info that would be helpful

Change 829188 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] SX Instrumentation: Fix event logging schema version

https://gerrit.wikimedia.org/r/829188

Change 829190 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] SX Instrumentation: Fix event logging access_method

https://gerrit.wikimedia.org/r/829190

@MNeisler thank you very much for your input! Both issues are fixed in the above (currently in-review) patches. @Pginer-WMF currently only SX logs events for the content_translation_event schema. However, as Megan also noted, there was a bug inside SX codebase, which led all those events to be logged with the access_method set to desktop, while in fact it should be mobile web.

Change 829190 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] SX Instrumentation: Fix event logging access_method

https://gerrit.wikimedia.org/r/829190

Change 829188 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] SX Instrumentation: Fix event logging schema version

https://gerrit.wikimedia.org/r/829188

Change 829549 had a related patch set uploaded (by Santhosh; author: Santhosh):

[mediawiki/extensions/ContentTranslation@master] CX3 Build 0.2.0+20220905

https://gerrit.wikimedia.org/r/829549

Change 829549 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] CX3 Build 0.2.0+20220905

https://gerrit.wikimedia.org/r/829549

Thanks @ngkountas for submitting those fixes! I rechecked the aggregate data in content_translation_event and confirmed that we are now logging events for the new SX entry points and they are all now correctly logged with access_method = 'mobileweb'.

Below are the events logged for today (9 September 2022)

The frequent_languages entry point currently seems to have the most events logged today.

I'm going to wait about a week to collect some more data for review and then will plan to explore the aggregate data in more detail and provide an initial report on trends identified for each entry point.

I created a ticket to instrument the follow-up invite: T317995: Instrument follow-up invite shown after publishing
This can provide an idea of how many of the translations are part of a sequence for the user.

Thanks a lot for this analysis @MNeisler. This is super informative and has already helped to surface ideas on what to investigate next. Regarding the report just a couple of minor considerations for adjustment:

• Events by Session shows how many sections are translated in a given session and numbers are organized in different buckets. Currently the initial bucket is for 1-5 translations and I was wondering if it makes sense to break it into smaller buckets since these seem to cover different scenarios. A user making just one translation represents a different behaviour than making a longer sequence. I'm not sure which is the right division but maybe something like 1, 2-3, 4-5, or 1, 2-5 could be more meaningful.
• In addition to capture the events per session it would be great to show them per user, to have a sense of which is the percentage of recurring users accessing the tool vs. those accessing just once.

Regarding the question of anonymous users, Nik can confirm, but I think entry points are only shown for logged-in editors. Thus, it is expected not to get anonymous users. This makes it more unexpected to get a high volume of logged-in editors without a previous edit (and more interesting to check if they end up publishing a translation).

I created some follow-up tickets as next steps to expand our learnings in this area (and correct/check different aspects):

• T318907: Provide the missing metadata for entry point instrumentation

• T318906: Instrument the publishing step of sections and article translations on mobile and desktop

• T318910: Investigate entry point with unexpected results

• T318916: Compare how much are entry points shown and accessed

• T318917: Instrument desktop entry points for which a mobile version exists