Page MenuHomePhabricator

Instrumentation of new SX entrypoints
Closed, ResolvedPublic

Description

Some new entrypoints have been deployed during the recent months and some are still being developed (T286641). In order to track how effective these new entrypoints are, we need to instrument them.

This task is intended to both support this instrumentation within the SX application and keep all the related information in a single place.

Event Timeline

The event names for each entrypoint should be the following:

  1. Entrypoint inside language selector (mflanguagesearcher): content_language_selector
  2. Entrypoint inside recently translated article (mfrecenttranslation) - Not yet completely supported: recent_translation
  3. Entrypoint inside recently edited article (mfrecentedit) - Not yet deployed: recent_edit
  4. Entrypoint inside recently edited article (mffrequentlanguages) - Not yet deployed: frequent_languages

Change 760890 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] SX: Instrumentation support for all entrypoints

https://gerrit.wikimedia.org/r/760890

Change 760890 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] SX: Instrumentation support for all entrypoints

https://gerrit.wikimedia.org/r/760890

I see validation errors from eventlogging

I am using mw cli and it has eventlogging support.

eventlogging_1                         | {
eventlogging_1                         |   "$schema": "/analytics/mediawiki/content_translation_event/1.0.0",
eventlogging_1                         |   "translation_type": "section",
eventlogging_1                         |   "wiki_db": "es",
eventlogging_1                         |   "access_method": "desktop",
eventlogging_1                         |   "user_name": "Admin",
eventlogging_1                         |   "user_is_anonymous": false,
eventlogging_1                         |   "content_translation_session_id": "cx_sx_c18ed0109578987ac734_desktop_es",
eventlogging_1                         |   "event_type": "dashboard_open",
eventlogging_1                         |   "event_source": "direct",
eventlogging_1                         |   "content_translation_session_position": 0,
eventlogging_1                         |   "user_global_edit_count": 152,
eventlogging_1                         |   "user_global_edit_count_bucket": "100-999 edits",
eventlogging_1                         |   "meta": {
eventlogging_1                         |     "stream": "mediawiki.content_translation_event",
eventlogging_1                         |     "domain": "es.mediawiki.mwdd.localhost",
eventlogging_1                         |     "id": "eb1853c9-3930-4de5-9091-da604920d88f",
eventlogging_1                         |     "dt": "2022-02-22T05:10:02.494Z",
eventlogging_1                         |     "request_id": "b0ad2ad0-939d-11ec-902c-edb38b77fe06"
eventlogging_1                         |   },
eventlogging_1                         |   "dt": "2022-02-22T05:10:01.491Z",
eventlogging_1                         |   "http": {
eventlogging_1                         |     "request_headers": {
eventlogging_1                         |       "user-agent": "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Mobile Safari/537.36"
eventlogging_1                         |     }
eventlogging_1                         |   }
eventlogging_1                         | }
eventlogging_1                         | {"name":"eventgate-wikimedia","hostname":"766f50909c18","pid":1,"level":40,"err":{"message":"Event failed validation with schema at /analytics/mediawiki/content_translation_event/1.0.0","name":"ValidationError","stack":"ValidationError: Event failed validation with schema at /analytics/mediawiki/content_translation_event/1.0.0\n    at validatorFor.then (/srv/service/node_modules/eventgate/lib/EventValidator.js:285:23)\n    at tryCatcher (/srv/service/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/srv/service/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/srv/service/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromiseCtx (/srv/service/node_modules/bluebird/js/release/promise.js:641:10)\n    at_drainQueueStep (/srv/service/node_modules/bluebird/js/release/async.js:97:12)\n    at _drainQueue (/srv/service/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/srv/service/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues [as _onImmediate] (/srv/service/node_modules/bluebird/js/release/async.js:15:14)\n    at runCallback (timers.js:705:18)\n    at tryOnImmediate (timers.js:676:5)\n    at processImmediate (timers.js:658:5)","errors":[{"keyword":"required","dataPath":"","schemaPath":"#/required","params":{"missingProperty":"content_translation_session_position"},"message":"should have required property 'content_translation_session_position'"}],"errorsText":"'' should have required property 'content_translation_session_position'"},"msg":"event failed schema validation.","time":"2022-02-22T05:16:52.785Z","v":0}

This is the payload:

$schema: "/analytics/mediawiki/content_translation_event/1.0.0"
access_method: "desktop"
content_translation_session_id: "cx_sx_c18ed0109578987ac734_desktop_es"
dt: "2022-02-22T05:16:51.779Z"
event_source: "suggestion_no_seed"
event_type: "dashboard_translation_start"
meta: {stream: "mediawiki.content_translation_event", domain: "es.mediawiki.mwdd.localhost"}
domain: "es.mediawiki.mwdd.localhost"
stream: "mediawiki.content_translation_event"
translation_type: "section"
user_global_edit_count: 152
user_global_edit_count_bucket: "100-999 edits"
user_is_anonymous: false
user_name: "Admin"
wiki_db: "es"

image.png (152×594 px, 19 KB)

The instrumentation of the entrypoints basically refers to properly logging the dashboard_open event, which is logged with content_translation_session_position property always set to 0. This is fine for the instrumentation of the entrypoints, since the dashboard is always the first step inside SX application for a user coming from an entrypoint. The errors mentioned above refer to other events (e.g. dashboard_translation_start) where content_translation_session_position property is missing. For these reasons I think that this task can be closed.

For reference these are the following entrypoints currently supported with their respective event_source property:

Entrypoint nameEvent source
mflanguagesearchercontent_language_selector
mfrecenttranslationrecent_translation
mfrecenteditrecent_edit
mffrequentlanguagesfrequent_languages

This will be moved to done column as well as the effort is the same as task T295756. See the mentioned task for details

Recent translation entrypoint is instrumented properly, as shown in the screencast below. As we can observe, the campaign URL parameter is properly set to mfrecenttranslation
and the event source of dashboard_open and dashboard_translation_start events, is properly set to recent_translation.

The mobile language selector entrypoint is instrumented properly. As we can check in the screencast below, the campaign URL parameter is correctly set to mflanguagesearcher and the dashboard_open and dashboard_translation_start events are properly logged with the event source set to content_language_selector.

The missing frequent languages entrypoint is NOT properly instrumented as displayed in the following screencast. The campaign URL parameter is wrongfully set to "mfmissinglanguages" instead of mffrequentlanguages and as a result the dashboard_open and dashboard_translation_start events are logged with the generic direct_preselect event source instead of the frequent_languages event source. This will be fixed in a upcoming patch.

Change 819487 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] Fix campaign URL param for mffrequentlanguages entrypoint

https://gerrit.wikimedia.org/r/819487

The instrumentation of the entrypoint when creating a new article on mobile is currently missing and this issue will fixed in a following patch.

Change 819488 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] SX: Support instrumentation of newbytranslationmobile entrypoint

https://gerrit.wikimedia.org/r/819488

Following the latest patches and after the tests, the status of the instrumentation of the entrypoints is displayed in the following table:

Entrypoint nameEvent sourceStatus
mflanguagesearchercontent_language_selectorDone
mfrecenttranslationrecent_translationDone
mfrecenteditrecent_editEntrypoint not yet deployed
mffrequentlanguagesfrequent_languagesEntrypoint not properly instrumented (wrong URL campaign param)
newbytranslationmobilecreation_inviteEntrypoint instrumentation not supported

Except for the recent edit entrypoint that has not yet been deployed, fixes have been submitted for the instrumentation of the other two pending entrypoints.

Change 819487 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] Fix campaign URL param for mffrequentlanguages entrypoint

https://gerrit.wikimedia.org/r/819487

Change 819488 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] SX: Support instrumentation of newbytranslationmobile entrypoint

https://gerrit.wikimedia.org/r/819488

Following the latest patches and after the tests, the status of the instrumentation of the entrypoints is displayed in the following table:

Entrypoint nameEvent sourceStatus
.........
newbytranslationmobilecreation_inviteEntrypoint instrumentation not supported

Based on input from @MNeisler, it seems that it may be better to name entrypoins without referring to the platform. If the same entry point is available on both desktop and mobile, it can use the same name (since it is the same idea connected to the same activity) and other data cana be used in the analysis to separate the data per platform if needed.

So I guess that it is better to name this event as "newbytranslation" instead of "newbytranslationmobile".

Following the latest patches and after the tests, the status of the instrumentation of the entrypoints is displayed in the following table:

Entrypoint nameEvent sourceStatus
.........
newbytranslationmobilecreation_inviteEntrypoint instrumentation not supported

@ngkountas
For this entry point, I believe we currently have a similar event_source already identified in content_translation_event for the desktop entry point created in T216500:

event_source = 'invite_new_article_creation' as a possible source for the dashboard_open event

As @Pginer-WMF mentioned above, I'd recommend we apply the same event source to the equivalent mobile entry point if possible and use the access_method field to distinguish between the two types of entry points. This would simplify analytics and align with how the other entry points have been instrumented (for example, event_source = content_language_selector was applied to both the mobile and desktop entry points).

Note: I will update the content_translation_event schema documentation to reflect these new event_source values as part of T287403.

Change 823521 had a related patch set uploaded (by Santhosh; author: Santhosh):

[mediawiki/extensions/ContentTranslation@master] CX3 Build 0.2.0+20220816

https://gerrit.wikimedia.org/r/823521

Change 823521 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] CX3 Build 0.2.0+20220816

https://gerrit.wikimedia.org/r/823521

Change 825717 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] Fix event source for "newbytranslationmobile" entrypoint

https://gerrit.wikimedia.org/r/825717

@MNeisler thank you for your input! I fixed the related event source to match the existing one

Change 825717 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] Fix event source for "newbytranslationmobile" entrypoint

https://gerrit.wikimedia.org/r/825717

Change 826546 had a related patch set uploaded (by Nik Gkountas; author: Santhosh):

[mediawiki/extensions/ContentTranslation@master] CX3 Build 0.2.0+20220825

https://gerrit.wikimedia.org/r/826546

Change 826341 had a related patch set uploaded (by KartikMistry; author: Santhosh):

[mediawiki/extensions/ContentTranslation@wmf/1.39.0-wmf.26] CX3 Build 0.2.0+20220825

https://gerrit.wikimedia.org/r/826341

Change 826546 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] CX3 Build 0.2.0+20220825

https://gerrit.wikimedia.org/r/826546

Change 826341 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@wmf/1.39.0-wmf.26] CX3 Build 0.2.0+20220825

https://gerrit.wikimedia.org/r/826341

Mentioned in SAL (#wikimedia-operations) [2022-08-25T13:32:13Z] <hashar@deploy1002> Started scap: Backport for [[gerrit:826341|CX3 Build 0.2.0+20220825 (T309986 T301222)]]

Mentioned in SAL (#wikimedia-operations) [2022-08-25T13:57:10Z] <hashar@deploy1002> Finished scap: Backport for [[gerrit:826341|CX3 Build 0.2.0+20220825 (T309986 T301222)]] (duration: 24m 56s)

Based on the ocnversation above it seems that the event source has been set to " 'invite_new_article_creation", but it is unclear which i the final entry point name for this one. Can anyone confirm the name to keep the documentation in sync?
Thanks!

@Pginer-WMF I updated the documentation setting the name of the entrypoint to newbytranslation. The name doesn't mean much in our code in any case, it's more a way to reference the entrypoint.