Page MenuHomePhabricator

Metrics instrumentation for CodeMirror 6
Open, Needs TriagePublic

Description

Background

Existing metrics use the legacy EventLogging system, with its own schema. There was a Grafana dashboard but it is now marked for cleanup. This task is about creating new instrumentation for CodeMirror 6, starting from a clean slate. This data will help us evaluate which features are the most popular, and influence what is decided as the default set of features in the longer-term goal of having CodeMirror on by default for interest projects (i.e. T288161).

As part of this task, we should retire the older instrumentation.

Data questions
  • How many people are using CodeMirror?
  • Which editors are being used with CodeMirror (2017 editor, WikiEditor, or standalone CodeMirror)?
  • Which CodeMirror features are being used?
  • What is the relationship of editor experience and the various features? (edit count bucketing)
Implementation

We will use the existing VisualEditorFeatureUse schema, despite it also being "legacy". This is to keep editing-related metrics in one place.

What we're trying to do is very similar to what was done for Realtime Preview (T306176).

We will use the new feature name codemirror, which has the following actions:

  • activated – when CodeMirror is activated by any means
  • deactivated – when CodeMirror is toggled off
  • search – when the CodeMirrorSearch panel is opened
  • keymap – when the keyboard shortcut dialog is opened
  • prefs-display – when the CodeMirrorPreferences panel is opened
  • prefs-activeLine – when the activeLine extension is applied
  • prefs-bidiIsolation
  • prefs-bracketMatching
  • prefs-lineNumbering
  • prefs-lineWrapping
  • prefs-specialChars
  • prefs-codeFolding
  • prefs-autocomplete
  • prefs-openLinks

After the code is merged, the FeatureUse data dictionary needs to be updated accordingly.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
MusikAnimal changed the task status from Open to In Progress.Nov 21 2024, 4:23 AM
MusikAnimal claimed this task.
MusikAnimal updated the task description. (Show Details)

Change #1121719 had a related patch set uploaded (by MusikAnimal; author: MusikAnimal):

[mediawiki/extensions/CodeMirror@master] Add metrics instrumentation using the VisualEditorFeatureUse schema

https://gerrit.wikimedia.org/r/1121719

Change #1121725 had a related patch set uploaded (by MusikAnimal; author: MusikAnimal):

[mediawiki/extensions/CodeMirror@master] Introduce CodeMirror#addMwHook to deduplicate mw.hook handlers

https://gerrit.wikimedia.org/r/1121725

Change #1121725 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@master] Introduce CodeMirror#addMwHook to deduplicate mw.hook handlers

https://gerrit.wikimedia.org/r/1121725

Change #1121719 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@master] Add metrics instrumentation using the VisualEditorFeatureUse schema

https://gerrit.wikimedia.org/r/1121719

This is now done. I've created a new chart at https://superset.wikimedia.org/superset/explore/p/JxRQJ9ynNWA/

I will move the chart to the CommTech dashboard once I'm given access to it: https://superset.wikimedia.org/superset/dashboard/309/

The FeatureUse data dictionary has also been updated: https://www.mediawiki.org/?diff=7494404

MusikAnimal reopened this task as In Progress.EditedMar 23 2025, 5:33 AM
MusikAnimal moved this task from Done to Improvement on the MediaWiki-extensions-CodeMirror board.

Okay, I was too quick to resolve! There are some follow-ups needed.

For starters, it looks like we're only recording data for 2017 editor users, hence why the numbers of line numbering, line wrapping and bracket matching are so high – those are the only features the 2017 editor supports (for now).

Meanwhile, there are a handful of rows where the editor_interface is visualditor, which should be impossible. I also see actions like prefs-specialChars and prefs-openLinks which are not available in the 2017 editor… so my guess is these are users who started in WikiEditor and switched to VE midway during the editing session. That would explain why the numbers are so low.

I'll get the WikiEditor logging fixed and we'll go from there.

Change #1130304 had a related patch set uploaded (by MusikAnimal; author: MusikAnimal):

[mediawiki/extensions/CodeMirror@master] CodeMirrorWikiEditor: include all fields in logging that WikiEditor does

https://gerrit.wikimedia.org/r/1130304

Change #1130304 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@master] CodeMirrorWikiEditor: include all fields in logging that WikiEditor does

https://gerrit.wikimedia.org/r/1130304

Change #1130650 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/WikimediaEvents@master] EditAttemptStep: set defaults from the first event

https://gerrit.wikimedia.org/r/1130650

Change #1130650 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] EditAttemptStep: set defaults from the first event

https://gerrit.wikimedia.org/r/1130650

Alright! The data looks accurate now. The updated chart has been added to the CommTech dashboard.

One thing I might do as a follow-up is only report feature usage when the user's preferences differ from the defaults. This would make it easier to answer "what features are the most popular?", since you can't really assume default features are desired. To that end, we may have an issue with discoverability of CodeMirror preferences. In WikiEditor (2010) a Settings icon is buried in the "Advanced" area of the toolbar, so it probably isn't seen too often.

The "activated" and "deactivated" counts can remain as-is, so overall usage tracking of CodeMirror regardless of feature preferences.

Change #1134321 had a related patch set uploaded (by MusikAnimal; author: MusikAnimal):

[mediawiki/extensions/CodeMirror@master] CodeMirror: only logEditFeature when the pref differs from default

https://gerrit.wikimedia.org/r/1134321

Change #1134321 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@master] CodeMirror: only logEditFeature when prefs differ from the defaults

https://gerrit.wikimedia.org/r/1134321

MusikAnimal moved this task from Improvement to Done on the MediaWiki-extensions-CodeMirror board.

I think we can call this resolved now. The data looks great.

I may invent an editor_interface value of wikitext-2025 so we can track usage of standalone CodeMirror, but that can come at a later time.

I spoke to soon. It seems there's now no reporting of a preference if it matches the default. I.e. I don't see prefs-lineWrapping or prefs-lineNumbering in the chart.

r1134321 was supposed to report all in-use features if any feature was enabled/disabled from the default state, effectively telling us which preferences are most popular among those who definitively took the time to change their preferences (no bias towards default features). Note the test case that passes :/

I've got more work to do.

Change #1140699 had a related patch set uploaded (by MusikAnimal; author: MusikAnimal):

[mediawiki/extensions/CodeMirror@master] extension.json: remove unused CodeMirrorUsage EventLoggingSchemas attr

https://gerrit.wikimedia.org/r/1140699

Change #1140699 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@master] extension.json: remove unused CodeMirrorUsage EventLoggingSchemas attr

https://gerrit.wikimedia.org/r/1140699