@nayoub

I finished reviewing the details on the UX desktop flow you provided on the Miro board and am starting to propose new events (documented in yellow text boxes on the miro board).

I just had a question regarding the workflow for when a user elects to dismiss the reference reliability feedback.

Based on the UX flow provided, it looks like the user would click "<" on the reference reliability feedback window and then be directed back to the "Add a citation" citoid window (same as users that select "Try a new source").

If the user decides to re-input the same reference that was deemed unreliable, would they continue to receive the reference reliability feedback or would they be allowed to insert their unreliable citation?

Hi @MNeisler, thanks for sharing!

Based on the UX flow provided, it looks like the user would click "<" on the reference reliability feedback window and then be directed back to the "Add a citation" citoid window (same as users that select "Try a new source").

Yes exactly.

If the user decides to re-input the same reference that was deemed unreliable, would they continue to receive the reference reliability feedback or would they be allowed to insert their unreliable citation?

They should continue to receive the feedback; given the domain is blocked they would not be able to add the citation in this case. At a later stage we'll support adding the citation even if considered unreliable but as long as it does not appear on the SpamBlocklist (which is the case for this iteration).

@DLynch

Below are my initial suggestions for new instrumentation to track the Reference Reliability feature. This is based on the latest UX desktop workflow and metrics defined in the measurement plan and T342930.

Let me know your thoughts. I'm not tied to any of the suggested event names - I tried to reuse existing VEFU actions where available but fine changing if there's a better option. I primarily want to make sure we can distinguish Edit Check (References) actions from Edit Check (Reference Reliability) feature.

VEFU events

Note: Per T352133#9451666, the user may continue to re-enter the same blocked source or a different blocked after returning to the "Add a citation" window but should continue to receive this edit check notice if they did so. We should be able to review subsequent editing session data to help determine if they successfully inserted a citation with their edit.

Revision Tags
Can we also add a new revision tag to indicate when this specific check was shown for published edits? editcheck-reliability-activated? This would be helpful for identifying revert rate for these edits for future analyses.

OPEN QUESTIONS/COMMENTS:

• The new workflow introduces a new citation onboarding step after the user selects "yes" to adding a citation on the edit check (references) dialog prompting the user to select "get started" to continue to the add a citation dialog.

I don't think we need to instrument this as we have not identified any specific metrics that require us to track this unless we believe it might lead users to abandon their edit at this point.

I've also noted new events on the UX workflow (noted in yellow) for reference.

I'd suggest that we went with something a little more bespoke -- citation-blocked perhaps?

citation-blocked sounds good to me but have a clarifying question:

If this notice is expanded to cover any source found to be unreliable (not just blocked) T348060, could that be instrumented as a separate event so we could differentiate between the reasons the notice was shown (blocked vs reliability likely to be questioned)? If not, then I recommend we use citation-unreliable as a less bespoke option that will cover both types of notices.

Seems doable, though we might want to make sure to clean it up once we're done. We'll theoretically wind up with a majority of citations getting some kind of popup here, once we get to the next stage. (e.g. we've considered having a "this source is unknown" level to be shown, which would appear on anything that doesn't get a more-specific ranking.)

If this tag will primarily be used just for the AB test analysis, I also have the option to join EditAttemptStep with mediawiki_history to obtain revert data for these events, which will work for me and avoid any future cleanup work.

Also, it seems that unlike the other edit check tags, there's less benefit enabling volunteers to see if this was shown.

could that be instrumented as a separate event so we could differentiate between the reasons the notice was shown

Speculating a bit here since we've not actually implemented that yet, but I think it's likely that we could. Probably some sort of citation-[category-from-API] label, when we get there. (citation-unreliable would be bad to apply to everything that comes from the future API, since we're going to include things like "this source is considered really trustworthy!" in there.)

sounds good. let's go with citation-blocked for this iteration. And citation-[category-from-API] label when we get to expanding it (if possible). That would allow us to decipher between the notice types in analytics and provide flexibility for the wide range of citation notices being considered.

(citation-unreliable would be bad to apply to everything that comes from the future API, since we're going to include things like "this source is considered really trustworthy!" in there.)

agreed that would be confusing! (did not realize that was one of the options being considered)

(did not realize that was one of the options being considered)

Since we've been using enwiki perennial sources list as a model for the kind of categorizations we might want, you can get an idea of the range we might wind up covering by checking out its legend: https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources/Perennial_sources#Legend

Change 990146 had a related patch set uploaded (by DLynch; author: Esanders):

[mediawiki/extensions/Citoid@master] Query editcheckreferenceurl and show error message

https://gerrit.wikimedia.org/r/990146

Change 990146 merged by jenkins-bot:

[mediawiki/extensions/Citoid@master] Query editcheckreferenceurl and show error message

https://gerrit.wikimedia.org/r/990146

I've confirmed that the new instrumentation added to track this feature appears to be logging as expected. See summary of QA checks and some initial data findings below:

Data reflects reference reliability events logged from deployment 29 Feb 2024 through today 22 March 2024:

• So far, there have been 1278 editing sessions and 812 registered users shown the reference reliability check.
• Overall across all wikis, it's currently being shown to about 0.04% of users that make an edit attempt.
• All expected ref reliability events have been logged. feature = editCheckReliability and action = citation-blocked or action = edit-check-confirm.
• Number of reference reliability events and sessions by action type

• The timing and order of these events appears as expected. The first reference reliability check event logged in all editing sessions is citation-blocked indicating that the editor was shown the reference reliability feedback followed by edit-check-confirm if the user selected "Try new source". Some editing sessions have these events appear more than once.
• Events logged for both registered and unregistered users

• Logged at 40 distinct wikis. Most Wikipedias. 1 event logged at enwiktionary.
• The majority of events occur VisualEditor with a small percentage (0.8%) on Wikitext-2017
• Logged on both desktop and phone platforms.
• Observing events for both junior (100 or fewer edits) and senior editors (over 100 edits). Slightly more for junior but not a significant difference at the moment.

• 68.5% of all editing sessions shown the reference reliability check have been saved successfully .

cc @DLynch

Note: I've also added these new VEFU values to the VEFU Data Dictionary.