Page MenuHomePhabricator

Investigate Citoid feature use
Closed, ResolvedPublic

Assigned To
Authored By
ppelberg
Jul 1 2024, 9:18 PM
Referenced Files
F57465140: citoid_success_funnel_byexp_Aug.png
Sep 5 2024, 7:37 PM
F57465131: citoid_success_funnel_Aug.png
Sep 5 2024, 7:37 PM
F57279980: citoid_edit_attempts_fail.png
Aug 19 2024, 12:17 PM
F57279660: citoid_success_funnel_exp.png
Aug 19 2024, 12:17 PM
F57279663: citoid_success_funnel_platform.png
Aug 19 2024, 12:17 PM
F57280045: citoid_insert_success.png
Aug 19 2024, 12:17 PM
F57280043: citoid_dependency_exp.png
Aug 19 2024, 12:17 PM
F57281267: citoid_frequency_users2.png
Aug 19 2024, 12:17 PM

Description

T362379 is, among other things, creating an opportunity for us to more deeply understand how people are using Citoid and its role within the broader set of editing experiences, which is precisely what we'd like to use this ticket to do.

Decisions to be made

  • 1. Decide – what if any – changes we will make to how the Editing Team/Foundation allocates attention to the maintenance and improvement of Citoid.

Note: see more in Citoid/Executive Updates.

Research questions

Research questionMetric(s)
Who uses Citoid? Asked another way: how does Citoid usage vary by experience level and platform?Of all the edits in the Wikipedia main namespace where Citoid is used, what proportion of these edits were completed by 1) brand new editors, 2) Junior Contributors, 3) Senior Contributors (grouped by platform (mobile / desktop)? NOTE: "use" in this context means someone successfully tapped Insert to add a citation into the article they're editing.
How heavily do people depend on Citoid to insert citations?Of all the unreverted new content edits made with VE in the Wikipedia main namespace that add a new reference, what proportion of those edits did people use Citoid to make? What percentage of those edits did people use other means to add references? NOTE: this question, knowingly excludes some edits people might use Citoid to make (e.g. adding a citation without adding any new content). We will broaden our investigation to include edits of this type if/when need be.
To what extent are people finding success with Citoid?1) Of the edits to content in Wikipedia's main namespace that involve someone using Citoid, what percentage of those edits involve people doing the following? i) Successfully/unsuccessfully inserting a reference (grouped by experience level and platform) and ii) Publishing/not publishing an edit with a new reference (grouped by experience level and platform) and 2) Of the times when people open Citoid and do NOT follow through to insert a reference, what reasons explain why this might be the case? //E.g. people abandon the workflow, Citoid failed to generate a citation (automatic-generate-fail), No results were found for the input someone provided (automatic-generate-fail-serch results), Network error while attempting to retrieve the necessary information (automatic-generate-fail-netowrk)

Curiosities

Work on these questions will only happen if/when we explicitly prioritize them which, as of 7 August 2024, we have not done.

  1. Of the people for whom Citoid is not able to generate a citation, what proportion of them act on the call-to-aciton T370561 instrumented to add a citation manually?
NOTE: Planning to re-run this analysis after September 3rd to review the impact of the August 3rd decrease of Citoid request volume requests on these results.

Event Timeline

Meta: I've updated the task description to clarify the requirements of this task based on what @MNeisler and I talked about offline on 17 July 2024.

Meta: I've updated the task description to clarify the requirements of this task based on what @MNeisler and I talked about offline on 17 July 2024.

I've made another round of updates to the task description based on the initial scope @MNeisler and I converged on offline this week.

MNeisler triaged this task as Medium priority.Jul 31 2024, 3:29 PM
MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

Here are the initial results exploring Citoid feature use by user experience level and platform. Please let me know if you have any questions or any further breakdowns would be useful. cc @ppelberg

Methdology Notes

Reviewed all edit attempts and saves recorded in July 2024 that were made to a main namespace on a Wikipedia Project. Data on engagement with the Citoid feature are tracked in visualeditorfeatureuse and other edit data was obtained from editattemptstemp and mediawiki-history.

How does Citoid usage vary by experience level and platform?

Of all the edits in the Wikipedia main namespace where Citoid is used, what proportion of these edits were completed by 1) brand new editors, 2) Junior Contributors, 3) Senior Contributors (grouped by platform (mobile / desktop)?

NOTE: "use" in this context means someone successfully tapped Insert to add a citation into the article they're editing. Limited analysis to edits that were published.
  • Overall, across both platforms, most edits (82.9%) where citoid was used were completed on desktop
platformNumber of citoid editsproportion of citoid edits
desktop9610482.9%
phone1989417.2%
  • The majority (76.2% of all citoid edits on desktop and 70.7% of all citoid edits on mobile) are completed by senior editors; however, if we look at the proportion of distinct users that completed at least one citoid edit, the frequency of usage is slightly higher among junior contributors. In other words, while junior editors complete less overall citoid edits than senior editors, a higher proportion of distinct junior contributors use the feature. See details in the charts below.

Proportion of distinct Citoid edits completed by experience level and platform

citoid_frequency_edits_2.png (563×970 px, 69 KB)

Proportion of distinct Citoid users by experience level and platform

citoid_frequency_users2.png (565×962 px, 70 KB)

How heavily do people depend on Citoid to insert citations

Of all the unreverted edits made with VE in the Wikipedia main namespace that add a new reference, what proportion of those edits did people use Citoid to make? What percentage of those edits did people use other means to add references?

  • Citoid was used for more than half (62%) of all edits with a reference that were not reverted within 48 hours.
  • There are similar rates of Citoid dependency on both mobile and desktop platforms as shown in table below.

Unreverted edits made with VE that add a reference

platformcitoid or other method used?number of editsproportion of edits on platform
desktopother method5132338.5%
desktopcitoid8210861.5%
phoneother method750334%
phonecitoid1453766%
  • The use of Citoid to complete edits with a reference is slightly higher for brand new editors. 71.4% of edits completed by brand new editors that were published with a new reference used Citoid compared to 60.1% of edits completed by senior editors.

citoid_dependency_exp.png (1×1 px, 122 KB)

To what extent are people finding success with Citoid?

Of the edits attempts in Wikipedia's main namespace that involve someone using Citoid, what percentage of those edits involve people doing the following? i) Successfully/unsuccessfully inserting a reference (grouped by experience level and platform)

  • A reference was successfully inserted for 50% of editing sessions where citoid was opened.
  • People are less likely to be able to successfully insert a reference after opening citoid on mobile compared to desktop. Across all experience levels, 29% of users successfully inserted a reference after opening citoid on mobile compared to 57% on desktop.
  • Senior editors are more likely to successfully insert a reference with Citoid compared to junior or brand new editors on both mobile and desktop platforms. Senior editors success rate on both platforms is around 60% while brand new editors successfully insert a reference for less than half of editing sessions where Citoid is opened.
  • Brand new editors are much more likely to insert a reference successfully on desktop compared to mobile. Only 12.8% of citoid edits attempts on mobile by brand new editors involved a reference being successfully inserted compared to 45.9% on desktop.

citoid_insert_success.png (982×1 px, 173 KB)

and ii) Publishing/not publishing an edit with a new reference (grouped by experience level and platform). Note: This excludes any edits reverted within 48 hours and is limited to edits that were published with a new reference as indicated by the editcheck-newreference tag.

  • 44.8% of all desktop citoid edit attempts were successfully saved with a new reference compared to 20.7% on mobile. Note: This includes users who may have opened the citoid editor but did not attempt to generate or insert a reference. The chart below clarifies the proportion of edits that reach each step of a successful citoid edit from open -> click -> insert -> successfully publish.
  • The largest percent decrease between stages is between the user opening the ciotid editor (action = open) to clicking the create button to generate a citation (action = create). 63.5% of editing sessions on desktop and only 35.4% of editing sessions on mobile include a click to generate a citation after opening Citioid.
  • Users are more likely to successfully publish an edit after clicking the create button. About 70.5% of edits that included a click to generate a citation were successfully saved on desktop and 59% on mobile.

    By Platform

citoid_success_funnel_platform.png (1×1 px, 183 KB)

  • As shown in the chart below, brand new editors are the least likely group to complete the remaining steps of publishing a citoid edit after opening citoid. Only 19.9% of edits by brand new editors that opened citoid were successfully published.

By Editor Experience

citoid_success_funnel_exp.png (1×1 px, 217 KB)

Of the times when people open Citoid and do NOT follow through to insert a reference, what reasons explain why this might be the case?

NOTE: There are no current events tagged with automatic-generate-fail-searchResults and automatic-generate-fail-network depite these being documented in the VEFU and indicated as instrumented in T363292. I'll follow-up to confirm if there is a data issue preventing these from logging correctly. At the time of this analysis, we are limited to automatic-generate-fail which indicates automatic citation generation failed and error message is displayed and dialog-abort, which indicates someone exited the citoid editor.
  • 81% of edit attempts when people open Citoid and do not insert a reference is because they abandoned the workflow. Only 19% of all citoid edit attempts failed to insert a reference because the automatic citation generation failed.
  • Per platform splits are provided below:
platformactionnumber of editsproportion of citoid edits that did not insert a reference
desktopautomatic-generate-fail993026.21%
desktopdialog-abort2795973.79%
phoneautomatic-generate-fail306510.07%
phonedialog-abort2738489.93%
  • People abandoning the workflow is the most common reason for not inserting a reference across all experience level groups as well. Senior editors that open citoid are the most likely group to not insert a reference due to the automatic citation generation failure (36.6% of citoid edits by senior editors that do not insert a reference)

citoid_edit_attempts_fail.png (1×1 px, 183 KB)

However, many of the citoid edits attempts were abandoned prior to clicking the create button to generate a citation. If we limit to editing sessions where people click the create button to insert a citation, then the primary reason for not inserting a citation is due to an automatic citation generation fail.

  • 63% of editing sessions on desktop and 61% of editing sessions on mobile where people click the create button do not follow through to insert a reference due to an automatic citation generation fail.
  • Similar rates were observed for junior editors and senior as well
  • For brand new editors that click the create button, there is closer to an even split between reasons for not inserting a reference. 48.3% of these editing sessions are abandoned and 51.7% of these editing sessions included an automatic citation generation fail.

Here are the initial results exploring Citoid feature use by user experience level and platform. Please let me know if you have any questions or any further breakdowns would be useful. cc @ppelberg

This looks great, @MNeisler. Comments/questions in-line below...

In parallel, I've pulled out – what I think are – the notable findings into this document for us to share internally and externally (by way of T370682).

Methdology Notes

Reviewed all edit attempts and saves recorded in July 2024 that were made to a main namespace on a Wikipedia Project. Data on engagement with the Citoid feature are tracked in visualeditorfeatureuse and other edit data was obtained from editattemptstemp and mediawiki-history.

Assuming doing so would not require a great deal of effort, I think it would be worthwhile to re-run this analysis for the 30 days that followed August 3, 2024 to see how – if at all – the decrease in Citoid request volume impacts these numbers.

How does Citoid usage vary by experience level and platform?

Of all the edits in the Wikipedia main namespace where Citoid is used, what proportion of these edits were completed by 1) brand new editors, 2) Junior Contributors, 3) Senior Contributors (grouped by platform (mobile / desktop)?

NOTE: "use" in this context means someone successfully tapped Insert to add a citation into the article they're editing. Limited analysis to edits that were published.

To be doubly sure, would it be accurate for me to understand the data in the table below as meaning the following?

Of all published edits where someone tapped the Insert button within Citoid, 82.9% of said edits occurred on desktop and 17.2% occurred on mobile.

Note: I assume 17.2% and 82.9% summing to >100% is simply a result of rounding.

Assuming doing so would not require a great deal of effort, I think it would be worthwhile to re-run this analysis for the 30 days that followed August 3, 2024 to see how – if at all – the decrease in Citoid request volume impacts these numbers.

This makes sense and would not require much effort. I'll make a note to rerun this around September 3rd. This would be 30 days after the citoid volume decrease on August 3rd and the August mediawiki_history snapshot should be available by then as well.

To be doubly sure, would it be accurate for me to understand the data in the table below as meaning the following?
Of all published edits where someone tapped the Insert button within Citoid, 82.9% of said edits occurred on desktop and 17.2% occurred on mobile

Yes that's correct (limited to main namespace edits on a Wikipedia project).

Note: I assume 17.2% and 82.9% summing to >100% is simply a result of rounding.

Yes. The more exact percentages are 82.85% and 17.15%

Assuming doing so would not require a great deal of effort, I think it would be worthwhile to re-run this analysis for the 30 days that followed August 3, 2024 to see how – if at all – the decrease in Citoid request volume impacts these numbers.

This makes sense and would not require much effort. I'll make a note to rerun this around September 3rd. This would be 30 days after the citoid volume decrease on August 3rd and the August mediawiki_history snapshot should be available by then as well.

Excellent. Let's do it.

To be doubly sure, would it be accurate for me to understand the data in the table below as meaning the following?
Of all published edits where someone tapped the Insert button within Citoid, 82.9% of said edits occurred on desktop and 17.2% occurred on mobile

Yes that's correct (limited to main namespace edits on a Wikipedia project).

Wonderful, ok.

Note: I assume 17.2% and 82.9% summing to >100% is simply a result of rounding.

Yes. The more exact percentages are 82.85% and 17.15%

Got it. Thank you for clarifying ^ _ ^

@ppelberg On August 3rd, overall citoid requests from MyBib dropped substantially. I re-ran this analysis to include 30 days post August 3rd and did not identify any significant changes to Citoid feature use due to this change in request volume.

See details below:

How does Citoid usage vary by experience level and platform?

platformJuly proportion of citoid editsAugust proportion of citoid edits
desktop82.85%82.12%
mobile web17.15%17.88%

Proportion of distinct Citoid edits completed by experience level and platform

editor experienceJuly proportion of citoid editsAugust proportion of citoid edits
brand new editor3.43%3.35%
junior21.19%20.06%
senior75.39%76.6%

Proportion of distinct Citoid users by experience level and platform

editor experienceJuly proportion of citoid usersAugust proportion of citoid users
brand new editor15.46%15.65%
junior45%44.81%
senior39.53%39.54%

How heavily do people depend on Citoid to insert citations

  • No significant changes in citoid dependency. Following August 3rd, Citoid has been used for 60% of all edits with a reference that were not reverted within 48 hours across both platforms, compared to 62% prior in July.

Unreverted edits made with VE that add a reference by platform

platformcitoid or other method used?July proportion of edits on platformAugust proportion of edits on platform
desktopother method38.5%40.9%
desktopcitoid61.5%59.1%
phoneother method34%35.1%
phonecitoid66%64.9%
  • The use of Citoid to complete edits with a reference is still higher for brand new editors following August 3rd. 72.1% of all published edits with a reference by brand new editors used Citoid compared to 57.3% of edits completed by senior editors. This is comparable to July numbers (71.4% by brand new editors and 60.1% by senior editors).

To what extent are people finding success with Citoid?

Before and after August 3rd, a reference was successfully inserted for 50% of editing sessions where citoid was opened. There have not been any significant changes in citoid success rate observed by platform or by experience level.

See updated per platform and per editor experience splits in the citoid editing session funnel charts below:

citoid_success_funnel_Aug.png (621×1 px, 62 KB)

citoid_success_funnel_byexp_Aug.png (620×1 px, 70 KB)

Of the times when people open Citoid and do NOT follow through to insert a reference, what reasons explain why this might be the case?

  • Before and after August 3rd, 81% of edit attempts when people open Citoid and do not insert a reference is because they abandoned the workflow. Only 19% of all citoid edit attempts failed to insert a reference because the automatic citation generation failed.
  • proportion of citoid edits that did not insert a reference by platform
platformactionJuly proportion of citoid editsAugust proportion of citoid edits
desktopautomatic-generate-fail26.21%26.98%
desktopdialog-abort73.79%73.02%
phoneautomatic-generate-fail10.07%10.16%
phonedialog-abort89.93%89.84%
  • Following August 3rd, people abandoning the workflow is still most common reason for not inserting a reference across all experience level groups. Senior editors that open citoid are the most likely group to not insert a reference due to the automatic citation generation failure. This is primarily because most senior editors do not abandon their edit prior to clicking on the citation create button compared to more inexperienced users.

After clicking the create button to generate a citation on desktop, 63% of all failed citoid editing sessions failed due to an automatic generation failure. This is the same rate observed prior to August 3rd.

We observed a slight decrease on mobile. 59% of failed citoid editing sessions on mobile failed due to an automatic generation failure after clicking the create button compared to 61% prior to August 3rd. We are unable to confirm if this slight decrease is the result of the Citoid volume decrease.

Thank you for pulling this all together, @MNeisler.

Per what we talked about offline, we have NOT yet detected any change in the likelihood that people will be successful using Citoid to generate a citation before/after the 3 August 2024 change in third-party request volume.

Now, in order for us to say – with some confidence – the 3 August 2024 change did, in fact, have no impact, we'd first need to be able to distinguish cases where Citoid fails because of a network failure (e.g. blocked request) and cases where Citoid fails because no search results were found for the provided input.

Note this work is happening in T374572.