Page MenuHomePhabricator

Further investigate Citoid feature use failure rate using VEFU events
Closed, ResolvedPublic

Description

In T368988, we did not find any overall changes in users' success using Citoid to generate and insert citations following the August 3rd decrease citoid requests from MyBib.

As part of this task, we will update the analysis to show the proportions of citoid edit sessions where no citation was generated because no search results were found ( automatic-generate-fail-searchResults ) or due to a network error while attempting to retrieve the necessary information (automatic-generate-fail-network).

This will help provide more details on why some Citoid users do not successfully insert a citation during an editing session.

Event Timeline

Of the times when people open Citoid and do NOT follow through to insert a reference, what reasons explain why this might be the case?

Overall proportion of failed citoid editing sessions by reason:

ReasonJuly 2024August 2024
Automatic citation generation failed19.1%19.3%
* Network error while attempting to retrieve the necessary information18.4%18.7%
* No search results were found0.7%0.6%
Session aborted80.9% 80.7%

As found in T368988#10123059, about 19% of all citoid editing sessions that failed to insert a reference were due to an automatic citation generation failure. Of these sessions, the majority of citation generation failures were due to a network error while retrieving citation info. Network errors account for 18.7% of all citoid sessions that failed and 97% of all citoid sessions that failed due an automatic citation generation failure.

There were no observed changes in observed failure rates before or after the decrease in Citoid traffic on August 3rd.

Per platform proportion of failed citoid editing sessions by reason:
Per platform results are very similar to the overall results. The majority of citation generation failures on both desktop and mobile were due to a network error. 97% of all citation generation failures on desktop were due to a network error and 95% of all citation generation failures on mobile were due to a network error.

Time Series Chart
I further investigated the daily number of Citoid editing events and confirmed there were no any sudden changes in the number of automatic citation generation failure events right around August 3rd. The number of daily citoid sessions that have failed to insert a reference due to a network failure to retrieve results has remained stable before and after the change in Citoid traffic.

citoid_failure_events_daily.jpg (540×960 px, 38 KB)

From this, it still appears that the 3 August 2024 change had no impact on overall user success or failure rates using Citoid on desktop or mobile.

cc @ppelberg

ppelberg added subscribers: DLynch, Mvolz.

Per what @MNeisler and I discussed offline today, work on this ticket is complete for now.

Emphasis on "for now" seeing as how what we learn through discussing the results Megan shared in T374624#10158451 with @Mvolz and @DLynch as part of T372438 could cause us to revisit this ticket/analysis.