In T374624#10158451, @MNeisler wrote:

Of the times when people open Citoid and do NOT follow through to insert a reference, what reasons explain why this might be the case?

Overall proportion of failed citoid editing sessions by reason:

Reason	July 2024	August 2024
Automatic citation generation failed	19.1%	19.3%
* Network error while attempting to retrieve the necessary information	18.4%	18.7%
* No search results were found	0.7%	0.6%
Session aborted	80.9%	80.7%

As found in T368988#10123059, about 19% of all citoid editing sessions that failed to insert a reference were due to an automatic citation generation failure. Of these sessions, the majority of citation generation failures were due to a network error while retrieving citation info. Network errors account for 18.7% of all citoid sessions that failed and 97% of all citoid sessions that failed due an automatic citation generation failure.

There were no observed changes in observed failure rates before or after the decrease in Citoid traffic on August 3rd.

Per platform proportion of failed citoid editing sessions by reason:
Per platform results are very similar to the overall results. The majority of citation generation failures on both desktop and mobile were due to a network error. 97% of all citation generation failures on desktop were due to a network error and 95% of all citation generation failures on mobile were due to a network error.

Time Series Chart
I further investigated the daily number of Citoid editing events and confirmed there were no any sudden changes in the number of automatic citation generation failure events right around August 3rd. The number of daily citoid sessions that have failed to insert a reference due to a network failure to retrieve results has remained stable before and after the change in Citoid traffic.

From this, it still appears that the 3 August 2024 change had no impact on overall user success or failure rates using Citoid on desktop or mobile.

cc @ppelberg

In T372438#10188550, @ppelberg wrote:

In T374624#10158451, @MNeisler wrote:

Of the times when people open Citoid and do NOT follow through to insert a reference, what reasons explain why this might be the case?

Overall proportion of failed citoid editing sessions by reason:

Reason	July 2024	August 2024
Automatic citation generation failed	19.1%	19.3%
* Network error while attempting to retrieve the necessary information	18.4%	18.7%
* No search results were found	0.7%	0.6%
Session aborted	80.9%	80.7%

As found in T368988#10123059, about 19% of all citoid editing sessions that failed to insert a reference were due to an automatic citation generation failure. Of these sessions, the majority of citation generation failures were due to a network error while retrieving citation info. Network errors account for 18.7% of all citoid sessions that failed and 97% of all citoid sessions that failed due an automatic citation generation failure.

There were no observed changes in observed failure rates before or after the decrease in Citoid traffic on August 3rd.

Per platform proportion of failed citoid editing sessions by reason:
Per platform results are very similar to the overall results. The majority of citation generation failures on both desktop and mobile were due to a network error. 97% of all citation generation failures on desktop were due to a network error and 95% of all citation generation failures on mobile were due to a network error.

Time Series Chart
I further investigated the daily number of Citoid editing events and confirmed there were no any sudden changes in the number of automatic citation generation failure events right around August 3rd. The number of daily citoid sessions that have failed to insert a reference due to a network failure to retrieve results has remained stable before and after the change in Citoid traffic.

From this, it still appears that the 3 August 2024 change had no impact on overall user success or failure rates using Citoid on desktop or mobile.

cc @ppelberg

Thanks @MNeisler. I'm a little confused by the language "network error" and "no results" with network error being 20%, with no results being so low. From the service end of things, ~20% is the right percentage of "no results found" - I don't see how that would translate into such a tiny percentage? What does "network error" and "no results" here mean in terms of status codes?

403 responses:

July:

August

I think this probably mostly reflects that our users use a lot more news sources, whereas third party usage was more academic, so more use of academic pages i.e. cell.com

1. How do the following metrics compare in the two weeks before and after 3 August 2024?

July 8-22: absolute failure rate 21%, 74.8% 200 OK for all, 73.3% OK for zotero (third party mostly), 87.9% OK for other formats (mostly us)
Aug 8-22: absolute failure rate 13.3%, 80.06% 200 OK for all, 78.68% OK for zotero (third party) 84.9% OK for other formats (mostly us)

So the weird thing about this is that it appears although overall things improved, actually for us specifically it got slightly worse; the improvements were more for other people using zotero format.

Request volume be origin (3rd party requests vs. Wikimedia requests)

Volume:

Overall request volume for zotero (red) went down, although you can see there are some spikes there. mediawiki requests remained stable (as predicted.)

Thanks @MNeisler. I'm a little confused by the language "network error" and "no results" with network error being 20%, with no results being so low. From the service end of things, ~20% is the right percentage of "no results found" - I don't see how that would translate into such a tiny percentage? What does "network error" and "no results" here mean in terms of status codes?

Hi @Mvolz That analysis was based on the two new Citoid feature events instrumented in T363292 . The meaning of these events are based on the following meaning documented in the VEFU data dictionary:
automatic-generate-fail-searchResults: Automatic citation generation failed because no search results were found for the provided input.
automatic-generate-fail-network: Automatic citation generation failed due to a network error while attempting to retrieve the necessary information.

I'm actually not sure on what these mean in terms of status codes. @DLynch - Do you know based on how these new CitoidInspector feature events were implemented how they might align with the status codes?

Looking at how it was done...

Which is to say: if citoid is returning an API error when we encounter a non-200 status code, all of that should get swept into the network category.

Per what @MNeisler and I talked about offline, we think a discussion with David, Marielle, Megan, and me is a good next step to resolve the discrepancy between what Megan discovered in T374624#10158451 and what Marielle discovered in T372438#10191087.

In T372438#10196542, @DLynch wrote:

Which is to say: if citoid is returning an API error when we encounter a non-200 status code, all of that should get swept into the network category.

In T372438#10196433, @DLynch wrote:

Looking at how it was done...

automatic-generate-fail-searchResults	The request made it all the way through the reliability check, tried to build the templates from the citoid results, and failed at that point (probably because there were no results, but could also include any other errors in building the templates)
automatic-generate-fail-network	The promise from the mw.Api request was rejected - without any testing for further details, so I think this is a consolidation of any possible API errors + any network errors that interrupted the request

I think these events are misleadingly labelled, then.

automatic-generate-fail-network is more like no search results from the api. When we don't have results, it returns a 404 (or a 415 if we don't have results because it's a pdf). This likely is the majority of cases and the numbers roughly match the api numbers. Only rarely would it be some sort of connection issue.

You don't get to network success unless there are search results. So the automatic-generate-fail-searchResults thing means there were definitely search results from the API. However, in some rare cases even if there are results from the API, we fail to build a template. One reason might be if the template listed for it doesn't have any template data. Then we can't build the template, so it fails.

I guess it's plausible that we should pass this mislabeling on to WMDE -- they added these specific logs in support of something, so it's possible they've drawn incorrect conclusions based on that?

Marking this as done. We no longer have these data, even if we wanted to do more!