Glancing at WMDE's patch that added this instrumentation, it doesn't quite work how it's documented on the VEFU page. Specifically, automatic-generate-fail-searchResults and automatic-generate-fail-network won't be logged under the "citoid" feature, but rather under the "CitoidInspector" feature. Depending on exactly how the queries were being run in T368988, that'd probably be enough to make it not show up in the analytics.

Said patch did also add the automatic-generate-fail action, which would be logged under citoid, and should theoretically just be the overall sum of those other two. So depending on what you were relying on to detect failure, the results might not even be impacted.

I've updated the VEFU dictionary to reflect reality.

Thanks @DLynch for checking and updating the documentation. I've confirmed I can find the automatic-generate-fail-searchResults and automatic-generate-fail-network under the CitoidInspector feature.

I was using automatic-generate-fail to account for all events where automatic citation generation failed so, as you indicated, I'm not expecting any changes to the overall failure rate observed in T368988#10072946 assuming that it is just sum of the fail-network and fail-searchResults events.

We were curious if there were any shifts in the reasons for citation generation failure using these two new events following the Aug 3rd decrease in traffic or at least have a clearer understanding of the reasons for citation generation failure.

@ppelberg I can re-run the failure rate portion of the analysis to provide a more granular breakdown of citoid failure reason and look at changes pre and post August 3rd. I've created an T374624 for that work. Feel free to update as needed.