Hi @Dominicbm, thanks for the report!

Indeed, categories whose leveraging pages have no pageviews return 404 in the pageviews-per-category-monthly endpoint.
And so do media files whose leveraging pages have no pageviews in the pageviews-per-media-file-monthly endpoint.

I understand that this is annoying from the user's point of view.

Our initial design included metrics for all categories and media files, as you suggested.
However, the sheer number of Commons categories and media files with 0 pageviews in leveraging pages made it so
that most of the storage resources consumed by some of the Commons Impact Metrics datasets were due to those 0-pageview elements.
One of the challenges of this project was reducing its monthly snapshot data size, so that the overall data size was sustainable,
for both the source datasets in our data lake and in our serving layers, and the public dumps to be of a manageable size.
So we made the decision to not store data for those mentioned categories and media files, drastically reducing the size of the datasets.
This allowed us to work with less constraints on other parameters, like the depth of the category tree, and the size of the allow-list, which are also multiplying factors.
It also helped us to deliver the data product on time with the allocated resources.

In T384752#10565218, @mforns wrote:

Hi @Dominicbm, thanks for the report!

Indeed, categories whose leveraging pages have no pageviews return 404 in the pageviews-per-category-monthly endpoint.
And so do media files whose leveraging pages have no pageviews in the pageviews-per-media-file-monthly endpoint.

I understand that this is annoying from the user's point of view.

Our initial design included metrics for all categories and media files, as you suggested.
However, the sheer number of Commons categories and media files with 0 pageviews in leveraging pages made it so
that most of the storage resources consumed by some of the Commons Impact Metrics datasets were due to those 0-pageview elements.
One of the challenges of this project was reducing its monthly snapshot data size, so that the overall data size was sustainable,
for both the source datasets in our data lake and in our serving layers, and the public dumps to be of a manageable size.
So we made the decision to not store data for those mentioned categories and media files, drastically reducing the size of the datasets.
This allowed us to work with less constraints on other parameters, like the depth of the category tree, and the size of the allow-list, which are also multiplying factors.
It also helped us to deliver the data product on time with the allocated resources.

I think I understand what you're saying, but isn't is already possible without storing more data to differentiate between the cases where the category isn't in the allow list vs. the case where the pageviews are 0? I just figured it out as a workaround by checking /pageviews-per-category-monthly/. Isn't there a workflow where rather than storing all the "0"s, you check if the category is valid by other means (such as that), and then you know if it's a valid 0 result that should have a 200 HTTP code, or if it's an actual 404?

You get this 404 even if the category does have page views, such as the specific wiki you put in the parameter doesn't have views (or maybe deep search has views and shallow doesn't, but I don't have an example). e.g.: https://wikimedia.org/api/rest_v1/metrics/commons-analytics/pageviews-per-category-monthly/Media_contributed_by_Colorado_State_University_Libraries/deep/ar.wiktionary/00000101/99991231

It's very simple to figure out in cases like this that the category is in fact loaded and just had no views, if I knew to check for that.