In T386886#10592367, @GFontenelle_WMF wrote:

Hi @Dominicbm! Just to let you know, 5 categories were already in the system:

Postcards_published_by_E._A._Bardol_%26_Co.
S._H._Kress_%26_Co.
Postcards_published_by_Cross_%26_Dimmitt
Postcards_published_by_Leighton_%26_Valentine_Co.
Postcards_published_by_Western_Publishing_%26_Novelty_Co.

In T381352#10577170, @Anthere wrote:

I finally tested today. Works pretty well. I have not tested all options... but several of those of interest to me and that was ok !
Will explain the team how to use that. Need to figure out which output we want to do graphs for, and then how to best do that.

On another note... @Dominicbm , I saw that WLA categories are in your tool till 2023. Would it be possible for you to add WLA 2024 for us ? Thanks !

In T384752#10565218, @mforns wrote:

Hi @Dominicbm, thanks for the report!

Indeed, categories whose leveraging pages have no pageviews return 404 in the pageviews-per-category-monthly endpoint.
And so do media files whose leveraging pages have no pageviews in the pageviews-per-media-file-monthly endpoint.

I understand that this is annoying from the user's point of view.

Our initial design included metrics for all categories and media files, as you suggested.
However, the sheer number of Commons categories and media files with 0 pageviews in leveraging pages made it so
that most of the storage resources consumed by some of the Commons Impact Metrics datasets were due to those 0-pageview elements.
One of the challenges of this project was reducing its monthly snapshot data size, so that the overall data size was sustainable,
for both the source datasets in our data lake and in our serving layers, and the public dumps to be of a manageable size.
So we made the decision to not store data for those mentioned categories and media files, drastically reducing the size of the datasets.
This allowed us to work with less constraints on other parameters, like the depth of the category tree, and the size of the allow-list, which are also multiplying factors.
It also helped us to deliver the data product on time with the allocated resources.

In T371909#10110664, @mforns wrote:

Thanks again @Dominicbm!

Things we could do here include:

• Improve documentation to require YYYYMM01 dates instead of YYYYMMDD ones. [DONE]
• Improve error message from Cassandra that requires YYYYMMDD, and should require YYYYMM01 instead.
• Make the service fail fast for dates that do not match the YYYYMM01 format, this would eliminate the confusion about using i.e.: 20240124.
• Allow start and end dates to be just YYYYMM. Keeping support for YYYYMM01 format for backwards compatibility. This would require a bit more work, because of the extra testing and changing docs.

@GFontenelle_WMF I have edited the list to remove the ones that were already on the allow-list but returning errors for other reasons. It's a bit shorter now!

In T382733#10480190, @mforns wrote:

@GFontenelle_WMF The deep metrics are only available for the primary categories (the ones in the allow-list).
This restriction is unfortunately necessary. The size of the data would be too big if we gathered those metrics for all non-primary categories.
I imagine the categories you mention are not primary?

@Dzahn Sorry, for the late reply. I am doing nothing on Phabricator at all besides occasionally viewing it or making the occasional ticket/comment. Certainly nothing high-volume. (And it's news to me that there are high-volume requests to Phabricator, even on an AWS IP, but I hoped being logged in would be enough to not get caught in any ban.)

I suspected something like that, it just seemed a bit aggressive. No other web site on the Internet prevents me from even reading it, especially when I am logged into a trusted account in this case. I guess the fact that it was a 429 error was what made me think there was a chance my own bot activity was related. While I don’t know what other activity there is on the IP, this error is given at all times l, so it’s kind of surprising if there are constantly enough new connections being opened to experience that.

In T383435#10464293, @Dzahn wrote:

Could you tell us more about that IP? Is that in "the cloud", as Amazon AWS or a similar cloud provider as opposed to a regular home IP?

I hadn't commented here before today because it is very de-motivating in situations such as this when you're being asked by the WMF to give the same feedback in 5 different places across the last 3 years and to multiple different WMF teams—even when it has already been made fairly clear that outcome that is clearly preferred by the community is not even on the table anyway. I agree with what essentially everyone else has ever said to the WMF on this matter. WCQS is only really useful insofar as it is reliable and accessible by bots over the endpoint. I am someone who has uploaded over 5 million image files to Commons and maintains tens of millions of SDC statements across all of them, and would have liked to have used the query service for very real applications in analyzing and synchronizing the data as part of this workflow. Instead, I long ago had to figure out ways to totally work around it, and now no longer think about using WCQS much. My other main use case is making reports for analytics purposes, but I literally cannot link anyone who is not already a logged-in Wikimedian to results from this service or they will be blocked from accessing the data. I can't make a third-party dashboard, since I am also prevented from, e.g., querying the data with some JS on another site.

In T350793#10378435, @Jelto wrote:

Yesterday, there were reports of commons-query.wikimedia.org being unavailable, so we rolled back the query-scholarly switch. However, the issue with commons-query persists and continues to return upstream request timeout.

Since the query-scholarly switch is independent of commons-query, I'll re-apply it.

This wouldn't block anything, it's just for UX. I think it's nice for the convenience factor, and I wouldn't expect expect it to be very difficult (the data is already there).

Yes, purely for UX. I am not blocked by it, but I could see how someone else might not figure out how to do this correctly.

I would like to echo this suggestion, for a couple of reasons. One observation is that the pageviews metric can sometimes overstate the true usage of a file. As an example, you have long article that is very popular, and are measuring the usage of an image embedded far down in the article. Realistically, most people are not viewing it or even loading it at all, considering how mobile page lazy loading works, and this is borne out by the mediarequests stats. I suspect this means high-usage collections, or especially collections with several images of popular subjects, are getting inflated.

In T370448#9996627, @mforns wrote:

It is possible! Try:
https://wikimedia.org/api/rest_v1/metrics/commons-analytics/pageviews-per-category-monthly/Media_contributed_by_the_Digital_Public_Library_of_America/deep/all-wikis/00000101/99991231

Now, because of you creating this task I saw that the documentation for the wiki field is cut out. It's actually cutting out the part where the "all-wikis" option is mentioned.
We can use this task as a bug report to fix that. It is a simple documentation change.

I think I disagree that they mean two things in this context. Because the Wikidata property scope is defined as one thing, which is "primary topic of a work". Similarly, DPLA's subject field has the definition "Topic of described resource," and is based on DCMI's subject element: "The topic of the resource". That's why we mapped these. So, while the word "subject" has two different meanings, to the extent that anyone is using the P921 property on photo to mean subject as "what is shown in the image" instead of subject as "main topic of the image"—then they are not using the Wikidata property properly according to its defined scope, and that is a data issue not a modeling issue. And if someone wants to say that first type of subject, that is what "depicts" is properly for. Conceivably, one image should be able to have one, both or neither of "depicts" and "subject" which might be the same or different values, and those could all be meaningfully valid depending on the case.

One other practical note I want to add is that there are many other examples where the subject is what is depicted. We can't add depicted statements programmatically at this time, since no metadata standard really includes that as an element, and so no institutions are describing their collections in that way. We can't assume. What they do have is "subject". So the situation is that there may be many subject statements that aren't depicted (as in the original example), but there are also many subject statements from bulk uploads that simply are not yet added as depicts statements by a human, but the subject statement is a good indicator that it might be a depicted entity.

Speaking for DPLA, I think your reading of the situation is right. We are adding P921 statements because that is as far as the data source allows us to go. It only states what the subject of the item (in its entirety is), but not what is contained within any individual image. I think there is understanding that this is less of a signal, in terms of search, than a P180 statement, which indicates the subject is actually depicted.

In T358695#9632553, @FRomeo_WMF wrote:

These are instructions for Wikimedia Foundation staff.

We're building a list of the categories already tracked by the GLAM Wiki Dashboard, Cassandra Dashboard, and BaGLAMa, and they will be part of the initial set-up. Are there other tools we should be looking at?

@xcollazo Are these instructions for the community at large, or for WMF? It seems like there should ideally be a Commons interface way of doing this, unless part of the goal/expectation is to limit the number of categories on the list. Currently, we have about 2000 Commons categories tracked, at least, in BaGLAMa and similar tools already. Would this process handle that number of requests?

Hi @mforns, sorry for late reply. I think I am not sure how the Commons Impact Metrics project is going to affect the existing AQS APIs. For my own part, if I have the ability to query data at the category level instead of per-image, I will do that for my own work. That seems to be the main difference conceptually between the two. So Commons Impact Metrics might lower the prioritization for this task for me personally. However, as long as the mediarequests API is still maintained, this feature is still needed for it, because it is a major performance and usability issue to not be able to query a Commons file without first getting its path in a separate query.

In T345786#9148039, @JJMC89 wrote:

# is permitted in Page object titles since it represents a link to a section. Spaces before it get removed since MediaWiki page titles don't have trailing whitespace.

In T345786#9148030, @JJMC89 wrote:

# is not permitted in titles. See mw:Manual:Page title#Invalid page titles.

In T345209#9129778, @Xqt wrote:

Instantiation the APISite tries to login with stored cookies but it does not use autocreate [1] for logging in. There might be reasons for it; or should there be a config setting to enable it? A possible way to prevent this issue is to login first via login script: pwb login -all -autocreate -async [2] but I am not really sure whether this works for oauth login because this is quite different from client login.

[1] https://doc.wikimedia.org/pywikibot/master/api_ref/pywikibot.site.html#pywikibot.site._apisite.APISite.login
[2] https://doc.wikimedia.org/pywikibot/master/utilities/scripts.html#login-script

Thanks for reporting this. I consider WCQS pretty much broken in this state, since its results are unreliable and not even showing any sort of error message making this clear. You can do things like changing the whitespace to trigger a refresh of the query, but it's not obvious when you've gotten the "real" answer except when it feels right.

I came here looking for this. To expand on the ask, PageViewInfo metrics currently available are pageviews, siteviews, and mostviewed.

I think it’s the commons.wikimedia.org cookie you need to clear.

In T188684#7521655, @rook wrote:

This was removed...However it was removed in October, not August. I've put in a PR to return the timeout to 24 hours. Though I wasn't able to recreate the results in minikube when I set a short timeout, job seemed to be running after the timeout and the ten minute cull loop.
https://github.com/toolforge/paws/pull/108

I am experiencing this issue as well. I can't upload a 6MB JSON file, when I have uploaded much larger files before—I think even hundreds of MB.

That seems to have done it, I'm in! Thanks so much.

I tried again, and get a different error message:

Logging out and in again didn't do it for me, even with clearing cookies just in case.

I am reopening, since nothing seems to be solved for me. I am getting nothing besides the 500 error for my account DPLA_bot.

When I try, I get: