Not sure if we would be able to do that kind of comparison in a SQL query

So this has been resolved? 13GB of the 20230701 dump was so large why? Because it contained duplicate documents? Otherwise it is unclear why it is just 9.6 GB now.

Those are very much useless because they contain wiki markup and this is the reason why Enterprise HTML dumps are so useful, because they contain rendered page. Wiki markup content is useless because a lot of content gets pulled in through templates and other mechanisms. HTML dumps contain all that.

Some time ago I made T300907 to ask for HTML Enterprise dump of Wikimedia Commons. I think that could be seen as 10% of what this issue is. Images namespace is already dumped for English Wikipedia and having it be also done for Commons would allow one to use it for AI training - you would get image description and other information and links to image/media. This is similar to how other AI training datasets look like, they just point to media on the Internet but generally do not include the media itself (also for copyright reasons). So it is something people are familiar with.

Is there any way I could help push this further? I really think it would be very useful to have coverage of commons as well. I would guess that also enterprise users would love to have those dumps now that everyone is training AI models on images + descriptions.

Is there any way I could help push this further? I really think this would be beneficial as it allows one to better understand structure of the content you can obtain from HTML dumps.

I made a topic at https://meta.wikimedia.org/wiki/Talk:Wikimedia_Enterprise#Public_access_to_on-demand_API

Is the On-Demand API (docs link I gave prior) what you're looking for in regards to functionality? It grabs the current version of a single article. That is what you're asking for yes?

And the article lookup API is not something which would be made publicly? Maybe at a deployment with lower SLA and rate limited differently than enterprise users? I simply find the contents aggregated in JSON per article very useful to get in one API query. Especially if you start with a dump and then want to fill in gaps or update local info about articles.

Is push endpoint available to the community members? Or just enterprise customers? Could you please provide a link to documentation? Maybe I missed something?

@FNavas-foundation Not sure what you mean. Maybe I am missing something, but revision.id by itself does not give you the count I am asking for, i.e., the id is not local to the article and does not increase by 1 every time the article has changed. Or am I mistaken?

so why not simply base the HTML dumps off them?

Sadly, the stats are not easily available for other namespaces (while dumps are made for them as well). I think currently the best way to get those counts is to use ElasticSearch endpoint, as described in T312200.

BTW, there is a related issue T300124 that some categories and templates are not always extracted.

You are right. It is listed now in docs.

@Sascha also added benchmarks for different compressions algorithms to https://github.com/brawer/wikidata-qsdump. Thanks for that!

Awesome! Thanks. This looks really amazing. I am not too convinced that we should introduce a different dump format, but changing compression seems to really be a low hanging fruit.

Yes, great summary. Thanks.

I think it would be useful to have a benchmark with more options: JSON with gzip, bzip (decompressed with lbzip2), and zstd. And then for QuickStatements the same. Could you do that?

To my knowledge it is. https://www.mediawiki.org/wiki/Wikimedia_REST_API#Terms_and_conditions still says that 200 requests/second per REST API endpoint is fine (unless documented to have less, for example transform API endpoint) and configuration says differently: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/varnish/templates/text-frontend.inc.vcl.erb#431.

Yes, I made a library for processing those dumps in Go.

In large dumps there are multiple files inside one archive. So tar serves as a standard way to combine those multiple files into one file, and then compression is made over all of that.

So tar format is really made for streaming, so I am surprised that this is hard to do in your programming language. Seeking is what is problem in tar, but streaming is really easy. It is really just concatenation of files. So it is similar to any other buffered stream.

I will respond about tar layer in T332045 you made.

In OIDC however the same data should also be returned alongside the access token

I agree. Let's close this issue then and track OIDC parameters in the issue you referenced.

So there is API call to get all categories/templates which is then included into dumps. The issue here is that sometimes that call seems to fail and not have data included.

You should use a parallel gzip decompressor. Just using standard gzip (which is invoked by tar) is not that.

How can you parse all the templates used from Parsoid HTML? Some templates do not have any output into HTML?

Since I opened that issue I learned more about OIDC and I think there is no need really for authenticate endpoint anymore. authorize and access_token endpoints currently are good enough. I think what throw me off is that (at least at the time, I have not checked recently) is that it looked like there is no way to get automatically redirected back to the app if user is still signed-in into the Mediawiki instance AND has already authorized the app. Other OIDC providers just redirect back. But Mediawiki (at the time at least) always showed the authorization dialog every time. So one click sign-in flow was less fluid than one would like to have.

Is this something a community contribution could help with?

So what is the plan about this? Is this something a community contribution could help with?

A gentle ping on this. I understand that it would be a large dump. But on the other hand it is a very important one: Wikimedia Commons is lacking any other substantial dump so having a dump of file pages would help one obtain at least descriptions of all files through a dump. That can be useful for many use cases, like training AI models, search engines, etc.

Any luck finding a reason?

A workaround is available in this StackOverflow question/answer: https://stackoverflow.com/questions/73223844/get-the-number-of-pages-in-a-mediawiki-wikipedia-namespace

Just HTML dumps. So what you provide here https://dumps.wikimedia.org/other/enterprise_html/ but also for commons wiki. (You already provide namespace 6 for other wikis.)

I tried now to use API to fetch things myself, but it is going very slow (also because rate limit on HTML REST API endpoint is 100 requests per second and not documented 200 requests per second, see T307610). I would like to understand if I should at least hope for this to be done at some point soon or not at all. I find it surprising that so many dumps are made but just this one is missing. Would that be just one switch to enable dump on one more wiki?

OK, it is not connected to characters in the filename. There are files in entities with above characters. But I do not get why not all files on Wikimedia Commons have entities.

Oh, what a sad issue T149410. :-(

Hm, I am pretty sure that I am doing rate limiting correctly on my side, but I am hitting 429s after a brief time when trying to do 1000/10s rate limit to the REST API endpoint. If I lower it to 500/10s then I do not hit 429s. No idea why, but I am doing many requests in parallel.

Hm, there was no response since February. :-( OK, I will wait.

Who could I ask from their team about this?

So if I understand correctly, those files have never been generated so that particular dump for that particular date will not be available?

@ArielGlenn: Do you think dumps of file descriptions (so not media files themselves, but wikitext rendered) could be provided for Wikimedia Commons as part of public Enterprise dumps? Given that so many other wikis are generated, why not also Wikimedia Commons? This could help me obtain descriptions for files on Wikimedia Commons (and given already no other dumps for Wikimedia Commons, it would help me hit its API less).

@Protsack.stephan Was there any progress on this?

I checked commons-20220620-mediainfo.json.bz2 and it contains title field (alongside other fields which are present in API).

I checked wikidata-20220620-all.json.bz2 and it contains now modified field (alongside other fields which are present in API).

So for the next dump which will run, this will now be included? Or is there some deployment which is still necessary?

Awesome. I will try to do so when you are online, but feel free also to just merge it without me. I do not know if I can be of much help being around anyway. :-)

What is this subsetting you are talking about?

So what is the next step here?

Yes, this change should fix both this issue and T278031.

Awesome, thanks!

Thanks for testing!

Done. Added it to June 7 puppet request window. Please review/advise if I did something wrong.

Awesome. Thanks for explaining.

So fix to the dump script has been merged to the Wikibase extension. It is gated behind a CLI switch. What is the process that this gets turned on for dumps from Wikimedia Commons (and ideally also for Wikidata)?

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/793934 is ready for a review, it has both opt-in configuration option and a test.

I thin this might be related to T274359.

I think this might be related to T305407.

I made another pass, adding configuration option to not include page metadata (then dump is without title and other page metadata).

I made a first pass. Feedback welcome.

So the plan is:

Most of that is controlled by the SRE team at a level in front of the REST API, since the frontend caching layer is a shared resource across everything.

Because our edge traffic code enforces a stricter limit of ~100/s (for responses that aren't frontend cache hits due to popularity), before the requests ever get to the Restbase service.

Sadly bulk downloads do not have HTML dumps, and Enterprise dumps do not offer them for template/module documentation (only articles, categories, and files). Also, there are no Enterprise dumps for Wikimedia Commons.

Hm, but documentation for REST API says I can use 200 requests per second? https://en.wikipedia.org/api/rest_v1/

Even if you request a single title, I think you still might get continue param.

Are you using Mediawiki API to obtain categories and templates? I am betting you are not processing continue properly to merge multiple API responses when one batch of data is distributed across multiple responses. You have to merge data, otherwise some pages look like they have no templates/categories. I just now encountered that when I was using API to populate templates/categories manually (because dumps are missing them randomly). I used the following API query and you can see with some luck that some of returned pages are missing templates/categories, because you have to follow continue params, but then other pages are missing. Only when batchcomplete is true you know you got everything (but you have to merge everything you got before that).

I think I misunderstood in T301039 from documentation that those pointers are pointing to the text table.

I would be interesting in doing that, but I probably need a helping hand to do it. So I have programming background, but zero understanding of where and how this could be fixed. My understanding is that hackathon would be suitable for this? Do I have to make a session? How do I find other people who might be able to help me?

I added it to wikimedia-hackathon-2022. I think it would be a nice thing to fix as part of it.

Thanks for linking to that task.

I was interested in this primarily for my own self-approved app used only by me. There it should be trivial to just change grants.

Hm, should this be prioritized more? It is 8 years now.

Is there any way I could help to push this further?