@abi_ Thank you for confirming, we will work on this in the next days. Thank you in advance for your patience.

@abi_ Kind of suspect that the strings should be stored in multiple files (like one per language)

@RBrounley_WMF Thank you for opening this ticket, I will follow up.

@abi_ THX!!!

@abi_ I confirm that the github "translatewiki" has write access to the repository and I have just created the qqq.jsonfile, see https://github.com/openzim/mwoffliner/pull/1771.

We are cursed...

@Rgaudin Do you have any hint about this last comment from @Hokwelum?

Just FYI, no need anymore on MWoffliner side as we use an online object cache meanwhile.

This ticket is really old. We have made many improvement in our toolchain software and infrastructure (many backoff strategies) in the meantime. I can not assess if the problem is still there or if it has been fixed; but whatever the current status, we don't suffer of it anymore. I would be fine to close it.

@abi_ Thank you for the PR, currently under review!

At MWoffliner we have started to study the problem. Because of this ticket but as well because our unstable experience with the usage of this end-point. See https://github.com/openzim/mwoffliner/issues/1730. Hopefuly Wikimedia does provide other better end-points allowing us to retrieve the same information.

@abi_ Thank you! I will create a qqq.json and remove the metadata. When can we expect a first PR from Translatewiki to check the process works fine?

I guess this ticket can be closed?

In T324866#8459496, @Ladsgroup wrote:

I can think of several (I don't know the details of your system and might have missed something):

• Use Wikimedia Enterprise (@RBrounley_WMF in this ticket is PM of WME), I think your case can be easily justified for a free tier: https://dumps.wikimedia.org/other/enterprise_html/

@Ladsgroup The MWoffliner scraper has already been quite optimised over years. I have no obvious improvement in mind for the moment but we will consider any concrete new proposal.

Would that https://github.com/openzim/mwoffliner/issues/1664 fix the issue, so far we are really not familiar with this new API end-point?

On my end, this looks better now. Thx!

The reported bug seems indeed to have "vanished". Thank you for the good work.

If you can delete the whole thread https://lists.wikimedia.org/hyperkitty/list/wikifr-l@lists.wikimedia.org/thread/J2FU23D4C5ERWIK2LWBYUBTYK3O6KP6Y/, then problem would be solved. It's a pity we can not hide it. I don't like that we manipulate the archive.

@Ladsgroup Thank you but I don't see these two options below the "Add to favorites"... although I'm owner on the list.

See also https://github.com/openzim/mwoffliner/issues/1482

@Dzahn Great, thank you very much!

I have achieved to create account an retrieve admin access to the list, but really no clue how to remove the few messages (from the public archive). Not even sure this is possible at all. As a consequence, I have put the whole ml archive in private. Unfortunately, this seems to fail, archive is still accessible at https://lists.wikimedia.org/hyperkitty/list/wikifr-l@lists.wikimedia.org/latest

In T260223#8311261, @Hokwelum wrote:

In T260223#8302299, @Kelson wrote:

During September we have detected that one of the mirror was full and therefore rsync client was keeping download and redownloading things. As a consequence a significant part of the bandwidth and the rsync daemon slots were always busy. We still lack of monitoring solution for this on our side honestly. We should figure a tool and a process to better monitor the server/service. If you have propositions, they are welcome.

Hello Kelson, We would like to know if the space problem has been resolved as we received the same error message today.

@nskaggs Not sure this is a question to me, but in the case it needed, could you please change the ZIM mirroring rsync command to:

• request on master.download.kiwix.org in place of download.kiwix.org
• sync the following repositories in addition to wikipedia: wikibooks, wikinews, wikiquote, wikisource, wikiversity, wiktionary, wikivoyage

It seem that this bug partly explains why ZIM files of Wiktionary in English are crappy, see https://github.com/openzim/mwoffliner/pull/1665

It seems the bug reported in T274359 is the same as this one, so reputting here the bug description:

@matmarex Strange to close this ticket considering the bug described in the ticket is not fixed!

During September we have detected that one of the mirror was full and therefore rsync client was keeping download and redownloading things. As a consequence a significant part of the bandwidth and the rsync daemon slots were always busy. We still lack of monitoring solution for this on our side honestly. We should figure a tool and a process to better monitor the server/service. If you have propositions, they are welcome.

@komla What would be the way to proceed:

• Can you delete everything except the DB (no sure what is the same BTW)?
• Should we somehow migrate the DB first?

@Audiodude You are better informed about the current situation but AFAIK, we only still the DB that we use in the VPS we have now, all the rest could be deleted. Can you confirm?

This ticket is a kind of blocker for new Kiwix/openZIM selection tools based on Wikidata https://github.com/openzim/wp1/issues/519 and therefore Wikipedia on Demand project https://meta.wikimedia.org/wiki/Kiwix/Wikipedia_on_demand

This ticket is a kind of blocker for new Kiwix/openZIM selection tools based on Wikidata https://github.com/openzim/wp1/issues/519 and therefore Wikipedia on Demand project https://meta.wikimedia.org/wiki/Kiwix/Wikipedia_on_demand

Same problem with an article of WPJA. Whereas it works fine for human interface:

$curl -L 'https://ja.wikipedia.org/wiki/%E3%81%8A%E3%81%A7%E3%81%8B%E3%81%91%E3%83%AC%E3%82%B9%E3%82%BF%E3%83%BC%E3%82%8C%E3%82%8C%E3%82%8C%E3%81%AE%E3%82%8C(%5E%5E%3B'

For the record, this 10 years old ticket, has been a real bug. It happened at the time we discovered we had to have relative links in HTML inside the ZIM to allow a certain flexibility in the way how the ZIM could be "mounted". This was and is particulary true with kiwix-serve. I believe this bug to have been open at a time the libzim tracker was in Phabricator, so the bug was at the right place. Unfortunately, the tagging seemed wrong and this ticket has - looks like - not been closed at the time the libzim bug tracker has been moved to openZIM organisation (which is already a long time ago). Obviously, since years, ZIM should provide (and provide) HTML content with relative URLs.

Cannot reproduce the problem here (central Europe):

@nskaggs @Aklapper Not sure to fully understand the question. The answer has been posted earlier this year at https://phabricator.wikimedia.org/T57503#7655695 (but you should not mirror "wikihow"). So just remove the "wikipedia" folder (which is already mirrored) and you will have the answer, which is a bit less than 0.5 TB.

@Arlolra @ssastry Have checked again this and the bug is still valid. Almost 3 years have passed and I wonder if you are not far enough to decide either you will fix it or close it as WONTFIX (But then a kind of communication/measure should probably be taken to inform Wiki editors.. and fix the few articles concerned).

Here an other MWoffliner ticket open by a user but for an other page https://github.com/openzim/mwoffliner/issues/1617

Bug still there:

• https://en.wikivoyage.org/w/api.php?action=query&format=json&prop=revisions&titles=Santa_Cruz_de_Mompox delivers revid 4430914 (CORRECT)
• https://en.wikivoyage.org/api/rest_v1/page/mobile-sections/Santa_Cruz_de_Mompox delivers revid 3705129 (INCORRECT, version of the 15 January 2019)

As far as I can check the ticket is not fixed. Here an example:

• https://en.wikivoyage.org/w/api.php?action=query&format=json&prop=revisions&titles=Santa_Cruz_de_Mompox delivers revid 4430914 (CORRECT)
• https://en.wikivoyage.org/api/rest_v1/page/mobile-sections/Santa_Cruz_de_Mompox delivers revid 3705129 (INCORRECT, version of the 15 January 2019)

Looping in @Andrew. @Kelson note that yes, we are installing new, more capable machines that have more capacity than in years past. Once they are up and running, we can explore mirroring this additional data.

@ArielGlenn Can you please reassign the ticket? I have no clue who - concretly - is WMCS?

Learning only now about this bug thank to @Krinkle. From MWoffliner perspective this has a serious impact on the way how it works. The reason: without the ETAG, we have no way to compare it to our media cache, therefore each media file will be downloaded again from the Wikimedia backend (in place of the MWoffliner S3 cache). For us, the performance impact is probably not that big (we download everything from Wikimedia backend instead of balancing things between the S3 cache and Wikimedia. But for the Wikimedia backend, this is not the same story.

@Andrew Thank you for finally completing this task!

@ArielGlenn It seems that T302981 has just been implemented. Does that mean you have no blocker anymore for this task?

@ArielGlenn What is the status here? We have kept a bit more slots and we have as moved to a stronger server, 2 months ago.

@Krinkle I have rechecked this bug/ticket with the given example and now it works. Might that be that the bug has been fixed?

I have rechecked these 4 URLs posted in my lastest comment 6 months again. I can confirmed that all the four of them still failed with a HTTP 504 error.

@Andrew VM recreated.

@Andrew OK, shit happends time to time, I will recreate the VM during the WE.

@Andrew All our VMs have been migrated to Debian-bullseye. Therefore this ticket can be closed

@Reedy Thank you!

@Reedy Would you be able please to help here?