A more in-depth analysis will be made by Kiwix mid-September 2025 and the results will be shared here (and with the requesters at the WMF).

@daniel Hi Daniel, I think you are mostly entitled as you had open the issue there. Are we back now to normal?

I have rechecked again the 4 URLs which were failing in 2021. All of them seem to work now.

@Benoit74 Considering that:

• This mobile-sections API is going to be deprecated (see https://phabricator.wikimedia.org/T264671)
• MWoffliner does not rely on it anymore since version 1.14.0

Small update here, we have released MWoffliner 1.14.0 a few weeks ago. This new version does not support the mobile-section endpoint anymore and therefore we don't rely anymore at all on mobile-section endpoint on our end.

@Eevans Thank you very much for helping on this issue. This is really important to us.

@ABreault-WMF Hi, have you had the opportunity to reproduce the bug... and maybe even start to analyse it?

@Andrew Thank you very much!

In the meantime we have seen that this is happening everywhere quite often and therefore is a blocking MWoffliner to move forward properly through all articles and create ZIM files from Wikipedias, Wiktionary, ... Does something additional is needed to triage this bug?

@MSantos Just a comment about how often it happens. From Kiwix perspective it happens really a lot, we are barely able to scrape many Wiktionaries because of this bug. Probably the most serious issue we have for the moment in https://phabricator.wikimedia.org/project/view/4381/

If I give a bit more context about this request, I would invite you to read https://phabricator.wikimedia.org/T349972. In this ticket, the WMF has implemented in the mobile-html output, the original thumbnailing width for use/Kiwix, so we are in position to render the real thumbnail size as given by the wikitext. See https://github.com/openzim/mwoffliner/issues/1925 for the upstream (MWoffliner) ticket.

@abi_ Thank you! Seems to work fine!

Error is in the config file: git@git@github.com:kiwix/apple.git, this is not a valid address.

@abi_ I'm a bit puzzled by the situation. Translatewiki bot has access to the whole Kiwix organisation on Github and have good permissions for the kiwix/apple

In T351949#9370182, @abi_ wrote:

@Kelson Our current implementation of Apple file format does not support multiline strings such as these

"loc-external-alert" = "An external link is tapped. \
However, your current setting does not allow it to be loaded.";
"loc-article-cell-template" = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, \
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.";
"loc-ZimFileDetail-Alert-Delete" = "The zim file and all bookmarked articles \
linked to this zim file will be deleted.";

Would it be easy to avoid using of these literal new lines?

In T351949#9367633, @abi_ wrote:

In T351949#9359466, @Kelson wrote:

3. Once the project is added on translatewiki, please add message documentation for the project.

I have created the ticket at https://github.com/kiwix/apple/issues/561. Do we have an example of macOS/iOS software with that?

See: https://github.com/wikimedia/wikipedia-ios/tree/main/Wikipedia/Localizations/qqq.lproj

Currently blocked on this due to: T352300: Newlines in "Apple" format not supported

@abi_ We are working on it. See https://github.com/kiwix/apple/issues/571. I will come back to you within a few days when this will be done.

In T351949#9359151, @abi_ wrote:

1. Do you want translatewiki to commit to feature branch and then create a PR to main branch?

@matmarex Thank you very much for this fix!

Yes, this is on the radar, but we face pretty much difficulties around mobile-html new end-point for the moment. So not a TOP prio our side. I honestly don't think we will be able to tackle this before 2024.

No sure what is the exact status here, but on the size of Kiwix we have implemented the support of the new mobile-html REST end-point. Unfortunately we are not really able to create ZIM files because of https://phabricator.wikimedia.org/T349972. We still rely therefore on the mobile-section end-point.

Actually this ticket should be a bug, image resolutions are far too big for a mobile end-point and seems to be a waste of bandwidth. I have open a ticket at https://phabricator.wikimedia.org/T349972

@MatthewVernon That would be welcome on MWoffliner side! For the rest I can not say, but respecting the RFC sounds here quite important.

@ssastry AFAIK what is described seemscorrect to me. @vadim-kovalenko and @cscott would certainly be able to tell more as they both know MWoffliner and the Wikimedia backend infra.

@MSantos Thank you very much

@akosiaris @MSantos May I underline Vadim's request: carifying if we (at Kiwix) can still benefit from the mobile-sections API until we finish with the MWoffliner code update is an important and relatively time-dependant question.

@daniel Thank you! Things are clearer now for me. At the core of my problem is that we have always been using $wgContentNamespaces to know which namespaces should be scraped... and here it does not work that well (anymore).

@Arlolra Thank you for your technical explanation but I have a hard time to use it to answer my questions which are:

• Is the "Story" namespace really only a JSON, so the JsonContentHandler is appropriate?
• If this is JSON, is that really appropriate to have this namespace in https://www.mediawiki.org/wiki/Manual:$wgContentNamespaces?
• Is that normal ("it's a feature, not a bug") that https://id.wikipedia.org/api/rest_v1/page/html/Story%3AWisata_Wajib_di_Toraja returns an error and not the corresponding HTML (like for end-users)?
• If this is normal/wanted that no HTML is returned, then is that appropriate to have a link pointing to https://phabricator.wikimedia.org/T324711 (seems a bit unclear to me).

@cscott @ssastry Really sure this is fixed? Assuming this has been deployed, this is still failing https://id.wikipedia.org/api/rest_v1/page/html/Story%3AWisata_Wajib_di_Toraja. Actually this bug seems to be the reason why we can not generated ZIM offline snapshot of id.wikipedia.org since two quarters. More details about the bug at MWoffliner level at https://github.com/openzim/mwoffliner/issues/1853.

@Ladsgroup Thank you!

@Dzahn I reopen the task because it seems people can still post and subscribe to the mailing list. This should not be the case (and as moderator I would like to not get notices for this).

@abi_ Thank you for confirming, we will work on this in the next days. Thank you in advance for your patience.

@abi_ Kind of suspect that the strings should be stored in multiple files (like one per language)

@RBrounley_WMF Thank you for opening this ticket, I will follow up.

@abi_ THX!!!

@abi_ I confirm that the github "translatewiki" has write access to the repository and I have just created the qqq.jsonfile, see https://github.com/openzim/mwoffliner/pull/1771.

We are cursed...

@Rgaudin Do you have any hint about this last comment from @Hokwelum?

Just FYI, no need anymore on MWoffliner side as we use an online object cache meanwhile.

This ticket is really old. We have made many improvement in our toolchain software and infrastructure (many backoff strategies) in the meantime. I can not assess if the problem is still there or if it has been fixed; but whatever the current status, we don't suffer of it anymore. I would be fine to close it.

@abi_ Thank you for the PR, currently under review!

At MWoffliner we have started to study the problem. Because of this ticket but as well because our unstable experience with the usage of this end-point. See https://github.com/openzim/mwoffliner/issues/1730. Hopefuly Wikimedia does provide other better end-points allowing us to retrieve the same information.

@abi_ Thank you! I will create a qqq.json and remove the metadata. When can we expect a first PR from Translatewiki to check the process works fine?

I guess this ticket can be closed?

In T324866#8459496, @Ladsgroup wrote:

I can think of several (I don't know the details of your system and might have missed something):

• Use Wikimedia Enterprise (@RBrounley_WMF in this ticket is PM of WME), I think your case can be easily justified for a free tier: https://dumps.wikimedia.org/other/enterprise_html/

@Ladsgroup The MWoffliner scraper has already been quite optimised over years. I have no obvious improvement in mind for the moment but we will consider any concrete new proposal.

Would that https://github.com/openzim/mwoffliner/issues/1664 fix the issue, so far we are really not familiar with this new API end-point?

On my end, this looks better now. Thx!

The reported bug seems indeed to have "vanished". Thank you for the good work.

If you can delete the whole thread https://lists.wikimedia.org/hyperkitty/list/wikifr-l@lists.wikimedia.org/thread/J2FU23D4C5ERWIK2LWBYUBTYK3O6KP6Y/, then problem would be solved. It's a pity we can not hide it. I don't like that we manipulate the archive.