Copying https://phabricator.wikimedia.org/T409450 content here for reference

@Wangombe yes, as mentioned in my previous comment ... I've just added this qqq documentation to main branch, should be all good now.

@abi_ yes, the bot should have sufficient access from org level, at least we do have exact some permissions than other repos which are working well.

Thank you for the explanation, this is way clearer. I was confused believing we've made something in recent days which caused us to go over-quota.

@dcaro can you develop a bit why we suddenly get this message, and where we can find our current object storage usage and total entitled storage? I struggle to find the info in Horizon console.

Looking at the current numbers, things got worse: we are now seeing nearly 10,000 requests per minute (shared between two versions of MWOffliner). See https://w.wiki/FCsA.

@Audiodude do you have any idea what this object storage is used for? Looks like DB backups from Oct 1 2024, but these are not supposed to be there and they never got updated. I'm a bit confused.

I confirm it is back online, thank you very much ; let's hope it will not crash again.

@Andrew this is probably for you ^^

Issue is fixed ... no clue what fixed it.

Looks like reason for doing so is https://phabricator.wikimedia.org/T285951 but I'm still not convinced this was done the right way. Offending code is in normalizeFragment of includes/titles/Title.php if I'm not mistaken.

I confirm this is still a bug and Kiwix would still love to see this fixed.

To me this is solved, at least Maleo and Craveggia of nap.wikipedia.org are now nicely stored in our ZIMs. Reducing priority for the time being.

Closing, upstream issue is solved, we do not use this backend anymore.

This issue is "only" about editing metadata found at https://nrm.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=general, not about moving the wiki in terms of domain name. The lang should be nrf and fallbacks should contain nrm for the time being

Reducing priority for Kiwix, we do not really use this backend anymore and action=parse result is OK for https://el.m.wiktionary.org/wiki/%CE%B5%CF%80%CE%B9%CF%86%CF%85%CE%BB%CE%B1%CE%BA%CE%AE

Reducing priority for Kiwix, we do not really use this backend anymore

Shall we close this issue? Time has passed and amount of traffic seems to be sustainable for now given it has been quite a long time now.

What is kinda weird is that volume is still available from inside the VM

Still quite high on our wishlist from Kiwix perspective, even if we have a workaround (not yet implemented, coming soon, see https://github.com/openzim/mwoffliner/issues/2212 for those interested) and will have to maintain this workaround for years (until decent share of mediawikis have been updated to the version supporting the "upgraded" API).

Same symptom (if not same issue) with https://fi.wikipedia.org/wiki/Pascalin_kolmio?useparsoid=1

Forgot to mention that it impacts Kiwix and that issue on our side is https://github.com/openzim/mwoffliner/issues/2256

I mostly wanted to check that you are aware of the problem, and that no solution is feasible. Closing this a duplicate of T387680 then, sorry for not finding this issue beforehand.

FYI, we (Kiwix) are currently considering moving away from mobile-html endpoint to rely solely on ActionParse. Not saying this issue should not be fixed, but it is definitely less high prio on our end at the moment.

Note that all "other issues" mentioned above + the navbox issue is "old", like some were already there with Wikimedia Desktop, and some where even already a problem 1 year ago when still using the former / deprecated mobile endpoint.

OK, so I've went a bit further into implementing this "ActionParse" approach.

Kiwix mwoffliner does not use this API anymore.

We are also still discussing with Wikimedia Enterprise, but next steps are still to confirm feasibility (Wikimedia Enterprise is working on product discovery to confirm this)

For the record, we are in the process of implementing @cscott suggestion of one single call to action=parse to retrieve both Parsoid HTML and modules

Kiwix has switched to REST API since few weeks now. We do not yet ran the big recipes due to https://phabricator.wikimedia.org/T379017 and other issues, but we've done mostly all the big wiktionaries for instance.

It has been fixed ... looks like a temporary problem ...

Let's close this then, we do not have a repro

Yes, these are intermittent issues we probably need to live with unless we experience them continuously on a given page and hence have a clear repro for team to resolve.

Yes, please close it (looks like I can't do that)

@Ladsgroup @TheDJ thanks for the ping on Kiwix, much appreciated.

I do not find exact example again, of course.

OK, thanks for confirming this.

Digging a bit deeper, ActionParse gives me only the HTML head and the parsed body.

Thanks a lot for all this insight! I begin to get a better landscape.

Hi there, I'm Benoît, new dev lead on mwoffliner at Kiwix.

This would be something worth capturing in detail (i.e. with a curl -v and a timestamp for each of those). Getting a different response for the same url, from two different locations, at the very same time could be an indication of a split-brain in storage (presumably each request is hitting a different DC). We've never had that, but of course anything is possible.

Thank you for the investigation and the confirmation this is a caching issue!

My bad, you're right about no-cache, sorry about that

Cache-Control: no-cache is a response header, allowing the server to indicate the client he should not cache the response. As a client, setting this header in a request means nothing. In general, there is no way for a client to indicate to a cache server that he doesn't want a cache value.

I think it is more correct to say that at some (very undetermined) point they start working again.

What I've observed:

• what is "broken" (i.e. returning empty content) from one machine can work on another machine ; typically at a given moment, a curl from a machine running into WM Cloud might given a different result than same curl from a machine on the "public Internet"
• on a "big" run like ca.wikipedia.org (these are a bit old, I took too long to report them here... sorry), all ran from same machine on the "public Internet":
  • on first run, from 2024-12-28T08:02:15.591Z to 2024-12-30T09:04:17.444Z, I had 4 articles returning empty content: Lipopterichthys_carrioni, Joan_Mosc, Maylandia, Rodolphe_William_Seeldrayers
  • on second run, from 2024-12-31T08:00:30.806Z to 2024-12-31T08:00:35.783Z, I had 2 articles returning empty content: Rodolphe_William_Seeldrayers and Maylandia
  • on third run, from 2024-12-31T08:01:05.289Z to 2025-01-01T23:47:35.926Z, I had 2 articles returning empty content: Rodolphe_William_Seeldrayers and Maylandia
  • on fourth run, from 2025-01-02T21:13:07.674Z to 2025-01-04T09:14:26.609Z, I had 3 articles returning empty content: Rodolphe_William_Seeldrayers, Maylandia and Xavier_de_Langlais

Our server is down, we are restoring.

Hello,
We are done with the tests, please restore the database quota back to its original values in mwoffliner project (volumes back to 75, ram back to 4096)
Thank you.

I confirm this work as intended, we've just received our first PR. Thanks a lot, from my side we can close this task.

Just for the record, we are a bit late on this and we've identified more tasks to perform, so will need the increased quota for a bit longer, sorry about that. Probably 1 or 2 more weeks. I hope it is not a concern.

I don't know what "claiming" a task actually mean, feel free to claim this if it is important to you, I honestly absolutely don't care ^^

This was due to me making a typo in fstab, which is now fixed. (I have access to a raw console which is annoying but which can bypass ssh failure.) Sorry for the commotion!

@Audiodude do you wanna do the same on WP1 instance mwcurator? I just checked and it has the same problem in /etc/fstab of using /dev/sdb1 instead of UUID

I rebooted all four instances (I modified mwoffliner4 myself) and everything is ok on mwoffliner2, mwoffliner3 and mwoffliner4.

We do have a header which indicate that /etc/fstab is managed by Puppet:

@Rgaudin probably has higher chances to remember, since I wasn't there at that time.

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

Up, anyone could at least triage this issue, please?