Page MenuHomePhabricator
Create Task
Maniphest T421581

Abstract Wikipedia is not compatible with new API rate limits
Closed, ResolvedPublicBUG REPORT
Actions

Description

Steps to replicate the issue (include links if applicable):

  1. Log out (or open a new private tab)
  2. Visit https://abstract.wikipedia.org/wiki/Q188815
  3. Let the page finish loading
  4. Repeat steps 2 and 3 about 7-10 times.

What happens?:

"Unable to render this fragment due to an unknown error." and 429 error in console

What should have happened instead?:

No 429 error

It seems AbWiki pages make one API request per statement. That page has over 100 statements. And it's really not all that big. What happens when a big page like https://en.wikipedia.org/wiki/United_States is imported?

And of course, this doesn't just break AbWiki; the user is now locked out of all API features on all wikis. Including, for example, DiscussionTools, when they go to ask for help.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
AbstractPreview: Use exponential backoff when retrying rendersmediawiki/extensions/WikiLambdamaster+20 -11
rest gateway: introduce policy for Abstract Wikipediaoperations/deployment-chartsmaster+138 -18
rest gateway: introduce policy for abstractwiki/wikifunctionsoperations/deployment-chartsmaster+172 -14
Extend queue processing times for abstract fragmentsmediawiki/extensions/WikiLambdawmf/1.46.0-wmf.22+9 -5
Extend queue processing times for abstract fragmentsmediawiki/extensions/WikiLambdamaster+9 -5
Customize query in gerrit

Related Objects
Search...

Event Timeline

suffusion_of_yellow created this task.Sat, Mar 28, 7:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSat, Mar 28, 7:05 PM
A_smart_kitten added projects: MediaWiki-Platform-Team, Abstract Wikipedia team.Sat, Mar 28, 9:30 PM
A_smart_kitten subscribed.

(as I believe that MWP may be the responsible team to tag for tasks about these new rate-limits, and as the rule to auto-tag Abstract Wikipedia team on these sorts of tasks seems like it may not currently be working)

suffusion_of_yellow updated the task description. (Show Details)Sat, Mar 28, 9:33 PM
daniel added a parent task: T412585: Epic: Enforce API rate limits (WE5.1.3c).Mon, Mar 30, 8:03 AM
matmarex moved this task from Inbox, needs triage to Q3 Kanban Board on the MediaWiki-Platform-Team board.Mon, Mar 30, 2:50 PM
matmarex edited projects, added MediaWiki-Platform-Team (Q3 Kanban Board); removed MediaWiki-Platform-Team.
matmarex moved this task from Essential Work to OKR Work on the MediaWiki-Platform-Team (Q3 Kanban Board) board.
daniel triaged this task as High priority.Tue, Mar 31, 8:25 AM
daniel edited projects, added MediaWiki-Platform-Team; removed MediaWiki-Platform-Team (Q3 Kanban Board).
gerritbot added a comment.Tue, Mar 31, 8:26 AM

Change #1265333 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] rest gateway: introduce policy for abstractwiki/wikifunctions

https://gerrit.wikimedia.org/r/1265333

gerritbot added a project: Patch-For-Review.Tue, Mar 31, 8:26 AM
gengh subscribed.Tue, Mar 31, 11:24 AM
gerritbot added a comment.Tue, Mar 31, 8:50 PM

Change #1265568 had a related patch set uploaded (by Genoveva Galarza; author: Genoveva Galarza):

[mediawiki/extensions/WikiLambda@master] Extend queue processing times for abstract fragments

https://gerrit.wikimedia.org/r/1265568

gerritbot added a comment.Wed, Apr 1, 10:52 AM

Change #1265568 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Extend queue processing times for abstract fragments

https://gerrit.wikimedia.org/r/1265568

ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.23; 2026-04-07).Wed, Apr 1, 11:00 AM
gerritbot added a comment.Wed, Apr 1, 11:42 AM

Change #1266219 had a related patch set uploaded (by Jforrester; author: Genoveva Galarza):

[mediawiki/extensions/WikiLambda@wmf/1.46.0-wmf.22] Extend queue processing times for abstract fragments

https://gerrit.wikimedia.org/r/1266219

gerritbot added a comment.Wed, Apr 1, 2:07 PM

Change #1266219 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@wmf/1.46.0-wmf.22] Extend queue processing times for abstract fragments

https://gerrit.wikimedia.org/r/1266219

Stashbot added a comment.Wed, Apr 1, 2:08 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T14:08:28Z] <jforrester@deploy1003> Started scap sync-world: Backport for [[gerrit:1266190|MemcachedWrapper: Hash key when longer than 250 characters]], [[gerrit:1266219|Extend queue processing times for abstract fragments (T421581)]]

Stashbot added a comment.Wed, Apr 1, 2:10 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T14:10:26Z] <jforrester@deploy1003> jforrester: Backport for [[gerrit:1266190|MemcachedWrapper: Hash key when longer than 250 characters]], [[gerrit:1266219|Extend queue processing times for abstract fragments (T421581)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Wed, Apr 1, 2:16 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-01T14:16:43Z] <jforrester@deploy1003> Finished scap sync-world: Backport for [[gerrit:1266190|MemcachedWrapper: Hash key when longer than 250 characters]], [[gerrit:1266219|Extend queue processing times for abstract fragments (T421581)]] (duration: 08m 14s)

ori subscribed.Wed, Apr 1, 2:29 PM
ReleaseTaggerBot edited projects, added MW-1.46-notes (1.46.0-wmf.22; 2026-03-31); removed MW-1.46-notes (1.46.0-wmf.23; 2026-04-07).Wed, Apr 1, 3:00 PM
Jdforrester-WMF moved this task from To Triage to Bugs to investigate on the Abstract Wikipedia team board.Wed, Apr 1, 5:03 PM
Jdforrester-WMF added 1 subscriber(s): DSantamaria.
gerritbot added a comment.Thu, Apr 2, 10:41 AM

Change #1265333 merged by jenkins-bot:

[operations/deployment-charts@master] rest gateway: introduce policy for abstractwiki/wikifunctions

https://gerrit.wikimedia.org/r/1265333

Maintenance_bot removed a project: Patch-For-Review.Thu, Apr 2, 11:32 AM
gerritbot added a comment.Thu, Apr 2, 4:45 PM

Change #1267122 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] rest gateway: introduce policy for abstractwiki/wikifunctions"

https://gerrit.wikimedia.org/r/1267122

gerritbot added a project: Patch-For-Review.Thu, Apr 2, 4:46 PM
daniel added a comment.Fri, Apr 3, 10:48 AM

ftr, the patch didn't go out, we have to sort out the gateway routing for wikifunctions first.

gerritbot added a comment.Wed, Apr 8, 11:49 PM

Change #1269079 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] AbstractPreview: Use exponential backoff when retrying renders

https://gerrit.wikimedia.org/r/1269079

gerritbot added a comment.Mon, Apr 13, 10:06 AM

Change #1267122 merged by jenkins-bot:

[operations/deployment-charts@master] rest gateway: introduce policy for Abstract Wikipedia

https://gerrit.wikimedia.org/r/1267122

gerritbot added a comment.Mon, Apr 13, 1:52 PM

Change #1269079 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] AbstractPreview: Use exponential backoff when retrying renders

https://gerrit.wikimedia.org/r/1269079

ReleaseTaggerBot edited projects, added MW-1.46-notes (1.46.0-wmf.24; 2026-04-14); removed MW-1.46-notes (1.46.0-wmf.22; 2026-03-31).Mon, Apr 13, 2:00 PM
daniel closed this task as Resolved.Mon, Apr 13, 2:05 PM
daniel claimed this task.

should be fixed now

Maintenance_bot removed a project: Patch-For-Review.Mon, Apr 13, 2:31 PM
Escargot_rouge reopened this task as Open.Thu, Apr 16, 9:22 AM
Escargot_rouge subscribed.

I cannot get https://abstract.wikipedia.org/wiki/Q90 to properly load. I always see "Reached max retries. Try again later."

image_2026-04-16_112124534.png (459×980 px, 41 KB)

It used to work fine for me before rate limiting was enforced.

Redmin closed this task as Resolved.Thu, Apr 16, 9:25 AM
Redmin subscribed.
In T421581#11828648, @Escargot_rouge wrote:

I cannot get https://abstract.wikipedia.org/wiki/Q90 to properly load. I always see "Reached max retries. Try again later."

image_2026-04-16_112124534.png (459×980 px, 41 KB)

It used to work fine for me before rate limiting was enforced.

It works after clicking 'Retry'; see T420630.

daniel added a subscriber: Jdforrester-WMF.EditedThu, Apr 16, 10:41 AM
In T421581#11828648, @Escargot_rouge wrote:

I cannot get https://abstract.wikipedia.org/wiki/Q90 to properly load. I always see "Reached max retries. Try again later."

@Jdforrester-WMF I suppose that's because you reduced the number of retries?

It used to work fine for me before rate limiting was enforced.

I think that's a coincidence - the retries here are not because of rate limits, but relate to wikifunctions evaluation in the background.

Jdforrester-WMF added a comment.Thu, Apr 16, 1:50 PM
In T421581#11828879, @daniel wrote:
In T421581#11828648, @Escargot_rouge wrote:

I cannot get https://abstract.wikipedia.org/wiki/Q90 to properly load. I always see "Reached max retries. Try again later."

@Jdforrester-WMF I suppose that's because you reduced the number of retries?

Not quite, the initial failure is because of T422299: Wikifunctions client wikis' cache config doesn't know about wf-wan, so embedded Functions’ output is never available which means fragments aren't getting cached (so every page view is a render), and the service can't run them fast enough.

The reduced, slower run rate means we are now much less likely to hit the rate limits (alongside your changes to isolate our requests), and once the content storage is fixed should restore functionality.

Sorry for the disruption!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits