Page MenuHomePhabricator

High numbers of HTTP 429 errors
Closed, ResolvedPublic

Description

Several users reported receiving HTTP 429 (too many requests) when trying to use Chinese wikipedia

Screenshot_20200910-131641_Wikipedia.jpg (1×1 px, 156 KB)

This may be impacting other wikis, too.

Event Timeline

Dbrant renamed this task from Error with zh wiki in Android app to High numbers of HTTP 429 errors.Sep 11 2020, 5:02 PM
Dbrant triaged this task as High priority.
Dbrant updated the task description. (Show Details)

I'm seeing this as we speak, when trying to read random articles on zhwiki.

I'm also seeing intermittent cirrussearch-too-busy-error when searching (on all wikis).

Thanks for filing this @LGoto

I have replied to the zh.wiki users who reported this issue.

I'm also seeing intermittent cirrussearch-too-busy-error when searching (on all wikis).

They were a couple reports on the Wikimedia Discord about this affecting searches in the main namespace.

cooltey raised the priority of this task from High to Unbreak Now!.Sep 13 2020, 7:44 PM
cooltey added subscribers: Charlotte, JoeWalsh.

We have received a large number of reports from users that the application shows error code 429.

Do we have any way of fixing the issue (maybe remove the ratelimit?) cc @Pchelolo @JoeWalsh @Charlotte

Note: change the ticket to Unbreak Now!.

We have received a large number of reports from users that the application shows error code 429.

Do we have any way of fixing the issue (maybe remove the ratelimit?) cc @Pchelolo @JoeWalsh @Charlotte

Note: change the ticket to Unbreak Now!.

Removing the ratelimit would be a temporary fix, if it would fix the issue at all. Have we any idea what the root cause of this is, and why it only seems to be affecting zhwiki?

We have received a large number of reports from users that the application shows error code 429.

Do we have any way of fixing the issue (maybe remove the ratelimit?) cc @Pchelolo @JoeWalsh @Charlotte

Note: change the ticket to Unbreak Now!.

Removing the ratelimit would be a temporary fix, if it would fix the issue at all. Have we any idea what the root cause of this is, and why it only seems to be affecting zhwiki?

T262705#6454901
There’s a comment from @Pchelolo
“ratelimit was added to mobile-html with non-standard language variant due to an outage caused by on-the-fly transformation of the variants in parsoid.“

Change 627279 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] varnish: raise the zhwiki-restbase rate-limit by 5x

https://gerrit.wikimedia.org/r/627279

Change 627279 merged by Giuseppe Lavagetto:
[operations/puppet@production] varnish: raise the zhwiki-restbase rate-limit by 5x

https://gerrit.wikimedia.org/r/627279

I believe the search issue is separate from the zhwiki rate limiting.

Change 627291 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] varnish: remove zhwiki-restbase special case

https://gerrit.wikimedia.org/r/627291

Change 627291 merged by Ema:
[operations/puppet@production] varnish: remove zhwiki-restbase special case

https://gerrit.wikimedia.org/r/627291

ema lowered the priority of this task from Unbreak Now! to High.Sep 14 2020, 3:53 PM
ema subscribed.

Users should not get 429s anymore, lowering priority to high while waiting for confirmation. For the deeper issue I'll have a conversation with @Pchelolo about related restbase topics later this week.

eprodromou subscribed.

OK, we'll take a look at getting this figured out. @Pchelolo , let's consult on what's needed.

BBlack subscribed.

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

Dbrant claimed this task.

I don't believe we've seen this recently, so closing.