Page MenuHomePhabricator

Fawiki article cannot be edited: "Service Temporarily Unavailable" timeout upon saving at API execution limit (200 seconds)
Closed, ResolvedPublic

Description

The article فهرست فیلم‌های دنیای سینمایی مارول at the Persian Wikipedia cannot be edited because of server error.

Steps to Reproduce:

  1. Go to https://fa.wikipedia.org/wiki/فهرست_فیلم‌های_دنیای_سینمایی_مارول
  2. Make a dummy edit.
  3. Try to save the page.

Actual Results:

Service Temporarily Unavailable
Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes.

This has been the case for at least several days.

Expected Results:

The page should have been saved like any other article.

What is wrong with this particular article?

Event Timeline

4nn1l2 created this task.Jan 3 2019, 7:51 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 3 2019, 7:51 AM
4nn1l2 updated the task description. (Show Details)Jan 3 2019, 7:51 AM
Dalba added a subscriber: Dalba.Jan 3 2019, 8:01 AM

Cannot reproduce with the given link, using the source editor. No error shown.
Which editor is this about? What is the full, complete error message (except for IP addresses)?

4nn1l2 added a comment.Jan 3 2019, 8:56 PM

Which editor is this about? What is the full, complete error message (except for IP addresses)?

I use the source editor, and this is the screenshot of the entire error:

I also tried the visual editor, but I couldn't save my dummy edit. Here is the screenshot of the error shown:

I am not sure what you mean by "the full, complete error message (except for IP addresses)". Could you please explain?

Which errors are shown in the "network" tab of your browser's developer tools? (If you have never used developer tools, see https://www.mediawiki.org/wiki/Help:Locating_broken_scripts for some information, though that page is about scripts instead.)

Do you know anyone else who sees the same problem?

Dalba added a comment.EditedJan 4 2019, 12:38 PM

This is a server-side error. I'm not using the visual editor, just the plain old editor, and I can reproduce the issue. I try to edit the page by adding a single space somewhere in the article and then pressing the publish button. After a while I get:

Service Temporarily Unavailable
Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes.

I have even tried to empty the page (blanking completely) with the same result.

There is no error message in browser's console (just the usual warnings that are shown on every page). The server is returning a 200 response.

Aklapper renamed this task from Fawiki article cannot be edited because of server error to Fawiki article cannot be edited: "Service Temporarily Unavailable".Jan 4 2019, 12:47 PM

Reminded me of T85536 but that got fixed in T73487. (Adding HHVM might turn out wrong.)

Huji added a subscriber: Huji.Jan 8 2019, 2:12 AM

Upon page save, it takes 2-3 minutes before servers give up and return an error message. If I were to make an uneducated guess, I would also think of an intermediary such as HHVM.

Joe added a subscriber: Joe.Jan 8 2019, 6:31 AM

FTR, I tried to save that page with php7 and failed as well. So I doubt this has to do with HHVM itself.

What we see is a timeout upon saving at the limit of execution for API, which is 200 seconds.

I'll remove the HHVM tag from this bug.

Joe removed a project: HHVM.Jan 8 2019, 6:31 AM
Aklapper renamed this task from Fawiki article cannot be edited: "Service Temporarily Unavailable" to Fawiki article cannot be edited: "Service Temporarily Unavailable" timeout upon saving at API execution limit (200 seconds).Jan 8 2019, 7:01 AM

It's been 3 months and it hasn't been fixed yet. :( It's featured list on FaWiki and I want to update it. :(

@Navidff: I followed the steps in the task description and have no problems:

  1. Go to https://fa.wikipedia.org/wiki/فهرست_فیلم‌های_دنیای_سینمایی_مارول?action=edit
  2. Click انتشار تغییرات button.
  3. My null edit gets 'saved' and no error message is shown. Cannot reproduce.

(Side note: مدیاویکی:Common.js is broken: TypeError: mw.Uri is not a constructor. See Help:Locating_broken_scripts. Please fix.)

Huji added a comment.Mar 18 2019, 11:50 PM

@Aklapper first of all, I don't see any indication on that page's history that you successfully saved an edit. Please try with a dummy edit instead (with null edits, the issue doesn't reproduce itself).

Second, I see that @Dalba was successful to save some changes to that page today. So this issue is not happening all the time; I tried it right now (by adding a space at the end of the first line of wikicode and trying to save the page) and I did get an error message after a lengthy delay. My edit was not saved.

Last: I cannot find a TypeError message in the browser console. Could it be that you are using a particular Gadget? Can you try with debug=true and point us to the script that causes the TypeError please?

@Aklapper first of all, I don't see any indication on that page's history that you successfully saved an edit. Please try with a dummy edit instead (with null edits, the issue doesn't reproduce itself).

Indeed, my fault for not reading closely enough. Sorry!

Last: I cannot find a TypeError message in the browser console. Could it be that you are using a particular Gadget? Can you try with debug=true and point us to the script that causes the TypeError please?

Garr, sorry again (was doing too many things at the same time)! Default gadget settings; it's the line this.href = new mw.Uri(this.href).extend({ in https://fa.wikipedia.org/w/index.php?title=مدیاویکی:Common.js/edit.js

Dalba added a comment.EditedMar 19 2019, 2:47 AM

I can understand why the page is taking so long to save (it's almost 300KB), but why is it timing out when trying to blank the page? Is MW trying to generate a diff or something like that that requires processing the previous content of the page, too?

Anyway, I think I've finally found a way to edit the page: Remove moderate portions of the article. By "moderate" I mean that those portions should be neither too small nor too large, otherwise MW won't be able to process the edit. I reverted my edits, so the issue is still there. My suggestion is to split the article into smaller ones, at least until the underlying issue is resolved.

(I couldn't share my observations here at the time because phabricator was down.)

I reduced it to under 250 kb and now edits can be saved, but it still takes so much time and you end up with that wikimedia error again. But then if you go back to the page you will see that the edit had been saved.
It's good for now but it cannot stay this way as there is so much that I have to add to this article, because I haven't been able to edit it for while.

Huji added a subscriber: Daimona.Mar 21 2019, 10:16 PM

@Dalba my hunch is that when you blank the page, AbuseFilter will try to work on a massive diff, and that slows thing down to the point of crashing. Smaller changes produce smaller diffs, and let the page go through.

I have no concrete ideas on how to check the AbuseFilter theory though. I know that we recently worked on enabling AbuseFilter profiling for many (all?) wikis; @Daimona do you know what the status of that is? Is profiling enabled on fawiki, and if so, is it possible to take a look at the data for this particular page to see if things are particularly slow?

Huji added a comment.Mar 21 2019, 10:23 PM

Last: I cannot find a TypeError message in the browser console. Could it be that you are using a particular Gadget? Can you try with debug=true and point us to the script that causes the TypeError please?

Garr, sorry again (was doing too many things at the same time)! Default gadget settings; it's the line this.href = new mw.Uri(this.href).extend({ in https://fa.wikipedia.org/w/index.php?title=مدیاویکی:Common.js/edit.js

@Aklapper I think this should have fixed the TypeError issue. Can you kindly verify?

@Huji Dang! I checked on logstash (from mobile phone, which is painful) and heck yes! AbuseFilter 101 is constantly taking ~150 seconds to execute on that page!
You are right, profiling is enabled on every wiki (and I already sent patches to remove profiling globals altogether). However, the avg execution time for that filter doesn't show up as very high due to the amount of edits (that's why there are patches to add the maximum recorded execution time). From a quick look I can see a couple of things to be optimized in that filter, please let me know if you need any help with it.

Huji closed this task as Resolved.Mar 22 2019, 2:16 AM
Huji claimed this task.

Thanks @Daimona! I am glad my hunch was right.

I disabled that filter, and upon doing so we can now edit that page without any issues! I am going to mark this task as resolved. I am eager to know what you think can be changed in that filter to improve its performance. (The aim of that filter is identify situations where a user removes a named <ref> tag (like <ref name=somename>...</ref>) but leaves behind future uses of it (like <ref name=somename/>).

@Huji I wasn't totally right (it happens when you think too much right before going to sleep ;-)). I thought it would have been possible to use get_matches, but actually it would stop at the first occurrence. However, the regex is still bad (see https://regex101.com/r/ARn7U4/1) and it ends up with a catastrophic backtracking. Unfortunately I don't have a fast replacement at hand.

Thank you everyone for participating in this topic.

@Huji Actually, another idea came to mind. You may use new_html and check if there's a span with a cite error (the one you get when the ref identifier is not defined). So to have, for instance,

new_html contains '<span class="error mw-ext-cite-error"'

However, this way you cannot check the old text (old_html is disabled due to poor performance). new_html is reported on mw.org as a slow variable, but this is only partly true: it's true that it takes a long time to compute it while checking the filter (especially given the size of the page), and the filter would show up as very slow. However, the page would be parsed anyway upon saving, so it's just matter of moving such parsing inside the "AbuseFilter phase" instead of doing it later.