@Multichill agreed. The replag solution works AFAICT, why not duplicate that.

I have given up on applying the patch, and just entered it manually:
https://phabricator.wikimedia.org/R2010:bd0fca9e0f531d27e2c2f7ec2de2599720f96235

@Addshore It depends on what the actual issue with creating/editing is. If it's database-related (most likely), then having a few sleeping processes for load-dependent server-side delays might be a viable solution.

So,

• either WMF expose the "overload status" so we can throttle on demand,
• or WMF delay the API response accordingly, so we don't have to do anything,
• or WMF scale to demand :-)

OK, let me summarize my situation, as of today:

• Essentially, all my Wikidata edits (both bots and JavaScript on Toolforge) run through the same OAuth PHP class I wrote
• Every single edit (including "create item") runs with a "maxlag=5" parameter.
• If the edit fails because of maxlag, it will wait 5 seconds, then try again, until it succeeds.
• This mechanism has been in place for months.
• As of today, after every edit (including "create item"), it will check the current lag on Wikidata.
• If the current lag is >1 sec, it will sleep three times the lag (in int seconds; so, if lag is 1.5 sec, it will sleep int(3*1.5)=4 seconds) before returning.
• If that check fails for some reason, it will sleep a pre-defined time for each edit type (create item=2sec, edit=1 sec)

That is a total of three different rate limitations (two dynamic, one hardcoded fallback). Personally, I consider that beyond due diligence, verging on paranoia. Therefore, I consider this matter resolved, as far as my tools are concerned.

All done.

Ah, you found it! Fixing now...

I can't see an easy way to change these over to the WMF tile server, without upgrading to OpenLayers 4.6, which is vastly different from the 2.x version I am using. This would require a partial/complete rewrite of these tools. I don't have the time for that, so I have deactivated those tools.

For one tool, per user, per tab? Sure. I'd just set both item creation and edit to 1 second delay.

I have added a 5 second delay after item creations. I also added a mechanism to have a delay after any other edit type.

Thanks to @Multichill for pointing out this discussion to me...

Restarted. Toolforge keeps killing my tools occasionally...

I, for one, have no idea what s51184 is. It doesn't show in the Toolforge "local" database list:

echo 'show databases;' | sql local | grep 51184

If there is an easy way to find the name of the tool to that, please let me know.

How do I apply this to the git repo?

huh, just saw this...

Hm, it seems to have stopped for now. Can't reproduce it anymore. I'll wait to tomorrow to close it though...

Sorry about tagging it Cloud-Services, I thought it might be a cloud config issue (stale DNS or somesuch).

Also, here are all files on Commons that were created in January and contain {{Original upload log}}. (Note: this query might run for a minute or so)

Of course, if WMF were to track somewhere which edit was doe using which OAuth consumer, that would give much better data...

I have altered CommonsHelper to add the #commonshelper tag to the upload comment, like so. This is active as of now.

The results page is configured almost entirely by the "calling wiki", via the respective version of the GeoTemplate. That includes which services are listed, in which order, etc. If a region is given in the geohack URL, a specific section of the template can be pushed to the top IIRC.

There's the access.log file, this one starting 13/Dec/2017, where we can see how geohack was called. I doubt there is any information on what links were clicked within geohack, so we don't know which services were used the most.

Thanks @Smalyshev Yes these queries are outliers, but there is a good reason to run them (in this case, find missing images for items), and just by Wikidata growth, the number of outliers (queries that timeout) will increase, to the point where they are no longer outliers. Technical improvements (improved software, better hardware) can buffer some of this, but not all. As you show, some queries can be re-formulated (though the reason why one works but not the other remains a mystery to most users), but again, not forever.

Semantics and drama aside, I think the point is that some useful queries are already creaking, or failing entirely.

How about a separate server just for "heavy" queries? For everyone? If that goes down occasionally, hey, so what?

This is from the "wikidata-terminator" tool (which you could probably have found out quicker than I).

@Niharika I have turned off notifications as per suggesestion

Yes, ListeriaBot (which edits Wikipedia) does not cause those (so far), so only Wikidata.

As this is my first patch accept in Differential, how do I merge this here? No "merge" button that I can see on this page. Probably not "commandeer", whatever that is?

Both bots use the Addwiki lib, not sure if that uses OAuth:
https://phabricator.wikimedia.org/project/profile/1490/

Several of my tools appear to be affected as well, example:

@jcrespo OK, but that only means the rows were missing since the last edit, presumably, on 5 August 2017‎.

I did ?action=purge on the item before I filed the bug.

"sql" is the recommended shell command to connect to the replicas on Toolforge, correct?

Addendum:

MariaDB [wikidatawiki_p]> select @@hostname;
+------------+
| @@hostname |
+------------+
| labsdb1003 |
+------------+
1 row in set (0.00 sec)

Please note that, while https://tools.wmflabs.org/admin/tools indeed lists all Toolforge "registered tools", many of them are containers for two or more "actual", distinct tools.

I will look into this once there is a realistic test site. QS2 is designed for multiple sites, see https://phabricator.wikimedia.org/source/tool-quickstatements/browse/master/public_html/sites.json

And now it's back. Something weird is going on. Is there a load record for that tools' webservice?

@chasemp The "missing animal audio" thing should be fixed now.

And it's stuck again...

It came back after the bastion reboot. Maybe slow bastion => slow filesystem => slow tool/webservice?

I have now set up phpunit version 4 in a single tool. Support for that version ended February 3, 2017. However, it is the latest version that can tolerate the ancient PHP 5.59 on Toolforge.

Thanks @Reedy . Is an OS upgrade planned any time soon?

Unless one has dozens of tools to install that one-at-a-time...

Update: I have added some more "re-run if failed" and "check connection/reconnect" code, and the GLAM data now seem to process more fluently. Unknown if that's because of the new code, or if upstream was fixed.

I will list some source below, just let me emphasize this is not limited to a specific query or type of query, programming language, tool, or webservice/shell. I even see this in PetScan, which is C++ and runs on its own VM, though it uses the DB replicas as well. Also, it does not seem to be reproducible in general, but tends to "cluster" time-wise, as in, there are some time periods where there is a lot of this, some where it's a rare occurrence. I have kept no log of those periods, just thought of it as the cost of doing business, especially after my initial bug report was declined.

• The example I gave (line 51)
• PetScan database (re-)connecting code (various queries would timeout, thus centralized query/connect code)

And yes, I know the disconnect happened before the *** line. In all the ~2sec it took me to type the command.

I am sure my client can handle ~800 rows, which is what I get when I use DISTINCT, as shown...

Trying to decompile all these into statements and/or checks:

OK, these are my old workaround for the removal of a wikidata DB table. (I did day the removal of the table will cause problems, but does anyone ever listen to me??!?)

The problem appears to have resolved itself. I have switched everything back to normal.

I'm not sure what causes the error, but the authentication itself seems to be successful. Can you confirm that?

Pull request merged.

I just started this manually, I do this once a month for the GLAM community. It should subside quickly now.

Everything works now after renaming/removing of .my.cnf

Daily list to work through:
https://tools.wmflabs.org/listeria/botstatus.php

It was written to do single edits, so it does log in before each one. I can try and change it if it's a problem?

@jcrespo But I am working as the respective "tool-user"!

Addendum: With some tools, I can connect to "local", but not create a new tool database:

@jcrespo But shouldn't the sql script automatically use the replica file? There's a "proper" user name in there, and 'sql dewiki' works fine.

Also for "flickr2commons" tool.

Applied by manual edit