I think I fixed the URL for Special:OAuth, someone please try it in the wild

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Different bug though. Fixed now.

Thanks, I added descriptions and toolhub now picked it up.

So it does not look like the crawler used the JSON I provided. For example, "drawshield" is clearly in https://magnustools.toolforge.org/toolinfo_buggregator.json but it does not show up in Toolhub search.

Ah it looks like the crawler runs once every hour: https://toolhub.wikimedia.org/crawler-history

Adding my own list of my tools, matched to the toolhub name where I could. Note that some of these are deactivated/broken (see notes), some are JavaScript on-wiki so not sure if that counts. The "open/closed' tickets are biased towards "open" while I go through them over time, but could become a quality score indicator down the line.

I think I had a misconfiguration in the cronfile, this is fixed now. Please re-open/ping me if this still happens.

First time I see this issue here, will investigate.

This was broken by the "toolforge subdomain migration".

Turns out that single catalog takes ~21GB:

The collected data (from the BaGLAMa2 tool) is quite valuable for GLAM, including the historic ones, which is why I try to keep all of it.

Fixed now.

In T282158#7100475, @Sannita wrote:

I'll just put this bug here, because I'm not 100% sure I can attend the session: https://github.com/magnusmanske/listeria_rs/issues/67

This is a rather urgent bug we noticed a couple of months ago, and might at this point be related to the 90/10 issue Andrew was discussing in a previous comment.

A few notes:

• happy to help with any new development, as time allows; prioritized list preferred
• issue tracker at https://github.com/magnusmanske/listeria_rs but it's a bit messy, partially because no one reads the existing issues...
• the "killed" errors come from Toolforge killing the processes even though there is enough memory, but they allocate memory in large blocks which is suspicious, apparently. I have an idea how to get around that, but will require a bit of time
• the "central location" is the Commons Data: namespace. This works ~80-90%, and requires a final push, both in Rust and Lua. See http://magnusmanske.de/wordpress/?p=650

Turns out flickr retired the secure.flickr.com domain without forwarding. Removed the "secure" but still using https, should work again.

Also, doesn't seem to work for "has a birth date"?
https://www.wikidata.org/w/index.php?search=haswbstatement%3AP21%3DQ6581072++haswbstatement%3AP569&title=Special:Search&profile=advanced&fulltext=1&advancedSearch-current=%7B%7D&ns0=1&ns120=1

In T179879#6859731, @So9q wrote:

Both of these can be done using CirrusSearch API and can't they?

Restarted wikidata-todo.

It does work, however, since there is currently not a single page on euwiki where the latest revision has a wp10 score, the results will be empty.

Thanks, I think I fixed it now.

Should restart now

On it

I would also like to point out https://tabernacle.toolforge.org/ which ATM is for editing existing items, but I could easily create a new blank item to edit.

Just saw this one. I think fixing wikibase to add to watchlist is the way to go. I could update the OAuth permissions but that would break everyone's server-based edits as they would use an old OAuth, correct?

Should be fixed now

I'm on it.

Worked with 10K batches. I still think this rapid disconnect in the middle of a query is a bug.

Trying 1M chunks, first two worked, but then:

mysqlimport: Error: 2013, Lost connection to MySQL server during query, when using table: osopenuprn_202006

I'll try even smaller ones, but it's getting a bit ridiculous. Why disconnect the server in the middle of a query?

I have switched the configuration to use the petscan db connections only.

Just to add, if some queries take too long I can likely just change a parameter to fix that. Someone with access to the stats, let me know.

The reason I use that many connections is precisely to avoid long-running ones, as I had in a previous version; they would time out or lose database connection, so I re-wrote the code to use more but shorter queries.

In T243848#6251382, @Aklapper wrote:

@Magnus: Does that mean that https://phabricator.wikimedia.org/source/tool-quickstatements/browse/master/ will get out of sync and should be archived?

Yes.

I have "forked" it into github. Using diffusion was a mistake in the first place. Now at https://github.com/magnusmanske/quickstatements

Tried again with ssh://vcs@git-ssh.wikimedia.org/source/tool-quickstatements.git, no joy:

Permission denied (publickey,keyboard-interactive).
fatal: Could not read from remote repository.

tools.quickstatements@tools-sgebastion-07:~$ git remote remove origin
tools.quickstatements@tools-sgebastion-07:~$ git remote add origin ssh://phabricator.wikimedia.org/diffusion/2010/tool-quickstatements.git/
tools.quickstatements@tools-sgebastion-07:~$ git push fatal: The current branch master has no upstream branch.
To push the current branch and set the remote as upstream, use

FWIW, I believe all my (active) tools now use the new canonical URL.

As I wrote five years ago, I added a limit to upload only two files concurrently.
No further comment has been made since, so I consider this issue to be solved.

Thanks, that seems perfect!

In T254640#6202698, @bd808 wrote:

This config is part of the default configuration generated by webservice. That means we need to fix it upstream of the usage by this particular tool.

OK thanks

As a horrible short term fix for reasonator, I did:

$ ln -s $HOME/public_html/reasonator_types.js $HOME/public_html/_types.js

In T254640#6201620, @LucasWerkmeister wrote:

I don’t understand why this is an UBN when as far as I understand we know a workaround (restart without --canonical)?

In T194953#6185511, @bd808 wrote:

Is there a movement towards Rust in any significant segment of the Wikimedia movement? Are there clear benefits for Toolforge tools to be written and maintained in Rust?

I think this is a blocker to moving all tools to the canonical URL. T234617 ?

I cleared the error log on reasonator and switched to the previous URL schema, then back to the canonical one. No server errors in the log (some unrelated PhP ones, now cleared again)

@LucasWerkmeister thanks that worked, for now

Same with https://geohack.toolforge.org/geohack.php . Geohack is kinda central for Wikipedia so please help!

Thanks @bd808 !

Other files load fine. Haven't checked all though

URL is https://reasonator.toolforge.org/reasonator_types.js

Restarted it now.
I am also working on a watcher script to restart it automatically when it goes down.

It's my dev server, should work for any query size. Feel free to use when the main site is down.

@Aklapper actually the repo is https://github.com/magnusmanske/petscan_rs (the bitbucket one is the old C++ version).

There are some PetScan outages, reason unknown so far, but outside those, the above queries all work fine.

Still happening.

Maybe this helps:

Yes, still seeing it. Runs for ~20sec so probably not a cache. Changed the JSONP callback(s) and still same error, so definitely no content-level cache.

It appears that the third-party bot framework quietly swallows this and just throws an exception with the "info" field :-(

In the error I mean

Hmm, I think what tripped my code is that there was no code:'maxlag' in the response?

Seems to clear up now

But it's all kinds of wikis that show up. Something with the dispatcher?

QuickStatements edits as the user who submitted the job, using OAuth, as is the WMF-preferred method for tools.

Should now be fixed with https://bitbucket.org/magnusmanske/listeria/commits/5b53c94ffb10

Getting this error for mix-n-match tool.

@bd808 Thanks, now running quickstatements as two repilcas

Update: Setting server.max-request-size = 1000000 in $HOME/.lighttpd.conf and restarting the webservice did not help.