In T323502#8419749, @taavi wrote:

In T323502#8416721, @Magnus wrote:

And while I'm at it, I would like to request that the maximum number of database connections be increased to 20. This is mostly for the tool user database, not so much for the DB replicas, in case that makes a difference.

Could you explain a bit why this is needed?

And while I'm at it, I would like to request that the maximum number of database connections be increased to 20. This is mostly for the tool user database, not so much for the DB replicas, in case that makes a difference.

--wait instead of --continuous seems to work?

I have other k8s jobs scheduled/running for this tool, they all work fine

The command is a thin bash wrapper around a Rust binary, compiled on toolforge. Binary starts just fine when run manually in shell.

Never mind, I just copied the binary to the tools directory and run it from there.

Moved all the jobs now except one, which depends on mysqldump, but that command is not in tf-bullseye-std. Is there any way to access that command from k8s?

Done (already using k8s but someone left the cronjobs in place for just-in-case restarting, this is removed now).

Done.

Done.

Done.

Done.

Done.

Done.

Done.

Done.

Done.

Done.

Done.

Done.

Done.

Done.

Done.

Update: I migrated most of the cronjobs (except two) to kubernetes. Best I can do for now, I'll have a look at the remaining ones again once I migrate all my other tools.

Done.

Hi, as you may know, I am a massive user of the toolforge environment. I am also an eager early adopter of new technologies. I will use this ticket (and not the other dozen or so I got for other tools) for some feedback, until it's specific to other tools.

I think I fixed the URL for Special:OAuth, someone please try it in the wild

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Fixed

Different bug though. Fixed now.

Thanks, I added descriptions and toolhub now picked it up.

So it does not look like the crawler used the JSON I provided. For example, "drawshield" is clearly in https://magnustools.toolforge.org/toolinfo_buggregator.json but it does not show up in Toolhub search.

Ah it looks like the crawler runs once every hour: https://toolhub.wikimedia.org/crawler-history

Adding my own list of my tools, matched to the toolhub name where I could. Note that some of these are deactivated/broken (see notes), some are JavaScript on-wiki so not sure if that counts. The "open/closed' tickets are biased towards "open" while I go through them over time, but could become a quality score indicator down the line.

I think I had a misconfiguration in the cronfile, this is fixed now. Please re-open/ping me if this still happens.

First time I see this issue here, will investigate.

This was broken by the "toolforge subdomain migration".

Turns out that single catalog takes ~21GB:

The collected data (from the BaGLAMa2 tool) is quite valuable for GLAM, including the historic ones, which is why I try to keep all of it.

Fixed now.

In T282158#7100475, @Sannita wrote:

I'll just put this bug here, because I'm not 100% sure I can attend the session: https://github.com/magnusmanske/listeria_rs/issues/67

This is a rather urgent bug we noticed a couple of months ago, and might at this point be related to the 90/10 issue Andrew was discussing in a previous comment.

A few notes:

• happy to help with any new development, as time allows; prioritized list preferred
• issue tracker at https://github.com/magnusmanske/listeria_rs but it's a bit messy, partially because no one reads the existing issues...
• the "killed" errors come from Toolforge killing the processes even though there is enough memory, but they allocate memory in large blocks which is suspicious, apparently. I have an idea how to get around that, but will require a bit of time
• the "central location" is the Commons Data: namespace. This works ~80-90%, and requires a final push, both in Rust and Lua. See http://magnusmanske.de/wordpress/?p=650

Turns out flickr retired the secure.flickr.com domain without forwarding. Removed the "secure" but still using https, should work again.

Also, doesn't seem to work for "has a birth date"?
https://www.wikidata.org/w/index.php?search=haswbstatement%3AP21%3DQ6581072++haswbstatement%3AP569&title=Special:Search&profile=advanced&fulltext=1&advancedSearch-current=%7B%7D&ns0=1&ns120=1

In T179879#6859731, @So9q wrote:

Both of these can be done using CirrusSearch API and can't they?

Restarted wikidata-todo.

It does work, however, since there is currently not a single page on euwiki where the latest revision has a wp10 score, the results will be empty.

Thanks, I think I fixed it now.

Should restart now

On it

I would also like to point out https://tabernacle.toolforge.org/ which ATM is for editing existing items, but I could easily create a new blank item to edit.

Just saw this one. I think fixing wikibase to add to watchlist is the way to go. I could update the OAuth permissions but that would break everyone's server-based edits as they would use an old OAuth, correct?

Should be fixed now

I'm on it.

Worked with 10K batches. I still think this rapid disconnect in the middle of a query is a bug.

Trying 1M chunks, first two worked, but then:

mysqlimport: Error: 2013, Lost connection to MySQL server during query, when using table: osopenuprn_202006

I'll try even smaller ones, but it's getting a bit ridiculous. Why disconnect the server in the middle of a query?

I have switched the configuration to use the petscan db connections only.

Just to add, if some queries take too long I can likely just change a parameter to fix that. Someone with access to the stats, let me know.

The reason I use that many connections is precisely to avoid long-running ones, as I had in a previous version; they would time out or lose database connection, so I re-wrote the code to use more but shorter queries.

In T243848#6251382, @Aklapper wrote:

@Magnus: Does that mean that https://phabricator.wikimedia.org/source/tool-quickstatements/browse/master/ will get out of sync and should be archived?

Yes.

I have "forked" it into github. Using diffusion was a mistake in the first place. Now at https://github.com/magnusmanske/quickstatements

Tried again with ssh://vcs@git-ssh.wikimedia.org/source/tool-quickstatements.git, no joy:

Permission denied (publickey,keyboard-interactive).
fatal: Could not read from remote repository.

tools.quickstatements@tools-sgebastion-07:~$ git remote remove origin
tools.quickstatements@tools-sgebastion-07:~$ git remote add origin ssh://phabricator.wikimedia.org/diffusion/2010/tool-quickstatements.git/
tools.quickstatements@tools-sgebastion-07:~$ git push fatal: The current branch master has no upstream branch.
To push the current branch and set the remote as upstream, use

FWIW, I believe all my (active) tools now use the new canonical URL.

As I wrote five years ago, I added a limit to upload only two files concurrently.
No further comment has been made since, so I consider this issue to be solved.

Thanks, that seems perfect!

In T254640#6202698, @bd808 wrote:

This config is part of the default configuration generated by webservice. That means we need to fix it upstream of the usage by this particular tool.

OK thanks

As a horrible short term fix for reasonator, I did:

$ ln -s $HOME/public_html/reasonator_types.js $HOME/public_html/_types.js

In T254640#6201620, @LucasWerkmeister wrote:

I don’t understand why this is an UBN when as far as I understand we know a workaround (restart without --canonical)?

In T194953#6185511, @bd808 wrote:

Is there a movement towards Rust in any significant segment of the Wikimedia movement? Are there clear benefits for Toolforge tools to be written and maintained in Rust?

I think this is a blocker to moving all tools to the canonical URL. T234617 ?

I cleared the error log on reasonator and switched to the previous URL schema, then back to the canonical one. No server errors in the log (some unrelated PhP ones, now cleared again)

@LucasWerkmeister thanks that worked, for now

Same with https://geohack.toolforge.org/geohack.php . Geohack is kinda central for Wikipedia so please help!

Thanks @bd808 !

Other files load fine. Haven't checked all though

URL is https://reasonator.toolforge.org/reasonator_types.js