User Details
- User Since
- Oct 23 2014, 3:02 PM (439 w, 2 d)
- Availability
- Available
- LDAP User
- Magnus Manske
- MediaWiki User
- Unknown
Thu, Mar 23
Apologies, my mistake, needs two underscores! All is well.
Thanks, better, but not quite there yet:
tools.wdqsbe@tools-sgebastion-10:~$ sql local Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 73249977 Server version: 10.1.44-MariaDB MariaDB Server
Wed, Mar 22
Tue, Mar 7
Found the problem, on my side. Apologies.
Mon, Mar 6
Consumer key is b5dc46b99399f49d03757216abd14e66 (QuickStatements). I didn't change the URL, and I think the toolforge clock is working fine.
Wed, Mar 1
Note: Oauth 1, not 2!
The reason I used bitbucket in the past was a (long removed) policy on github that limited the number of free repos. I also have some repos on github, which is owned by M$ and probably not very libre either.
Fri, Feb 24
FWIW I started developing an new tool called GULP, now under early development.
Feb 17 2023
After T329853 can we please add something so databases don't stay vanished until I complain at a "a proper support venue" (which seems to be only IRC, with mattermost link broken)?
Feb 16 2023
Feb 7 2023
Thanks, I forgot to check the "URL needs to start with" checkbox...
Jan 24 2023
Just for posterity, I'd like to mention my own wikibase diff engine in Rust: https://gitlab.com/tobias47n9e/wikibase_rs/-/blob/master/src/entity_diff.rs
3 CPUs would be plenty, I think that's actually the per-tool limit, including the webservice?
This stopped working about an hour ago
Jan 9 2023
I have successfully moved all data over to trove, and took a snapshot of the toolsdb version.
The web interface and the background tools have been switched over to the trove version and are reading/writing successfully.
As far as I am concerned, the toolsdb s51203__baglama2_p can be deleted.
Should I do that, or do you want to do the honors?
Dec 23 2022
It took a few days but the database has been successfully copied over to trove. I am taking a final mysqldump from toolsdb now, then s51203__baglama2_p can be removed. I will post here when it's done.
Dec 20 2022
Yes that's fine. Let me know when I can re-import it
Seems to have happened again just now. I was importing a rather large table (views), that has been running for hours(?). Not sure if that's related.
In other words, I would be hesitant to switch to a system where I have to manually restart the MySQL server every other week. I don't have time to work on all my tools as I would like, I can't run around kicking infrastructure as well.
OK I restarted the DB instance via horizon, when I saw that even horizon couldn't connect to it any more.
More concise, from toolforge:
tools.glamtools@tools-sgebastion-10:~$ mysql --defaults-file=~/replica.trove.my.cnf -h pwupvyu6i6k.svc.trove.eqiad1.wikimedia.cloud baglama2 ERROR 2002 (HY000): Can't connect to MySQL server on 'pwupvyu6i6k.svc.trove.eqiad1.wikimedia.cloud' (115)
Everything worked fine but (after a few days) I now can't connect to the instance any more:
Used command: /usr/bin/ssh -v -N -S none -o ControlMaster=no -o ExitOnForwardFailure=yes -o ConnectTimeout=10 -o NumberOfPasswordPrompts=3 -i /Users/mm6/SpiderOak Hive/Configurations/ssh/id_rsa -o TCPKeepAlive=no -o ServerAliveInterval=60 -o ServerAliveCountMax=1 magnus@tools-login.wmflabs.org -L 61284:pwupvyu6i6k.svc.trove.eqiad1.wikimedia.cloud:3306
Dec 14 2022
@Andrew I have created a new baglama2 DB there, and am currently importing the tooldb database. For that, I made a new replica file (~/replica.trove.my.cnf), and run in a screen:
mysqldump --defaults-file=~/replica.my.cnf --host=tools-db s51203__baglama2_p | mysql --defaults-file=~/replica.trove.my.cnf -h pwupvyu6i6k.svc.trove.eqiad1.wikimedia.cloud baglama2
This seems to be mostly done (after ~14h). I will then point everything to the trove DB. I might also dump the tooldb, and set up a regular trove dump. What's the best place to store compressed dumps, just the tool directory?
Dec 9 2022
Dec 8 2022
Maybe this is a more general issue as well. I checked and it looks like I "own" 4 of the 10 largest tool databases. Is there a case for these large ones to move to their own instance, to take pressure of the toolforge DB system?
@nskaggs Thanks, the Cloud VPS DB option looks very interesting, but I think it would be overkill to move the 120GB DB over. I'll stick with the 10 connections for now, unless you recommend that this is hosted more efficiently (for both you and me) on Cloud VPS.
Dec 6 2022
Thank you @nskaggs!
Dec 1 2022
Also, the per-container limit still seems to be 1 CPU?
Nov 23 2022
And while I'm at it, I would like to request that the maximum number of database connections be increased to 20. This is mostly for the tool user database, not so much for the DB replicas, in case that makes a difference.
Nov 22 2022
Nov 21 2022
Oct 27 2022
--wait instead of --continuous seems to work?
I have other k8s jobs scheduled/running for this tool, they all work fine
The command is a thin bash wrapper around a Rust binary, compiled on toolforge. Binary starts just fine when run manually in shell.
Oct 20 2022
Oct 17 2022
Never mind, I just copied the binary to the tools directory and run it from there.
Moved all the jobs now except one, which depends on mysqldump, but that command is not in tf-bullseye-std. Is there any way to access that command from k8s?
Oct 14 2022
Done (already using k8s but someone left the cronjobs in place for just-in-case restarting, this is removed now).
Done.
Done.
Done.
Oct 13 2022
Done.
Done.
Oct 12 2022
Done.
Done.
Done.
Done.
Done.
Done.
Done.
Done.
Done.
Done.
Update: I migrated most of the cronjobs (except two) to kubernetes. Best I can do for now, I'll have a look at the remaining ones again once I migrate all my other tools.
Done.
Oct 7 2022
Hi, as you may know, I am a massive user of the toolforge environment. I am also an eager early adopter of new technologies. I will use this ticket (and not the other dozen or so I got for other tools) for some feedback, until it's specific to other tools.
May 31 2022
I think I fixed the URL for Special:OAuth, someone please try it in the wild
May 19 2022
Fixed
May 18 2022
Fixed
Fixed
Fixed
Fixed
Fixed
May 17 2022
Fixed
Fixed
Fixed
Fixed
Fixed
Fixed
Mar 20 2022
Different bug though. Fixed now.
Oct 28 2021
Thanks, I added descriptions and toolhub now picked it up.
Oct 27 2021
So it does not look like the crawler used the JSON I provided. For example, "drawshield" is clearly in https://magnustools.toolforge.org/toolinfo_buggregator.json but it does not show up in Toolhub search.
Oct 22 2021
Ah it looks like the crawler runs once every hour: https://toolhub.wikimedia.org/crawler-history
Jul 28 2021
Adding my own list of my tools, matched to the toolhub name where I could. Note that some of these are deactivated/broken (see notes), some are JavaScript on-wiki so not sure if that counts. The "open/closed' tickets are biased towards "open" while I go through them over time, but could become a quality score indicator down the line.
Jul 21 2021
I think I had a misconfiguration in the cronfile, this is fixed now. Please re-open/ping me if this still happens.
First time I see this issue here, will investigate.
Jul 12 2021
This was broken by the "toolforge subdomain migration".
Jun 30 2021
Turns out that single catalog takes ~21GB:
The collected data (from the BaGLAMa2 tool) is quite valuable for GLAM, including the historic ones, which is why I try to keep all of it.
Jun 7 2021
Fixed now.
May 21 2021
May 13 2021
A few notes:
- happy to help with any new development, as time allows; prioritized list preferred
- issue tracker at https://github.com/magnusmanske/listeria_rs but it's a bit messy, partially because no one reads the existing issues...
- the "killed" errors come from Toolforge killing the processes even though there is enough memory, but they allocate memory in large blocks which is suspicious, apparently. I have an idea how to get around that, but will require a bit of time
- the "central location" is the Commons Data: namespace. This works ~80-90%, and requires a final push, both in Rust and Lua. See http://magnusmanske.de/wordpress/?p=650
Mar 5 2021
Turns out flickr retired the secure.flickr.com domain without forwarding. Removed the "secure" but still using https, should work again.
Feb 25 2021
Also, doesn't seem to work for "has a birth date"?
https://www.wikidata.org/w/index.php?search=haswbstatement%3AP21%3DQ6581072++haswbstatement%3AP569&title=Special:Search&profile=advanced&fulltext=1&advancedSearch-current=%7B%7D&ns0=1&ns120=1
Feb 19 2021
Restarted wikidata-todo.
Jan 28 2021
It does work, however, since there is currently not a single page on euwiki where the latest revision has a wp10 score, the results will be empty.