this has resolved itself

Yes, works for me, thanks

Ran it again now, works fine. Closing this issue, will re-open if it keeps happening.

Now starting to see error 429 (too many requests), which is better (for debugging). Did something change in the meantime?

This happens ~50% of the time, re-running the exact same command often works.

Thanks, that works for me. It is much slower than before, as it needs to run full compilation every time, instead of just the changed bits. Ah well.

I tried that and got

[step-results] 2024-02-21T11:49:25.959261833Z Built image tools-harbor.wmcloud.org/tool-listeria/tool-listeria:latest@sha256:24ac86ba3d8e4c11568e158a172016dcf9b8e697312866c674ad5ae6f6b4954b

But the tutorial shows me only how to deploy a webservice. How do I start the bot executable as a normal toolforge job?

Ah, gridengine doesn't compile it either... Works fine locally though

Seems fixed now

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

The funny thing is, if you click on the movie https://commons.wikimedia.org/wiki/File:The_Sea_Beast_(1926).webm to play it, the moment the player comes up, it shows the correct duration. Where does it get that from?

I have limited it to 50 API connection at a time. Still throwing 104 errors. Either I did it wrong, or there is some other issue.

Thanks to the recently relaxed limitations per job on Toolforge, I am running this with more async threads, so without actual code change, it would have increased the number of connections.

Please use the bot as you like for testing!

This issue is persisting for the listeria tool. The bot is now down from ~40K edits/day to ~500. Something needs to be done, soon.

I am open to this, as I already have a Trove instance for baglama2, and it works well (after some initial problems).

FWIW I just saw this ticket, and added an index to the log table which should speed things up. I can also rewrite the query if that does not suffice, please let me know. Let's discuss Trove in T350862 .

D'oh!

Webservice is not running, returns only blank page, no error

Seems to be fixed now?

Aaaaand its back again.

@taavi Success! I tried /Users/mm6/php/magnustools/public_html/php/ToolforgeCommon.php which had magically reverted to non-group-writable again, so I changed it to group-writable now, and yes I can edit as before!

Other test case: /Users/mm6/php/magnustools/public_html/php/ToolforgeCommon.php

Odd. Now it does. Same issue.

FWIW, I can edit the files if I become the tool, but I don't like editing in shell, especially for larger code.

Changed the query so it's now working as before the ORES change.

Resolved, see T347367

PetScan restored for now. Waiting for the CORS issue to be resolved.

Restated again, and filed T346126

Done

Manually restarted webservice, as toolforge cannot possible do that automatically </sarcasm>

In T340844#8985436, @taavi wrote:

In T340844#8981291, @Magnus wrote:

And it doesn't answer the question why the webservice stops pretty much every time I start kubernetes jobs?

Please report such bugs so they can be fixed. toolforge jobs delete works properly now.

It might be worth mentioning that the limitations for Mix'n'match have been increased at some point, I can now run a job with 2CPUs/3GB RAM, which is much better.

I have restarted the webservice, all good now.

Ah here's a new one. I start kubernetes job rustbot. It runs. I start rustbot2. Now rustbot2 runs but rustbot has vanished:

tools.mix-n-match@tools-sgebastion-10:~/mixnmatch_rs$ toolforge jobs list
Job name:    Job type:    Status:
-----------  -----------  ---------
rustbot      continuous   Running
tools.mix-n-match@tools-sgebastion-10:~/mixnmatch_rs$ toolforge-jobs delete rustbot2 ; \rm ~/rustbot2.* ; \
> toolforge-jobs run --image tf-php74 --mem 1000Mi --continuous --command '/data/project/mix-n-match/mixnmatch_rs/run.sh second' rustbot2

@dcaro Listeria was one of them, and quickstatements at some point.

@aborrero To be fair, Mix'n'match used to fire off small/medium-sized jobs to gridengine, unlikely to bring a machine down. The requirement for uniquely named kubernetes jobs, and the limitation on the number of jobs, with no ""waiting queue", forced me to rewrite much of my code, into my own "job engine" that runs as a single kubernetes job. However, now the restrictions to individual kubernetes jobs bite me.

And it doesn't answer the question why the webservice stops pretty much every time I start kubernetes jobs?

That is to say, you say I'd have to move to VPS because you changed the tech stack, not because of resource usage. Resource usage was fine under grid engine. In fact, it was fine running the PHP version of the code. Don't tell me kubernetes can not support the Rust version which uses a lot less resources?

In principle it would (I am already running the petscan VPS), but this is missing my point.

This should do it. Tested and seems to work.

I confirm that "Silly Matlin" has been appearing in Mix'n'Match edits, but it is a straight lie that I "refused to comply". Not entirely surprising though. Anyway, it's a real issue, but User:Solidest seems to have found a bug, which I just saw since I don't follow the [[User_talk:Silly_Matlin]] page. I will look into it, again.

All good now

Thanks, I'll replace it when I get a chance

Apologies, my mistake, needs two underscores! All is well.

Thanks, better, but not quite there yet:

tools.wdqsbe@tools-sgebastion-10:~$ sql local
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 73249977
Server version: 10.1.44-MariaDB MariaDB Server

Found the problem, on my side. Apologies.

Consumer key is b5dc46b99399f49d03757216abd14e66 (QuickStatements). I didn't change the URL, and I think the toolforge clock is working fine.

Note: Oauth 1, not 2!

The reason I used bitbucket in the past was a (long removed) policy on github that limited the number of free repos. I also have some repos on github, which is owned by M$ and probably not very libre either.

Relevant code: https://github.com/magnusmanske/mediawiki_rust/blob/master/src/api.rs#L644

FWIW I started developing an new tool called GULP, now under early development.

After T329853 can we please add something so databases don't stay vanished until I complain at a "a proper support venue" (which seems to be only IRC, with mattermost link broken)?

Thanks, I forgot to check the "URL needs to start with" checkbox...

Just for posterity, I'd like to mention my own wikibase diff engine in Rust: https://gitlab.com/tobias47n9e/wikibase_rs/-/blob/master/src/entity_diff.rs

3 CPUs would be plenty, I think that's actually the per-tool limit, including the webservice?

This stopped working about an hour ago

I have successfully moved all data over to trove, and took a snapshot of the toolsdb version.
The web interface and the background tools have been switched over to the trove version and are reading/writing successfully.
As far as I am concerned, the toolsdb s51203__baglama2_p can be deleted.
Should I do that, or do you want to do the honors?

It took a few days but the database has been successfully copied over to trove. I am taking a final mysqldump from toolsdb now, then s51203__baglama2_p can be removed. I will post here when it's done.

Yes that's fine. Let me know when I can re-import it

Seems to have happened again just now. I was importing a rather large table (views), that has been running for hours(?). Not sure if that's related.

In other words, I would be hesitant to switch to a system where I have to manually restart the MySQL server every other week. I don't have time to work on all my tools as I would like, I can't run around kicking infrastructure as well.

OK I restarted the DB instance via horizon, when I saw that even horizon couldn't connect to it any more.

More concise, from toolforge:

tools.glamtools@tools-sgebastion-10:~$ mysql --defaults-file=~/replica.trove.my.cnf -h pwupvyu6i6k.svc.trove.eqiad1.wikimedia.cloud baglama2
ERROR 2002 (HY000): Can't connect to MySQL server on 'pwupvyu6i6k.svc.trove.eqiad1.wikimedia.cloud' (115)

Everything worked fine but (after a few days) I now can't connect to the instance any more:

Used command:  /usr/bin/ssh -v -N -S none -o ControlMaster=no -o ExitOnForwardFailure=yes -o ConnectTimeout=10 -o NumberOfPasswordPrompts=3 -i /Users/mm6/SpiderOak Hive/Configurations/ssh/id_rsa -o TCPKeepAlive=no -o ServerAliveInterval=60 -o ServerAliveCountMax=1 magnus@tools-login.wmflabs.org -L 61284:pwupvyu6i6k.svc.trove.eqiad1.wikimedia.cloud:3306

@Andrew I have created a new baglama2 DB there, and am currently importing the tooldb database. For that, I made a new replica file (~/replica.trove.my.cnf), and run in a screen:

mysqldump --defaults-file=~/replica.my.cnf --host=tools-db s51203__baglama2_p | mysql --defaults-file=~/replica.trove.my.cnf -h pwupvyu6i6k.svc.trove.eqiad1.wikimedia.cloud baglama2

This seems to be mostly done (after ~14h). I will then point everything to the trove DB. I might also dump the tooldb, and set up a regular trove dump. What's the best place to store compressed dumps, just the tool directory?

In T323502#8454473, @Andrew wrote:

In T323502#8453521, @Magnus wrote:

Is there a case for these large ones to move to their own instance, to take pressure of the toolforge DB system?

The answer to that question is YES! If you're willing to pick a database (not necessarily mix-n-match) and be a test subject I'd love to work on you with that.

Maybe this is a more general issue as well. I checked and it looks like I "own" 4 of the 10 largest tool databases. Is there a case for these large ones to move to their own instance, to take pressure of the toolforge DB system?

@nskaggs Thanks, the Cloud VPS DB option looks very interesting, but I think it would be overkill to move the 120GB DB over. I'll stick with the 10 connections for now, unless you recommend that this is hosted more efficiently (for both you and me) on Cloud VPS.

Thank you @nskaggs!

Also, the per-container limit still seems to be 1 CPU?

In T323502#8419755, @aborrero wrote:

Accepted the CPU quota change. The DB request is pending discussion.