Awesome, thank you so much for the explanation! It might be worth adding that to the README here: https://dumps.wikimedia.org/other/pageview_complete/readme.html

Yes it seems to have worked! Thanks so much.

Awesome, thanks for the feedback. I'll look into Trove.

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

I'm completely new to Kubernetes but have been reading through https://wikitech.wikimedia.org/wiki/Kubernetes/Kubernetes_Workshop. Does WM Cloud provide k8s clusters, or is it expected that we would provision our own cluster on individual cloud VPS instances?

Thank you for all the information, it is very helpful! We can stick to asynchronous communication if that's what works best, no problem. I guess we can keep using this ticket for Q&A?

Looking at that wiki page I linked, it seems at least somewhat out of date. I'd like to work on upgrading Python to at least 3.11, since 3.7 is EOL since June of 2023. Of course this might require upgrading dependencies as well. I see that @Framawiki has some quarry-dev-* instances with a puppet "skip" note of:

https://github.com/toolforge/quarry/pull/25

@SD0001 I found: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Quarry, but I don't think I fully understand it

I thought I was going crazy because REPLICA_HOST does in fact exist in default_config.yaml, but it turns out it isn't used anywhere in the repo so it must be a vestige of an old way of calculating the hostname.

Forked discussion to T348364

Shouldn't REPLICA_DOMAIN be set to analytics.db.svc.wikimedia.cloud for this to work? I haven't tried it myself yet. Then you would get enwiki.analytics.db.svc.wikimedia.cloud which would be correct right?

FWIW I set up the dev environment without any issue and was able to run queries against mywiki.

Confirmed: I got the github invite. I can also access the instances with my wikitech account, thanks!

I'm audiodude on github. Thanks!

I assume we need some kind of access to the Github repo too? https://github.com/toolforge/quarry

Another puzzling part is that MariaDB doesn't appear to be returning results as floats. I exposed the mywiki MariaDB in docker and ran this:

Documenting my investigation (no solution found).

So is it correct that we're looking for a new maintainer, but only in the capacity of migrating all usage of Quarry to Superset? That is, no new features are planned for or expected of Quarry and we expect to turn it down once Superset has feature and use case parity?

My pleasure. Did I save mwdiffs and mwpersistence? That's the goal. If so should one of us update the announcement to [Cloud-announce]?

I successfully built the singleuser image with jupyterlab=3.6.3 and this PR: https://github.com/toolforge/paws/pull/309

Maybe not quite that. Looks like mwpersistence requires deltas -> yamlconf -> PyYAML 5.4.1:

My guess is that the implicated libraries (mwdiffs and mwpersistence) are written in Python 2 and can't upgrade to the latest version of PyYAML, but that's just a guess.

I think removing those libraries simply causes pip to resolve the dependency to a later version of PyYAML which doesn't have the issue, as seen here: https://github.com/flyteorg/flytekit/pull/1752/files

I tried bumping jupyterlab to 3.6.3 as seen in this commit: https://github.com/toolforge/paws/pull/308/commits/425a53a6449198beca9e3466c32f3604bdfbe31e

Thank you!

2fa request file (following instructions here) is at:

Yes we'd like to continue to use toolsdb.

Okay I've stopped the gridengine job using tools.enwp10@tools-sgebastion-10:~$ webservice --backend=gridengine stop and applied the redirect using the commands you linked @bd808. It looks like it's working. Do we still need to delete our toolsdb database as well?

Our tool has been running at wp1.openzim.org for multiple years now. I don't think we actually require the redirect, but if it's easy enough to setup we could look into it. Thanks for the pointers.

@nskaggs So it sounds like the answer is "No, we cannot keep our toolsdb database" and we would have to migrate to the linked VPS solution?

That's right. We're still using the Toolforge database, but we don't have any jobs running in GridEngine.

Copied from the duplicate bug:

Okay I've migrated s51114_enwp10 to s51114__enwp10. There are a few tables left in the old database, but they are leftovers from the tool migration process and can be deleted.

Shouldn't step 5 be CREATE DATABASE not CREATE TABLE?

s51114_enwp10 is definitely still in use. I don't know how to rename it safely, so would need help with that.

Yes it's all resolved now, you can publish it if you like, I don't have an opinion and am not familiar with the policy.

It worked! Thanks so much both of you for your help!

In T269513#6683826, @DannyS712 wrote:

Should T269898: Bot needs account unlocked be merged here? Or is there anything that can be done publically there?

In T269513#6672167, @Reedy wrote:

That seems to be @OlegAlexandrov who is still active on enwiki. CC'd them to this task...

You are correct, this database is quite old. Thank you for your help with this, I have confirmed that I can read from the database again.

How do you not have the login password for it? Does someone else?

Regarding https://phabricator.wikimedia.org/T269513#6671553 it looks like the credentials in replica.my.cnf allow login access to the databases, but the user doesn't have permissions to s51114_enwp10 which is the name of the database that the tool uses on tools.db.svc.eqiad.wmflabs. Would it be possible to grant permissions?

Thank you for this information, but I don't have access to the login password for WP 1.0 bot, I only have the bot password.

I've got a new replica.my.cnf, but I'm getting the following:

Thank you so much Andrew! How do I get new credentials?

I also leaked the "bot password" for the WP 1.0 Bot. I don't know where or how to reset or rescind this password. When I log into mediawiki.org I see "Audiodude@WP_1.0_bot" but I believe that makes edits as my user account. The account I need to reset is "WP 1.0 bot@WP_1.0_bot"

Thank you, sorry about this. I looked around a bunch for my recovery codes but I can't remember at all where I put them!

Okay I've deployed the fix to WP 1.0 bot as of tonight's update. It should be good the next time you run the numbers. Please let me know.

Wanted to come here and acknowledge this issue. I've opened a bug against the WP 1.0 project to fix this. It should be taken care of this weekend. Thank you @DeltaQuad for the link to mwclient workaround!

That seems to have resolved the issue!

I'm going to try replacing DictCursor with SSDictCursor and see if that helps.

Hi @Kelson , I think you gave extra parameters to your restart command. I tried:

Okay, I had previously added a setenv command for adding the PERL5LIB to the environment, but I didn't add it for list2.fcgi and log.fcgi, which now have their own sections in their configs for the environment variable. Things seem to be working now

This isn't entirely accurate. The tool was in fact migrated to stretch last week. However, certain pages may still be erroring out, and that needs to be addressed. I will look at it later today when I have my laptop.

And yes, the ORM I was speaking of was SQLAlchemy. It was much too slow with the session management strategy I was using, and manually managing multiple sessions and object eviction in order to speed it up would make the code too complicated in my opinion and negate the benefits of using the ORM.

Thanks for the comments. I realized that the test db never needs to be live in production and that helped me figure out the solution.

That looks promising for the Travis CI environment yes. I guess my problem is that I'm not developing locally, I'm developing directly on the Toolforge instance and I guess I just need to provision a database for test there?

The web tool (https://tools.wmflabs.org/enwp10) looks up to me. What specifically is not working?

Restarted it again. Let me know if it works.

I just restarted it. The tool seems to die every 30 days or so, this last happened on Sept 28.

Actually, I believe plaintext would be more useful, but HTML would be a good starting point since it's reasonable to strip out.

In fact the XML is not the problem, it's the wiki markup. From what I understand, the wiki markup is not defined by any formal grammar, so parsing it is non-trivial. Tools like mwparserfromhell help, but can be difficult to configure and not always accurate. There's also of course the problem of recursive template expansion.