Page MenuHomePhabricator

[bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO)
Closed, ResolvedPublicBUG REPORT

Description

What happened?

I run Quarry queries several times a day, generally with no problem. Today, none of them will run and I get this message on the page:

*Access denied for user 'quarry'@'172.16.2.72' (using password: NO)

I don't know whether or not that is my IP address, if it has been blocked or how to change it (since, as far as I know, I'm not blocked). I am logged in when I'm running these queries.

What should have happened?

The queries should have run the same as always, giving me back a list of pages with problems. Please help.

Event Timeline

Sorry, but whatever was the problem is fixed, it's working now. Feel free to close this. Thanks.

Aklapper renamed this task from [bug] Quarry bug to [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO).Mon, May 20, 4:51 PM
SD0001 added subscribers: rook, SD0001.

Still occurring intermittently. Error message suggests a failure in accessing the Trove db. @rook did we make any changes to the setup lately?

I spoke too soon. It's happening again.

Sometimes, when I hit the "Submit query" button two or three times after no success, the query runs. Other times, no luck at all. "Intermittent" is a good description, I agree.

But sometimes I have to hit "Submit" 10 of 15 times before it runs.

Sometimes I'm getting the message with a slightly different IP number:

*Access denied for user 'quarry'@'172.16.2.252' (using password: NO)

I always get this error if I enter a ToolsDB database as the db name (e.g. s55771__wsstats_p).

If you want to see an example of this, try Category redirects and look at the history.

I could never get this one to run. I use to be able to submit a dozen or more times and the query would eventually run. No more. It looks like this is happening with other editors (see Recent queries list ) so how can it be resolved?

Thanks to whomever fixed this problem.

GTrang claimed this task.
GTrang subscribed.

Bug appears to have been fixed.

Many failures from today. (Could it be a coincidence that it's occurring only on the days someone is trying to test T348407?)

I suspect they are related yes. Maybe Quarry is trying to connect to another database but using ToolsDB credentials, or viceversa.

Ah well, looks like the deployment of https://github.com/toolforge/quarry/pull/40 didn't fully go through. The keys TOOLS_DB_USER and TOOLS_DB_PASSWORD are missing in config.yaml on the pods, although I could have sworn this wasn't the case the last time around when I looked into it.

The default values are empty. So quarry is trying to connect to ToolsDB with no password and username defaulting to the server username ("quarry"). The (using password: NO) in the error message checks out.

But this doesn't explain why the issue occurs when querying the wiki replicas.

Sorry to post a duplicate notice. I couldn't find this one when I wanted to post about this bug.

Well, for the last 21 hours no querry could run. So it's pretty bad.

Ah well, looks like the deployment of https://github.com/toolforge/quarry/pull/40 didn't fully go through. The keys TOOLS_DB_USER and TOOLS_DB_PASSWORD are missing in config.yaml on the pods

Good spot! I think I modified the wrong file, this one should fix it: https://github.com/toolforge/quarry/pull/46

Xaosflux triaged this task as High priority.EditedWed, Jun 12, 2:35 PM
Xaosflux subscribed.

Problem still occurring today and all recent requests have failed (see https://quarry.wmcloud.org/query/runs/all)

Requests fail with:

{F55264241}

fnegri changed the task status from Open to In Progress.Wed, Jun 12, 2:45 PM
fnegri claimed this task.
fnegri moved this task from Backlog to Bugs on the Quarry board.

But this doesn't explain why the issue occurs when querying the wiki replicas.

Figured out the cause. I've filed https://github.com/toolforge/quarry/pull/47.

https://github.com/toolforge/quarry/pull/46 and https://github.com/toolforge/quarry/pull/47 should probably fix the issue, but I'm not sure how to deploy those after they are merged.

I found the following instructions at https://github.com/toolforge/quarry/blob/main/README.md#deploying-to-production

From the quarry-bastion in the git checkout that has the state file. bash deploy.sh mysql -uquarry -h <trove hostname created in last step> -p < schema.sql In horizon point the web proxy at the new cluster.

I'm not sure either about how to deploy in the new k8s-based setup. However, /home/rook/quarry seems to be the git checkout referenced above.

I'm confused because that directory has a checkout of a non-main branch:

root@quarry-bastion:/home/rook/quarry# sudo -u rook git status
On branch T362213-queued

root@quarry-bastion:/home/rook/quarry# sudo -u rook git log --oneline |head
9fe166f auto update of  tag
573b9f0 Do not offer stop button on queued
c417f9f Add ToolsDB credentials (#40)
49862cc Tofu state to object store (#39)
dc3e220 Update helm on pr (#33)
5b2ff73 quarry on k8s (#31)
01d3c4a improve grammar of synchronization delay message (#32)
312990c minikube helm chart (#28)
f866d1f Link to new filename to make VM happy (#30)
03900ff Migrate from uwsgi to gunicorn + make dockerfile suitable for prod (#29)

FYI: runs in the last hour are passing now

Some of my queries are running okay now, others are just spinning their wheels, like they are running but they never conclude or take an unusual length of time to do so. So, a partial fix. And with one query, I get an error message

"Unknown column 'pl_title' in 'field list'"

which might mean that there is suddenly a code problem for some reason.

Some of my queries are running okay now, others are just spinning their wheels, like they are running but they never conclude or take an unusual length of time to do so. So, a partial fix. And with one query, I get an error message

"Unknown column 'pl_title' in 'field list'"

which might mean that there is suddenly a code problem for some reason.

@Liz this this is a separate issue that was a planned change see https://meta.wikimedia.org/wiki/Tech/News/2024/20 (i missed it as well so also have broken queries)

The specific error described in this bug report (Access denied for user 'quarry'@'172.16.2.72' (using password: NO)) is no longer happening, so I'm marking this task as Resolved.

Please open new tasks if you find other issues (slow queries, etc.)

I'm not getting that message any more but some queries are just running endlessly and never finishing. I had one that ran for more than 3 hours before I finally cancelled it. Does this warrant a new case to be opened?

@Liz are the queries that never finish different from other queries, or are they similar but sometimes they randomly fail? Did similar queries work fine in the past?

If this keeps happening, please open a new task, adding an example of a query that is failing to complete.

I'm not getting that message any more but some queries are just running endlessly and never finishing. I had one that ran for more than 3 hours before I finally cancelled it. Does this warrant a new case to be opened?

See T367654.