In T169452#8886544, @IKhitron wrote:

Well, as a regular and very pleased user of Quarry, I tried Superset. Not helpful, it doesn't have most of databases. So, Quarry doesn't work any more, and I can't use Superset, so there is no way for me to run an urgent query (72767).

https://github.com/toolforge/quarry/pull/22/

In T302154#7724373, @Stashbot wrote:

Mentioned in SAL (#wikimedia-cloud) [2022-02-20T19:49:50Z] < @Andrew > moving nfs service from quarry-nfs-1 (bullseye) to quarry-nfs-2 (buster), testing to see if T302154 is a kernal or nfs-version issue

Add API endpoint(s) to https://hashtags.wmcloud.org/ that can be used to replace the prior database queries.

I've tried Superset for the first time and found some small points:

• query csv exports have a b' prefix in each cell. Here is the export of gendergap query:

https://github.com/toolforge/quarry/pull/20

@zhuyifei1999 any usecase in mind that need life-storage of the results? it may be a possibility to delete results after some time (ie. after 2 years). After such time links to queries shared on wikis are far less relevant.

To understand correctly, is this docker image used anywhere yet? I guess server is still using git clone and virtualenv without any container stuff ?

Deployed, thanks for the patch.

T272319: Frequent "Nonce already used" errors in scripts and tools was closed

Thanks for the revert reedy.
Tried to fix recent Oauth errors with a loop to retry login, but some connections were still failing. Will commit that properly.

Can someone import from logstash stack trace linked to this Request ID before it disappear?

Hi, here is a comment on a 6 years-old task :)
Having Sentry instance on labs for tools and websites is a long dream for me, that would these days more than ever be useful for lot of people.
I have been trying in recent days to set up a test instance, and use it with Quarry real-data. Found lot of things to fix with this :)

From the logs regarding the last stuck query (https://quarry.wmflabs.org/query/52267):

I have just read this ticket, sorry for the code I added and thanks for the remediation.

Seen 1897 errors and counting on both of bots I maintain, on both frwiki and wikidata. Looks like first seen at Nov 3, 2020 22:50 CET.
But it really started multiple time an hour starting Jan 15, 2021 01:02 CET (mediawiki deployment?).

See also this Grafana dashboard showing increase of daily PDF rendering by Proton from 80k to 20k, since beginning of August.

Two "page_title" columns exists in your query, and it looks like it's not handled correctly. So this works well when you rename your columns: https://quarry.wmflabs.org/query/49300.

From my records it was down for 9 hours and 20 minutes.
Logs on that day are full of:

sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'quarry-db-01.quarry.eqiad.wmflabs' ([Errno 111] Connection refused)")

I no longer see the message.

Thanks @Kb03 for the hint, you're right. Old habits to update :)

@MusikAnimal May I suggest you a quick temporary solution: as I see uptimerobot monitors only the website uptime, not its content. To have it trigger alerts when no new entries are detected since some time, you can add a special endpoint in the webservice to serve GOOD or BAD keywords. This healtcheck would be defined by sql query "does there any new entry in last 4 hours for each wiki". Uptimerobot can be configured to assert the presence of a unique keyword: in this case you can minitor the presence of GOOD one in the healtcheck. If no new entries are added you'll then receive an alert. That solve this task, but in a hacky way, I agree.

Local config located at https://vi.wikipedia.org/wiki/Wikipedia:Huggle/Config.yaml needs to be adapted to your wiki, by someone with programming basic knowledge. See this page for explanation about the values.

Other tools/wmcs monitoring related tasks i'm aware of:

• T53434: Establish an internal system or a recommended external system for monitoring user-created Toolforge web services, the big goal for tools platform
• T256361: PAWS: get new service and cluster metrics into prometheus
• T205150: Develop the monitoring of Quarry
• T152235: Simple logrotate service for users of Tools as stopgap before central logging talks about "future central logging" but can't find any task for it

Indeed,
Due to T238375: quarry-web-01 leaks files in /tmp, there were 10G of tmp files in /tmp, making / (/dev/vda3) full, again. 775 files.
Thanks for the report, it should be good now.

Hi @Bstorm, anything I can do here?

In T258330#6332705, @Lucas_Werkmeister_WMDE wrote

Unless it happens more frequently, I think we can ignore it.

@Addshore do you think it may be related to T212282: Timeout in Special Page Constraints check for France (Q142) ?

In T166995#6241197, @DannyS712 wrote:

So, to make sure I'm reading this right:
Bot goes there each page not editing in the last year

• Skips pages with incoming links
• Skips templates and modules with existing transclusions
• Skips categories with pages in them

And deletes the page? If this is a correct understanding, I should be able to do this