Page MenuHomePhabricator
Create Task
Maniphest T397570

Problem with SPARQL endpoint response and crawling on Toolforge
Closed, ResolvedPublic
Actions

Description

We have a Toolforge tool, Scholia. we currently seeing a lot of activity that looks like an incompent bot activity. We have specified a robots.txt https://scholia.toolforge.org/robots.txt switch specify that bots can only index very few pages and not crawl the site. However, we see in the uwsgi.log file request that I interpret as a bot that does not honor robots.txt. Du to the Toolforge proxy, it is difficult for us to throttle or ban the bot.

We also see a lot of problems with reaching the SPARQL endpoint from Toolforge. We have some server-side WDQS request and they often (or always?) fail at the moment. The bot crawls the pages with server-side WDQS requests, so there might be a connection.

We are currently moving some queries from SPARQL to API, but whether this will help on the more fundamental problem, I do not know.

Do Toolforge/Wikimedia people have suggestions for what we can do? One approach would be to move Scholia behind a login.

Event Timeline

Fnielsen created this task.Jun 21 2025, 4:37 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptJun 21 2025, 4:37 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Maintenance_bot added a project: Wikidata.Jun 21 2025, 5:29 PM
Alien333 subscribed.Jun 21 2025, 9:14 PM
Samwilson subscribed.Jun 22 2025, 12:08 PM
pfischer added a project: Discovery-Search (2025.06.13 - 2025.07.04).Jun 23 2025, 1:52 PM
pfischer subscribed.

We could at least investigate the reason for failing requests. Are those requests identifiable by a distinct user agent etc.?

pfischer moved this task from Incoming to User Support on the Wikidata-Query-Service board.Jun 23 2025, 1:56 PM
Fnielsen added a comment.Jun 23 2025, 2:04 PM

I do not see a User-Agent in my log. What I see is, e.g., something like:

[pid: 14|app: 0|req: 648/2912] 192.168.36.85 () {46 vars in 1028 bytes} [Mon Jun 23 13:56:07 2025] GET

Currently, I do not see far fewer failing requests now. Maybe the situation has improved. The author-disambiguator (at https://author-disambiguator.toolforge.org/ - not my tool) also had problem, but seems to work now, although it now has "Too many requests" on the main page.

The above log line is (probably) for a bot that crawls a robots.txt-restricted page.

bd808 subscribed.Jun 25 2025, 12:22 AM

Very few web crawlers these days actually respect robots.txt instructions. Even if you could see the IP addresses you would have difficultly using them effectively to block just the bad crawlers based on my personal experience in {T393487}. Requiring OAuth authentication via a Wikimedia SUL account is currently the best thing I can recommend to tools hosted on Toolforge or Cloud VPS generally that want to screen out aggressive web crawlers.

Fnielsen closed this task as Resolved.Jun 25 2025, 2:34 PM
Fnielsen claimed this task.

Thanks for the explanation.

We have some problems getting a good response from WDQS (legacy). We would need to improved our code.

Gehel moved this task from Incoming to Done on the Discovery-Search (2025.06.13 - 2025.07.04) board.Jun 27 2025, 8:33 AM
pfischer moved this task from Done to Reported on the Discovery-Search (2025.06.13 - 2025.07.04) board.Jul 11 2025, 3:36 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits