Thank you for tagging this task with good first task for Wikimedia newcomers!

Was set in the meantime:

User-Agent: *
Disallow: /

Your change looks good, thanks @TheresNoTime

This change looks good to me.
Looks like I lost merge rights since Gerrit>Github transition, so if some power user could merge this change, that would be great :)

An alternative to Oauth login of the users using their wiki account, could be to block crawlers could be a simple html page sent when a cookie is missing, with only a button to add the cookie in the browser. https://sebsauvage.net/wiki/doku.php?id=stupidantibot

Gadget is now enabled by default for all autopatrolled users on frwiki, starting today (2026-08-05).

I just filled out their contact form to ask them to add TWL proxy's IP address to the Cloudflare exclusion list. They quickly replied, understood the issue and submitted a whitelist request to their IT department. It might take a few weeks to be processed due to summer vacation.

Where did you push the commit that rolled-back \r changes? On https://github.com/toolforge/quarry/pull/98/commits I still see only 1 commit with \r edits.

Adding cloud-services-team in case they've more up-to-date tips to block crawlers on individual tools or a global solution for the platform.

According to the statistics, this tool records more than 10,000 page views per day, with peaks of up to 70,000.
Twoyears ago, this figure was a tenth of that; this is therefore likely to be the root cause.
The global issue to track bot abuse on toolforge is T226688: Block web crawlers from accessing Cloud Services but it hasn't any useful solutions yet.

If that helps, related issue for wmf-hosted tools: T226688: Block web crawlers from accessing Cloud Services. Consensus appears to be to set Anubis in front of webservers.

If someone wants to correct the edits on its local wiki, here is the script I ran on frwiki:

def run_bloomberg_fix():
    for page in site.search("insource:\"bloomberg.com/tosv2.html\"", namespaces=[0, 2], content=True):
        pywikibot.info(page)
        wikicode = mwparserfromhell.parse(page.text)
        for template in wikicode.filter_templates(recursive=True):
            if template.name.lower().strip() in ['lien web', 'cite web'] and template.has('url'):
                template_url = str(template.get('url').value)
                if not template_url.startswith('https://www.bloomberg.com/tosv2.html'):
                    print(f'not good url: {template_url}') 
                    continue

Hey @Pathoschild , do you know what are the performances limits that have been hit by the tool?
Perhaps there may be entries in the logs that might indicate, for example, that it get killed, or memory-related issues? Does the tool shut down and restart regularly, or does it just hang there unresponsive until a maintainer reboots it?
By identifying what's causing the problem, we should be able to request that the default Toolforge limit be increased.

It was also reported on metawiki on June 1st https://meta.wikimedia.org/wiki/Talk:The_Wikipedia_Library#Cannot_access_Cairn.info

According to Fdroid documentation:

The implementation of proprietary tracking or advertising libraries and analytics tools such as Google Play Services and Firebase and Crashlytics and proprietary ad/tracking SDKs are strictly forbidden in all applications.

We need to be careful not to block Android users who don't use Google Play.
Devices sold in mainland China and Huawei smartphones sold worldwide lack Google Mobile Services due to local regulations and sanctions: https://arstechnica.com/gadgets/2019/09/huaweis-new-flagship-smartphone-ships-without-google-apps/. I’m not sure whether this policy is still in force today, but it certainly applies to all Huawei phones sold at that time.
New sanctions such as this can arise as a result of geopolitical events, which have been far from calm these days.
Also, some phones are sold in Europe with Android forks like https://en.wikipedia.org/wiki//e/_(operating_system) or https://en.wikipedia.org/wiki/IodéOS, that may not support Play Integrity API.
I also remember having a bit of a struggle with FireOS, Amazon’s Android fork, because of Google Integrity issues that were preventing me from accessing certain apps.

If it can help, this tool list additions of external links in the pages with the timestamp across all wikis: https://spamcheck.toolforge.org/by-domain?q=africaintelligence.fr It collects data since November 2023.

In T150056#12106521, @Shivang9856 wrote:

Hi! I have successfully installed Huggle on Windows and tested launching it with command-line parameters (for example --safe). I also captured the required screenshots. Before editing Manual:Huggle/Startup, I noticed it is a translation source page. Could you please point me to the preferred page or section where this Windows command-line guide should be added? I want to make sure I edit the correct location.

You should be able to edit this page: https://www.mediawiki.org/wiki/Manual:Huggle/Startup. As the original page name is in English, we can only edit the English page, translation process will follow after that. You can upload some screenshots to Wikimedia Commons if needed. Probably the terminal parameters that can be found in terminalparser file would also be great to add to this doc page.

The only problem is that this status isn't only for approval, it could also be for declining the application, for example.

Understood, I rolled back this change from the PR. Thanks for the explanation about this one.

Thanks for your review @Samwilson! I've updated my PR with your suggestions: https://github.com/WikipediaLibrary/TWLight/pull/1537/files

Hello and welcome to wikimedia! all contributions are welcome. As it can be seen in this task description at the top of this page, contributions are supposed to be done on Meta-wiki directly, so don't hesitate to paste here links to your edits if you would like some feedback.
However, I don't check my email every day, so it might take me a little while to reply, but rest assured that I will reply sooner or later :)

It seems to work one day and then stop the next, on and off. It’s back working for me right now, but I’ll post again if the issue comes back.

Hello, frwiki and fr.wikisource contributors are also experiencing this access issue. Some would like to https://www-degruyterbrill-com.wikipedialibrary.idm.oclc.org/document/doi/10.7208/chicago/9780226816715/html but this page lists the resource as unlicensed, even if it's hosted on the main De Gruyter domain.

Same bug for this frwiki page: https://fr.wikipedia.org/wiki/Agn%C3%A8s_Vahramian

Indeed I don't see Cookie header in requests Huggle is sending, @Petrb if you have a moment to take a look around Cookiejar in ApiQuery ?

I was able to get rate-limited while patrolling on a single wiki at once. @Joe could it be possible to take a quick look in the logs to confirm you indeed see correct UA sent to the server?

Used UA is:

mer. janv. 21 00:19:05 2026   DEBUG[1]: UserAgent: Huggle/3.4.13 (http://en.wikipedia.org/wiki/WP:Huggle; 3.4.13 build: 4342 3.4.14-26-ge69a9117)

Hey @taavi , pinging you as you are the last person that have updated Quarry. When you have a minute, if it could be possible to merge this small PR, thanks! No need to deploy it specifically I guess, merging would be sufficient, this small css change could wait to be deployed when next code change is pushed to prod.

Similarly, translated links often contain determiners, even though they appear earlier in the original article. I am currently using a bot to remove the following determiners from the beginnings of links in translations published on frwiki:

In T400927#11156127, @Nikerabbit wrote:

Which translation engine?

In T214771#11156611, @KartikMistry wrote:

@Framawiki is it still happening? I tested a few articles and couldn't reproduce the issue.

Also reported on frwiki by @Olaf : https://fr.wikipedia.org/wiki/Discussion_Projet:Lingua_Libre#Lingua_Libre_not_working

Reported several times on frwiki too.

• https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Le_Bistro/10_ao%C3%BBt_2025#Bug_de_l'%C3%A9diteur_visuel
• https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Le_Bistro/25_juillet_2025#c-Lewisiscrazy-20250725084800-Bug_%C3%A9diteur?

Either duplicate of T218420: CX2: <span class="Z3988"... or T287936: CX2: Citation is generated twice with <cite class="citation web cs1" data-ve-ignore="true">

Still visible several times a day. 3 cases on yesterday translations: https://fr.wikipedia.org/w/index.php?title=Robert_Stanbury&diff=prev&oldid=227715029, https://fr.wikipedia.org/w/index.php?title=George_Gilder&diff=prev&oldid=227715018, https://fr.wikipedia.org/w/index.php?title=Slave_to_the_Rhythm_(chanson_de_Michael_Jackson)&diff=prev&oldid=227715005

anubis (https://anubis.techaro.lol/docs/) is a middleware tool that adds a challenge to suspicious requesters, that requires some computing time discouraging unsolicited users. It can be configured with a list of bad, strange or good user-agent, sending them a challenge or not depending of this information.
Beta cluster, or even all WMCS web proxy, could have it in front of all requests, with a default blocking policy to get out of the worst crawlers.
This soft is used on well-known websites with similar crawling issues, like https://git.kernel.org, https://gitlab.gnome.org, https://wiki.archlinux.org.
Another similar tool: https://git.gammaspectra.live/git/go-away

For information, here is the task for toolforge spam blocking, if it can help to choose a solution: T226688: Block web crawlers from accessing Cloud Services

In T255594#10809213, @Mvolz wrote:

Permissions issue on the paste.

Should be better. Paste contains two replacement functions, one for myprivacy gdpr portals, and another for perfdrive.com portals. You'll have to adapt it with your local template variable names.

Hi @Mbch331 , we've some similar issues on frwiki. Temporary solution we found in the meantime is to run a simple pywikibot bot running daily to replace automatically the portal links by the real citations. This way, the user is not impacted during its edits.
In case it could help you, here is the code I used for our case: P75891

Portals domains found on nl.wikipedia as explained in T255594: privacy.vakmedianet.nl / toestemming.ndcmediagroep.nl / cookies.rtlnieuws.nl. Issue reported on-wiki here and here

Other case: some citations are using URL from the portal "Radware Bot Manager Captcha" at the domain https://validate.perfdrive.com, like https://www.usinenouvelle.com/article/parrot-investit-dans-la-robotique-de-surveillance-avec-une-participation-dans-eos-innovation.N245941

There is a similar issue (T213262: CX: Avoid including rendered HTML from adapted transclusions in translated output) when {{better source needed}} is translated to {{Référence nécessaire}}, it is followed by this garbage:

Mass closing: Unfortunately, this project is no longer ongoing, see T158739: Create a Github notification bot for phabricator for next steps.

Mass closing: Unfortunately, this project is no longer ongoing, see T158739: Create a Github notification bot for phabricator for next steps.

Mass closing: Unfortunately, this project is no longer ongoing, see T158739: Create a Github notification bot for phabricator for next steps.

Mass closing: Unfortunately, this project is no longer ongoing, see T158739: Create a Github notification bot for phabricator for next steps.

Mass closing: Unfortunately, this project is no longer ongoing, see T158739: Create a Github notification bot for phabricator for next steps.

Mass closing: Unfortunately, this project is no longer ongoing, see T158739: Create a Github notification bot for phabricator for next steps.

Mass closing: Unfortunately, this project is no longer ongoing, see T158739: Create a Github notification bot for phabricator for next steps.

Mass closing: Unfortunately, this project is no longer ongoing, see T158739: Create a Github notification bot for phabricator for next steps.

Mass closing: Unfortunately, this project is no longer ongoing, see T158739: Create a Github notification bot for phabricator for next steps.

Mass closing: Unfortunately, this project is no longer ongoing, see T158739: Create a Github notification bot for phabricator for next steps.

Mass closing: Unfortunately, this project is no longer ongoing, see T158739: Create a Github notification bot for phabricator for next steps.