Tagging MediaWiki-Core-Platform-Team; I would be interested to hear your thoughts on this.

Not sure it would have an effect, robots.txt already has Disallow: /w/. Though I don't see any harm in trying.
(But how did Bingbot get blocked in the first place?)

In T347463#9202382, @Tgr wrote:

Not sure it would have an effect, robots.txt already has Disallow: /w/. Though I don't see any harm in trying.
(But how did Bingbot get blocked in the first place?)

Open proxy blocks targeting Microsoft Azure servers.

CCing the Editing team for info, as their analytics may be impacted.

@MNeisler This is (potentially) going to result in a us no longer logging (a lot of?) immediately-abandoned editattemptstep sessions with the wikitext editor -- the ones that only get the single server-side init event. If we see a sudden hit to editing numbers after this ticket is acted upon, this should be the reason. There's a chance we're already filtering them out because of the permission-denied status, but I'd need to go double-check exactly when in the process we log that.

That said, I do agree with @Tgr -- we already forbid /w/ for *, so I'd have thought that bingbot wouldn't be following these links already. Since it clearly is, based on the blocked edit logging, it might be a badly-behaved bot (or one that's just pretending to be bingbot, etc) and won't respect nofollow either.

Also worth considering that ?action=edit does work on the regular article URLs, so if there's anywhere on the wiki that generates the links in that form, the bot might have a completely legitimate way to be reaching these edit pages.

I haven't looked at the raw data, but based on the user agent and the observed behaviour of not honouring robots.txt I would expect that this is not an authorized spider for Microsoft Bing, but rather some other script or process that is using Bing's user agent for other purposes, in an attempt to fly under the radar through an SEO-friendly user agent.

Without going into details here, I will say that there is precedence for malicious activity using those exact charactistics.

I have no concerns with trying this, but more generally I would caution that:

• Making one link "different" and "special" in this way is potentially expensive for core maintenance. It encodes a new kind of business logic that would require preserving and supporting at every level and thus sacrifices a level of flexibility and potential.
• Attempting to control undesired side-effects from a plain URL navigation suggests to me there is a problem with the side-effect. At our scale, we probably can't be concerning ourselves with something as innocent as a script openly identifying as a bot, fetching a first-party promoted URL over GET (i.e. not a realistic human browser, and nothing beyond following stateless/cookieless/POST-less navigation, and not a randomised or hand-crafted URL). At our scale, this is going to happen, a lot, whether we want to or not.

If this a concern, one may want to think about how the side-effect works under the hood to see if we can tackle that in another way. For example, if the reason for this server-side beacon is to support detecting the "start" of edit sessions when using source mode (not VisualEditgor, and before JS can/will/has arrived for WikiEditor), one could start that client-side instead for most clients, and imply the start for no-js saves by the absence of an "init" given the high proportion of crawlers (or possibly through <noscript><img>, but I guess that's hard nowadays given EventLogging requires POST now, but possibly Statsv could be used). The general interest and proportion of Grade C browers in edits is presumably deducable from the "attempt" and "complete" events already, complemented with a one-off research that already took place in ther past to determine whether this population has a different abandonment rate.

TLDR: At our scale, if something like curl "https://en.wikipedia.org/w/index.php?title=Main_Page&action=edit" does something undesirable, that may be an "us" problem.

In T347463#9215707, @Krinkle wrote:

I haven't looked at the raw data, but based on the user agent and the observed behaviour of not honouring robots.txt I would expect that this is not an authorized spider for Microsoft Bing, but rather some other script or process that is using Bing's user agent for other purposes, in an attempt to fly under the radar through an SEO-friendly user agent.

I did some quick analysis in this notebook, and as far as I can tell, 99.5% of bingbot attributed events, at least when looking at IP and IP range blocks, are coming from IP ranges associated with Microsoft, which are also documented as IP ranges for bingbot.

Maybe check the enwiki robots.txt with the Bing robots.txt tester tool?

In T347463#9203576, @DLynch wrote:

Also worth considering that ?action=edit does work on the regular article URLs, so if there's anywhere on the wiki that generates the links in that form, the bot might have a completely legitimate way to be reaching these edit pages.

I think this is the most likely explanation, and also not something we can easily prevent. We could canonicalize edit URL by responding with a 301, either from Varnish or from MediaWiki::tryNormaliseRedirect().

In T347463#9216398, @Tgr wrote:

Maybe check the enwiki robots.txt with the Bing robots.txt tester tool?

Checked it, and Bing has no complaints about it.

In T347463#9216453, @Tgr wrote:

In T347463#9203576, @DLynch wrote:

Also worth considering that ?action=edit does work on the regular article URLs, so if there's anywhere on the wiki that generates the links in that form, the bot might have a completely legitimate way to be reaching these edit pages.

I think this is the most likely explanation, and also not something we can easily prevent. We could canonicalize edit URL by responding with a 301, either from Varnish or from MediaWiki::tryNormaliseRedirect().

Depends... we can't stop random people from linking manually to it, but the scale of the blocked edits is sufficient that it'd pretty much have to be an automatically generated link, likely by mediawiki itself, so if we can work out where that's happening we can just canonicalize that to the /w/ version.

Can anyone find a place that generates such a link? I did some cursory looking around and didn't see anything.

In T347463#9216129, @kostajh wrote:

In T347463#9215707, @Krinkle wrote:

[…] I would expect that this is not an authorized spider for Microsoft Bing, but rather […].

Without going into details here, I will say that there is precedence for malicious activity using those exact charactistics.

[…] 99.5% of bingbot attributed events, at least when looking at IP and IP range blocks, are coming from IP ranges associated with Microsoft, which are also documented as IP ranges for bingbot.

Given a similar case where the activity is malicious (and not crawler-like), this may be due to Azure Cloud. The bingbot docs mention a lot of disparate ranges there. It seems to me either Microsoft is trying to artificially create favourable treatment for Azure customers by knowingly listings its IP ranges as affiliated with Bing (unlikely), or (more likely, but no less shocking) Microsoft runs part of Bing in Azure and fails to differentiate its outgoing network, or (Ocam's Razar) Microsoft is scared to accidentally leave out anything for Bing's spider so they simply document every IP range owned by Microsoft just in case.

Whatever the case, the end result is that Azure customers can trivially run around the web immitating Bingbot.

My assumption until this moment was that the latter is certainly the case, people do use public clouds like Azure, Google, and Amazon to reduce suspicion through fairly generic IP spaces with known brands in their AS names; and there are indeed malicious bots running in Azure that use Bingbot user agents. What I didn't realize until now is that Bingbot own documentation lists out Azure IP ranges. Whateve ther reason for that, that makes the problem significantly worse.

In T347463#9226885, @Krinkle wrote:

The bingbot docs mention a lot of disparate ranges there. It seems to me either Microsoft is trying to artificially create favourable treatment for Azure customers by knowingly listings its IP ranges as affiliated with Bing (unlikely), or (more likely, but no less shocking) Microsoft runs part of Bing in Azure and fails to differentiate its outgoing network, or (Ocam's Razar) Microsoft is scared to accidentally leave out anything for Bing's spider so they simply document every IP range owned by Microsoft just in case.

The bingbot IP list is weirdly long but still much, much smaller than the full Azure range. The bingbot range is 3968 IPs in total, which is not *that* large. The full Azure range list contains 79007270 IPv4 addresses (and a lot of IPv6 ones).

In T347463#9225279, @DLynch wrote:

Can anyone find a place that generates such a link? I did some cursory looking around and didn't see anything.

We could log the referrer. Or correlate the existing logs with the webrequests table, which is more effort but does not require a code change.

In T347463#9216129, @kostajh wrote:

In T347463#9215707, @Krinkle wrote:

I haven't looked at the raw data, but based on the user agent and the observed behaviour of not honouring robots.txt I would expect that this is not an authorized spider for Microsoft Bing, […]

I did some quick analysis in this notebook, and as far as I can tell, 99.5% of bingbot attributed events, at least when looking at IP and IP range blocks, are coming from IP ranges associated with Microsoft, which are also documented as IP ranges for bingbot.

In T347463#9230043, @Tgr wrote:

We could log the referrer. Or correlate the existing logs with the webrequests table, which is more effort but does not require a code change.

I queried the following from Hive, using the same 2023-10-01 date as the Kosta's notebook:

SELECT
  COUNT(*) AS hits,
  client_ip,
  SUBSTR(uri_path,0,6) AS my_path,
  user_agent
FROM
  wmf.webrequest
WHERE
  year = 2023
  AND month = 10
  AND day = 1
  AND (uri_path = "/" OR uri_path = "/w/index.php" OR uri_path LIKE "/wiki/%")
  AND uri_query LIKE "%redlink=1%"
GROUP BY client_ip, SUBSTR(uri_path,0,6), user_agent
ORDER BY hits
DESC LIMIT 50;

This returned about 2 million hits in total from these top 50 results. Of these:

• exactly 0 are from a bingbot user-agent
• exactly 0 are from a /wiki/% path.

The singular IP on top of this query is responsible for 1.7 million of the 2 million events, and uses generic Chrome on Windows user agent (no bot identification), from an IP not affiliated with any well-known spider or public cloud.

The other top 49 contenders, with 2K-70K entries each, include:

• alleged Googlebot client, not from Google-affiliated IPs, but from IPs belonging to public cloud that is not Microsoft/Google/Amazon.
• a client with the literal string "Googlebot"
• alleged Amazonbot client, from the same popular public cloud (but not Microsoft/Google/Amazon)
• a client with the literal string "Chrome"
• alleged Googlebot clients, from IPs belonging to a well-known consumer ISP in Europe.
• alleged Firefox/1.5 clients.
• alleged Opera/8.54 clients.
• alleged MSIE 7.0 clients.

I believe this means anything else that made requests to / or /w/index.php or /wiki/% with the query redlink=1 (likely with action=edit) on 1 Oct 2023, must've made fewer than ~2000 requests from any given IP.

I looked for redlink=1 as those are the most likely action=edit links to be prioritised by a crawler, since we link them from within the content area.

Raw data in an NDA-restricted paste at P52844.

Repeating the same query for any action=edit (not just red links, so most likely discovered from p-cactions links in the skin) we get about 2.5 million results from the top 50 by (client_ip,my_path,user_agent). Of these:

• also, exactly 0 from a bingbot user-agent
• exactly 2 of the 50 involve /wiki/% paths, responsible for 25K and 27K hits respectively, both using genuine-looking Googlebot user agents from genuine Googlebot addresses.

The top one is responsible for 2.1M of the 2.5M results and originates from the same Chrome/Windows user-agent and same non-cloud IP as the top for the redlink=1 dataset. The rest made 7K-160K hits each, and breakdown in a similar way as the above one. Raw data in an NDA-restricted paste at P52845.