Weekly update

• Kick-off meeting scheduled for next week

Great kick-off meeting today!

Pablo, as promised, a recent silent-drop dataset (covering approx the previous month) is available at stat1009.eqiad.wmnet:/home/cdanis/silentdrop.2024-07-24.tsv.gz

Thanks @CDanis! I'll take a look at it and share the results with you all.

Weekly update (some notes from the kick-off meeting)

• The expected deliverable will be an algorithm for an automated system that identifies malicious IPs (improving existing intuition-based approaches), e.g,. a list with a confidence value probably.
• The systems will not be real-time, the only part might have real-time component in the future is detecting when attack starts and the signature of the attack.
• Attack window detection is not a necessary feature, but it is appreciated.
• API requests will be ignored given previous analysis revealing the great presence of good-faith bots (in the future, we could track concurrency).

In T368389#10012122, @CDanis wrote:

Pablo, as promised, a recent silent-drop dataset (covering approx the previous month) is available at stat1009.eqiad.wmnet:/home/cdanis/silentdrop.2024-07-24.tsv.gz

@CDanis, I checked the webrequests of the most frequent silenced IPs and got results like:

(hidden IP) on 2024-06-26

(hidden IP) on 2024-07-08

(hidden IP) on 2024-07-03

I understand that most of the queries are against the Math extension and I wonder if this is an expected behavior and/or any explanation for it.

No, this is not expected behavior. I'm guessing this is external Mediawiki instances using our "Mathoid as a service" . I also see some requests there for "Instant Commons".

Thanks for finding this.

I'll change the silent-drop protection to ignore these calls.

Oh, and, silent-drop occurring during attack time windows should still make for a good signal, I think.

Change #1059126 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] Exclude some requests from concurrency tracking

https://gerrit.wikimedia.org/r/1059126

Thanks @CDanis! One consistent pattern I have found in the latest analysis is that attack time windows with very few abusive IPs involved (I consider abusive IP any IP with more than 1 request per second) barely contain IPs reported in AbuseIPDB (the best ground truth I have):

Below an exploration of the first 3 "attacks".

Attack time window 1

• begin=(2024, 7, 30, 18, 11)
• end=(2024, 7, 30, 18, 16)
• signature="(http_method='GET' and uri_host='en.wikipedia.org' and uri_path='/')")
• https://grafana.wikimedia.org/d/O_OXJyTVk/home-w-wiki-status?from=1722362400000&to=1722366000000&orgId=1 (peak on HTTP 5xx error responses (CDN) and MediaWiki backend response time)

The most common UA is about https://developer.amazon.com/support/amazonbot but I also found other well-known bots (https://openai.com/gptbot, http://yandex.com/bots, http://mj12bot.com, http://www.bing.com/bingbot.htm, http://www.google.com/bot.html).

Attack time window 2

• begin=(2024, 7, 30, 4, 59)
• end=(2024, 7, 30, 5, 4)
• signature="(http_method='GET' and uri_host='en.wikipedia.org' and uri_path='/')")
• https://grafana.wikimedia.org/d/O_OXJyTVk/home-w-wiki-status?from=1722312000000&to=1722319200000&orgId=1 (peak on Total HTTP request volume (Varnish frontend CDN))

The most common UAs in the 4 abusive IPs are about Timpibot (http://www.timpi.io) and ClaudeBot (claudebot@anthropic.com), but also the bots listed above (https://developer.amazon.com/support/amazonbot, https://openai.com/gptbot, http://yandex.com/bots, http://mj12bot.com, http://www.bing.com/bingbot.htm).

Attack time window 3

• begin=(2024, 7, 26, 20, 34)
• end=(2024, 7, 26, 20, 39)
• signature="(http_method='GET' and uri_host='en.wikipedia.org' and uri_path='/')")
• https://grafana.wikimedia.org/d/O_OXJyTVk/home-w-wiki-status?from=1722024000000&to=1722027600000&orgId=1 (peak on HTTP 5xx error responses (CDN) and MediaWiki backend response time)

This is very similar to the previous case. Many requests from ClaudeBot (claudebot@anthropic.com) and some other bots like https://developer.amazon.com/support/amazonbot.

Discussion

• None of the abusive IPs analyzed above are reported at AbuseIPDB so I was considering these cases as false positives. However, it is easy to find recent posts on the Internet reporting the bad behaviour of some of these bots:
  • https://herrbischoff.com/2022/07/timpibot-is-yet-another-badly-behaved-crawler
  • https://www.reddit.com/r/singularity/comments/1cdm97j/anthropics_claudebot_is_aggressively_scraping_the

• ClaudeBot made over 40K requests with cache_status in ('miss','pass','int-front') between 2024-07-26:20 and 2024-07-26:21.
  • I also think that silent-drop occurring during attack time windows might be a good signal... @CDanis, could you share with me an expanded version of the silentdrop dataset covering the entire month of July to see if these bots triggered any silentdrop action?

In T368389#10038133, @Pablo wrote:

• I also think that silent-drop occurring during attack time windows might be a good signal... @CDanis, could you share with me an expanded version of the silentdrop dataset covering the entire month of July to see if these bots triggered any silentdrop action?

Absolutely, available at stat1009.eqiad.wmnet:/home/cdanis/silentdrop.2024-08-02.tsv.gz
It covers much of June and all of July, across all cache hosts.

Thanks again @CDanis!

Weekly update: https://phabricator.wikimedia.org/T368389#10038133.

Change #1059126 merged by CDanis:

[operations/puppet@production] haproxy: exclude some requests from concurrency tracking

https://gerrit.wikimedia.org/r/1059126

Weekly update:

• Exploration of a subset of the most abusive IPs in July 2024 ('2001:67c:2f4c:2::xxx', '149.154.157.xxx', '2001:67c:2f4c:2::xxx', '192.116.36.xxx', '79.16.139.xxx, '75.89.152.xxx, '174.101.160.xxx'). Only 75.89.152.xxx was found at stat1009.eqiad.wmnet:/home/cdanis/silentdrop.2024-08-02.tsv.gz: { "time": "2024-07-09T11:59:28", "hostname": "cp1106", "prefix": "silent-drop_for_300s", "data": "75.89.152.xxx"}.

• For all these IPs their webrequest time series (hourly granularity) were retrieved:

2001:67c:2f4c:2::xxx	149.154.157.xxx	2001:67c:2f4c:2::xxx

192.116.36.xxx	79.16.139.xxx	174.101.160.xxx

75.89.152.xxx

• As I understood from https://phabricator.wikimedia.org/T353547#9654460, when an IP address surpasses a threshold of concurrently-executing requests against our CDN, we temporarily block that IP address and stop replying to anything at all from it for 5 minutes. Now running a notebook to analyze the webrequests of those IP that do appear in the silent drop datasets to identify differences in time series.

Note: I have added "xxx" suffixes to the IPs for privacy reasons on this public ticket.

Weekly update:

• Below are the time series of the top 3 IPs in silentdrop.2024-08-02.tsv.gz (the most blocked ones). Although the silent drop threshold is not about the volume of requests in a period of time but the number of concurrently-executing requests, I am a bit surprised that their request volume is noticeably lower than the abusive IPs shown in my previous message (which were not affected by silent drop protection).

70.32.23.xxx	2804:1454:1004:100::xxx	2001:1600:4:13:1a66:daff:fea5:xxx

• Below is the ratio of IPs reported in AbuseIPDB by the number of silent drop blocks the IPs have received (greater or equal). The higher the number of blocks, the higher the ratio. This is true up to a certain value (8) beyond which the trend reverses. This could be explained by the fact that the top silent drop blocked IPs often correspond to (a) good-faith external Mediawiki instances using our "Mathoid as a service" (no longer affected by this protection), and (b) IPs with requests to APIs such as Commons API, typically used by good-faith volunteer bots. Out of curiosity I have checked a sample of IPs with at least 8 silent drop blocks (the value where AbuseIPDB ratio is maximum:~80%). Many cases were about requests to "www.wikipedia.org/" on July 31 and requests to "en.m.wikipedia.org/wiki/united_states" on July 15. These were not found with the in-progress abusive IP detection approach as I only consider cache busting requests where cache_status IN ("miss","pass","int-front")(i.e., cache busting attacks) and these requests got hit-front. Given that IPs with multiple silent drop blocks are just a few thousands (those with at least 5 blocks are even less than 1K), it is worth considering directly checking these IPs on AbuseIPDB in order to expand our identification of abusive IPs that do not perform cache busting.

• With the in-progress abusive IP detection approach, I identified the following attacks and IPs (with over 100 requests):

June 2024

July 2024

• I created datasets for each month with features for every (attack, IP) pair including request_count, response_size_mean, uri_query_count, not_null_referer_ratio, user_agent_count, not_null_accept_language_ratio, is_pageview_ratio, desktop_ratio, mobile_web_ratio, mobile_app_ratio, not_null_referer_class_ratio, spider_ratio, abuseipdb_isReported. Then, first experiments with a standard XGBoost model were run:

train:June 0.8 split	train:July 0.8 split	train:June+July 0.8 split	train:June
test:June 0.2 split	test:July 0.2 split	test:June+July 0.2 split	test:July

• High precision and recall when training and testing with splits of the same dataset (June, July, June+July) but lower when training with June dataset and testing with July one. Therefore, next steps will focus on further experimentation, i.e., dataset preparation (including Spur metadata and August IP data), feature engineering, model settings, etc.

Weekly update:

• A new dataset with suspected attacks in August and involved IPs was created (e.g., attack 1, attack 2, attack 3)

• A new XGBoost model has been trained with data from June and July and tested with data from August. Results already meet the requirements of the hypothesis statement:

• As one could expect request_count and response_size_mean are the most important features to predict abusive IPs.

• I have also assessed how the model even improves when only IPs with a min. number of request are considered; it decreases the ratio of false negatives.

request_count ≥ 0; 100% IPs	request_count ≥ 10; 85% IPs	request_count ≥ 100; 73% IPs	request_count ≥ 1K; 55% IPs	request_count ≥ 10K; 21% IPs	request_count ≥ 100K; 2% IPs

• All the notebooks are available at stat1009.eqiad.wmnet:/home/paragon/ip-traffic such as stat1009.eqiad.wmnet:/home/paragon/ip-traffic/3_analysis_2024_08.ipynb for data collection and stat1009.eqiad.wmnet:/home/paragon/ip-traffic/4_xgb_2024_train_0607_test_08.ipynb for model training/testing. @CDanis, it would be great if you could let me know how you would need me to transfer the code/model.

Reminder: our ground-truth implies that an IP is abusive if it is reported in AbuseIPDB. However, I showed in previous updates some (very) abusive IPs that were not reported in that database. As a consequence, the real number of false positives should be lower than the one I am able to compute :)

No updates this week

No updates this week

No updates this week

Questions:

1. What form does this model take?
2. Where is the code?
3. How can MW code use this model?
4. Where's the documentation that explains how to retrain the model / deploy it?

@SCherukuwada

1. We have provided two approaches: 1) hard-coded logic based, 2) model. The two approaches arrive very similar performance metrics, which suggested that the latter (model) can be seen as merely a confirmation of the former (logic based), i.e. if the ML model does not out-smart the logics, it may imply that the problem in hand may not need a ML based approach. Should we observe other behaviors in the future, we could come back to revisit.

2. From Aug 30 update: "All the notebooks are available at stat1009.eqiad.wmnet:/home/paragon/ip-traffic such as stat1009.eqiad.wmnet:/home/paragon/ip-traffic/3_analysis_2024_08.ipynb for data collection and stat1009.eqiad.wmnet:/home/paragon/ip-traffic/4_xgb_2024_train_0607_test_08.ipynb"

3. We suggest that, for this application, T&S could start with implementing hard-coded logic. The reasons being
   • Performance metrics: as stated from the first point, the performances of logic vs model do not differ.
   • Maintenance: this ML model would require periodic retraining, while logic is unchanged going forward. Given the similar performance metrics, we recommend using logic based approach for now.
   • Deployment: we anticipate overhead for deployment as the current state of the code is self-contained in research env and may not meet the required deployment standard.

4. If we decide to use model (instead of logic), please see notebook stat1009.eqiad.wmnet:/home/paragon/ip-traffic/4_xgb_2024_train_0607_test_08.ipynb for model training/testing.

I will let @CDanis to update the engineering adoption side of the project.

Thank you.

How do research folks follow version control? Is there a repo where this should be checked in? If there isn't could you please choose one?

In T368389#10221202, @SCherukuwada wrote:

Questions:

3. How can MW code use this model?

Given our current goal is to integrate this data in concurrency limiting at the TLS terminator layer of our CDN, we haven't considered exposing the data to MediaWiki. But, if I had to think of serving those data to MediaWiki, I'd probably make them available via ipoid with its own ingestion pipeline.

@SCherukuwada thank you for your questions.

@XiaoXiao-WMF thanks for your response, let me provide two additional comments:

• ML was applied to formally verify the hypothesis statement. That said, as shown in T368389#10105538, request_count and response_size_mean are the most important features for predicting abusive IPs (i.e., IPs already reported in AbuseIPDB). Therefore, for integration, I would implement a threshold-based approach with those two features in place.
• As for the repository, I will discuss today with @fkaelin the best way to make existing code available in Gitlab.

The repo with the notebooks created for this project is https://gitlab.wikimedia.org/repos/research/abusive-ips (for privacy reasons, the data is not publicly released and all IPs in notebooks have been masked).