Maniphest T420833

Beta cluster is being crawled to death by bot traffic coming from residential proxies
Open, HighPublicBUG REPORT
Actions

Assigned To

None

Authored By

	AlexisJazz
	Mar 22 2026, 4:15 AM

Project Tags

Referenced Files

	F99313181: Solution to beta cluster being crawled to death.pdf
	Aug 21 2026, 12:58 AM

	F91582600: Screenshot 2026-07-02 at 11.40.00.png
	Jul 2 2026, 5:42 PM

	F83049286: Screenshot 2026-05-20 at 19.53.50.png
	May 21 2026, 1:54 AM

	F74917361: Screenshot 2026-04-03 at 11.27.54.png
	Apr 3 2026, 5:39 PM

	F73765830: Screenshot 2026-03-26 at 08.36.23.png
	Mar 26 2026, 2:51 PM

	F73665242: Screenshot 2026-03-25 at 10.46.47.png
	Mar 25 2026, 4:48 PM

	F73533287: Screenshot 2026-03-23 at 11.06.40.png
	Mar 23 2026, 5:08 PM

	F73533258: Screenshot 2026-03-23 at 10.59.46.png
	Mar 23 2026, 5:08 PM

Subscribers

View All 38 Subscribers

Description

Steps to replicate the issue (include links if applicable):

https://commons.wikimedia.beta.wmcloud.org/wiki/Main_Page
https://commons.wikimedia.beta.wmcloud.org/wiki/Special:Random
https://en.wikipedia.beta.wmcloud.org/wiki/Special:Random
Beta enwiki main page loads quickly though. Beta commons main page does not

What happens?:
It's taking minutes to load anything.
https://en.wikipedia.beta.wmcloud.org/wiki/Special:Random just gave me:

Error

Our servers are currently under maintenance or experiencing a technical issue

If you report this error to the Wikimedia System Administrators, please include the details below.

Request served via deployment-cache-text08 deployment-cache-text08, Varnish XID 248807457
Error: 503, Backend fetch failed at Sun, 22 Mar 2026 04:06:51 GMT

With wget:

wget https://commons.wikimedia.beta.wmcloud.org/wiki/File:Test_upload_header_3_$100_bill8.svg
--2026-03-22 05:22:33--  https://commons.wikimedia.beta.wmcloud.org/wiki/File:Test_upload_header_3_00_bill8.svg
Resolving commons.wikimedia.beta.wmcloud.org (commons.wikimedia.beta.wmcloud.org)... 185.15.56.36
Connecting to commons.wikimedia.beta.wmcloud.org (commons.wikimedia.beta.wmcloud.org)|185.15.56.36|:443... connected.
HTTP request sent, awaiting response... 503 Backend fetch failed
2026-03-22 05:24:10 ERROR 503: Backend fetch failed.

From https://commons.wikimedia.beta.wmcloud.org/wiki/Special:Upload trying to upload something:

Error

Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes.

See the error message at the bottom of this page for more information.

If you report this error to the Wikimedia System Administrators, please include the details below.

Request from [IP redacted] via deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud, ATS/9.2.11
Error: 504, Connection Timed Out at 2026-03-22 04:30:46 GMT

What should have happened instead?:
Hmeep hmeep!

Other information (browser name/version, screenshots, etc.):

Unsure if related, but these ping results are strange? Why are different domains answering?

PING commons.wikimedia.beta.wmcloud.org (185.15.56.36) 56(84) bytes of data.
64 bytes from upload.wikimedia.beta.wmflabs.org (185.15.56.36): icmp_seq=1 ttl=53 time=109 ms
64 bytes from upload.wikimedia.beta.wmflabs.org (185.15.56.36): icmp_seq=2 ttl=53 time=106 ms
64 bytes from config-master.wikimedia.beta.wmflabs.org (185.15.56.36): icmp_seq=3 ttl=53 time=102 ms
64 bytes from upload.wikimedia.beta.wmflabs.org (185.15.56.36): icmp_seq=4 ttl=53 time=102 ms
64 bytes from beta.wmflabs.org (185.15.56.36): icmp_seq=5 ttl=53 time=103 ms
64 bytes from instance-deployment-cache-text08.deployment-prep.wmcloud.org (185.15.56.36): icmp_seq=6 ttl=53 time=102 ms
64 bytes from upload.wikimedia.beta.wmflabs.org (185.15.56.36): icmp_seq=7 ttl=53 time=103 ms
64 bytes from upload.wikimedia.beta.wmflabs.org (185.15.56.36): icmp_seq=8 ttl=53 time=103 ms

Related Objects
Search...

Status	Subtype	Assigned	Task
Open	Goal	None	T129368 Usages of unittest.expectedFailure or unittest.skip should be solved (goal)
Open	BUG REPORT	None	T431173 site_tests.TestProductionAndTestSite fails with TimeoutError
			Restricted Task
Open	BUG REPORT	None	T420833 Beta cluster is being crawled to death by bot traffic coming from residential proxies

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

• bd808 renamed this task from Beta cluster is slow as sludge / serves 503 and 504 to Beta cluster is being crawled to death by bot traffic coming from residential proxies.Mar 27 2026, 12:40 AM

In T420833#11752055, @bd808 wrote:

We need the https://wikitech.wikimedia.org/wiki/Requestctl stack in Beta.

I guess that’s T226688: Block web crawlers from accessing Cloud Services / T409759: Enable use of requestctl for filtering Toolforge and Cloud VPS web traffic?

In T420833#11757613, @LucasWerkmeister wrote:

In T420833#11752055, @bd808 wrote:

We need the https://wikitech.wikimedia.org/wiki/Requestctl stack in Beta.

I guess that’s T226688: Block web crawlers from accessing Cloud Services / T409759: Enable use of requestctl for filtering Toolforge and Cloud VPS web traffic?

Beta has it's own CDN that is disconnected from everyone else, but yes generally this.

PerfektesChaos mentioned this in T421654: Use HTTP Cookie to legitimate BETA cluster access.Mar 29 2026, 9:19 PM

Novem_Linguae subscribed.Apr 2 2026, 5:42 AM

SomeRandomDeveloper subscribed.Apr 2 2026, 10:53 AM

TheDJ subscribed.Apr 2 2026, 11:35 AM

Hakimi97 subscribed.Apr 2 2026, 11:54 AM

Johannnes89 subscribed.Apr 2 2026, 12:15 PM

Tenshi_Hinanawi subscribed.Apr 2 2026, 3:27 PM

Sjoerddebruin subscribed.Apr 2 2026, 3:37 PM

We are still seeing quite a bit of what is likely residential proxy traffic with 12,699 distinct /24 networks each sending 2 requests in the last ~17.5 hours (with 23,607 distinct /24 networks seen in total).

We are still actively blocking quite a bit of traffic at the CDN edge, but for now the more persistent attackers seem to have moved on allowing our servers to keep up with the request volume:

Screenshot 2026-04-03 at 11.27.54.png (2,092×1,366 px, 454 KB)

Let's block all of AS8075 Microsoft Corporation. A number of ranges from this ASN are currently showing as the most active /24 networks. I will be using the list of networks from https://github.com/ipverse/as-ip-blocks/blob/ba0a0415a3d35c44367fd347f5106f7dc101ed5f/as/8075/ipv4-aggregated.txt to do this.

https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/ad0c6b6ee5a8e201dd2f3fba777058e03e47e021%5E%21/#F0

diff --git a/deployment-prep/_.yaml b/deployment-prep/_.yaml
index 3c3eedf..423c57f 100644
--- a/deployment-prep/_.yaml
+++ b/deployment-prep/_.yaml

@@ -1,6 +1,7 @@
 abuse_networks:
   blocked_nets:
     networks:
+    - 1.186.0.0/16
     - 2.0.0.0/11
     - 2.32.0.0/12
     - 2.48.0.0/14
@@ -9,9 +10,28 @@
     - 2.56.0.0/13
     - 2.64.0.0/10
     - 2.128.0.0/9
+    - 2.58.103.0/24
     - 4.192.0.0/12
+    - 4.144.0.0/12
+    - 4.160.0.0/11
+    - 4.192.0.0/10
     - 5.0.0.0/8
+    - 5.23.34.0/24
+    - 5.102.37.0/24
+    - 9.141.0.0/16
+    - 9.145.0.0/16
+    - 9.146.0.0/16
+    - 9.160.0.0/16
+    - 9.163.0.0/16
+    - 9.169.0.0/16
+    - 9.205.0.0/16
+    - 9.223.0.0/16
+    - 9.234.0.0/15
     - 13.0.0.0/8
+    - 13.64.0.0/11
+    - 13.104.0.0/14
+    - 13.254.0.0/24
+    - 13.254.4.0/24
     - 14.0.0.0/9
     - 14.128.0.0/11
     - 14.160.0.0/12
@@ -35,6 +55,26 @@
     - 20.150.0.0/15
     - 20.152.0.0/15
     - 20.192.0.0/10
+    - 20.0.0.0/11
+    - 20.33.0.0/16
+    - 20.36.0.0/14
+    - 20.40.0.0/13
+    - 20.48.0.0/12
+    - 20.64.0.0/10
+    - 20.135.0.0/16
+    - 20.136.0.0/17
+    - 20.143.0.0/16
+    - 20.150.0.0/15
+    - 20.152.0.0/15
+    - 20.157.0.0/16
+    - 20.160.0.0/12
+    - 20.184.0.0/13
+    - 20.192.0.0/10
+    - 23.96.0.0/14
+    - 23.100.0.0/15
+    - 23.102.0.0/16
+    - 23.103.64.0/18
+    - 23.103.128.0/17
     - 24.0.0.0/8
     - 27.0.0.0/8
     - 31.0.0.0/12
@@ -45,6 +85,9 @@
     - 31.32.0.0/11
     - 31.64.0.0/10
     - 31.128.0.0/9
+    - 31.56.106.0/24
+    - 31.58.221.0/24
+    - 31.59.230.0/24
     - 37.0.0.0/10
     - 37.64.0.0/11
     - 37.96.0.0/12
@@ -78,6 +121,8 @@
     - 37.247.64.0/18
     - 37.247.128.0/17
     - 37.248.0.0/13
+    - 37.228.121.0/24
+    - 37.228.126.0/24
     - 38.128.0.0/10
     - 38.192.0.0/11
     - 38.224.0.0/12
@@ -94,6 +139,33 @@
     - 38.244.0.0/14
     - 38.248.0.0/13
     - 40.0.0.0/8
+    - 40.47.0.0/16
+    - 40.64.0.0/15
+    - 40.66.0.0/17
+    - 40.67.0.0/16
+    - 40.68.0.0/14
+    - 40.74.0.0/15
+    - 40.76.0.0/14
+    - 40.80.0.0/12
+    - 40.96.0.0/13
+    - 40.104.0.0/14
+    - 40.108.128.0/17
+    - 40.110.0.0/15
+    - 40.112.0.0/13
+    - 40.120.0.0/14
+    - 40.124.0.0/16
+    - 40.125.0.0/17
+    - 40.126.0.0/18
+    - 40.126.128.0/17
+    - 40.127.0.0/16
+    - 40.146.0.0/16
+    - 40.148.0.0/16
+    - 40.155.0.0/16
+    - 40.159.0.0/16
+    - 40.169.0.0/16
+    - 40.170.0.0/15
+    - 40.212.0.0/16
+    - 40.253.0.0/16
     - 41.0.0.0/10
     - 41.64.0.0/13
     - 41.72.0.0/15
@@ -136,6 +208,11 @@
     - 45.152.0.0/13
     - 45.160.0.0/11
     - 45.192.0.0/10
+    - 45.8.43.0/24
+    - 45.66.80.0/24
+    - 45.143.224.0/23
+    - 45.156.164.0/23
+    - 45.157.47.0/24
     - 46.0.0.0/12
     - 46.16.0.0/13
     - 46.24.0.0/14
@@ -175,6 +252,8 @@
     - 46.116.0.0/14
     - 46.120.0.0/13
     - 46.128.0.0/9
+    - 46.29.242.0/24
+    - 46.254.176.0/21
     - 47.0.0.0/9
     - 47.128.0.0/12
     - 47.160.0.0/12
@@ -197,7 +276,34 @@
     - 47.231.224.0/19
     - 47.232.0.0/13
     - 47.240.0.0/12
+    - 48.192.0.0/11
     - 49.0.0.0/8
+    - 50.85.0.0/16
+    - 50.117.58.0/24
+    - 50.117.87.0/24
+    - 51.1.0.0/16
+    - 51.4.0.0/15
+    - 51.8.0.0/16
+    - 51.10.0.0/15
+    - 51.12.0.0/15
+    - 51.42.0.0/16
+    - 51.51.0.0/16
+    - 51.53.0.0/16
+    - 51.56.0.0/14
+    - 51.103.0.0/16
+    - 51.104.0.0/15
+    - 51.107.0.0/16
+    - 51.109.0.0/16
+    - 51.111.0.0/16
+    - 51.116.0.0/16
+    - 51.120.0.0/16
+    - 51.124.0.0/16
+    - 51.126.0.0/16
+    - 51.132.0.0/16
+    - 51.136.0.0/15
+    - 51.138.0.0/16
+    - 51.140.0.0/14
+    - 51.144.0.0/15
     - 52.132.0.0/14
     - 52.136.0.0/13
     - 52.145.0.0/16
@@ -205,7 +311,40 @@
     - 52.148.0.0/14
     - 52.152.0.0/13
     - 52.160.0.0/11
+    - 52.96.0.0/12
+    - 52.112.0.0/14
+    - 52.120.0.0/14
+    - 52.125.0.0/16
+    - 52.136.0.0/13
+    - 52.146.0.0/15
+    - 52.148.0.0/14
+    - 52.152.0.0/13
+    - 52.160.0.0/11
+    - 52.224.0.0/11
     - 57.141.0.0/16
+    - 57.150.0.0/15
+    - 57.152.0.0/13
+    - 57.160.0.0/12
+    - 62.12.56.0/22
+    - 62.12.60.0/23
+    - 62.97.228.0/24
+    - 62.132.26.0/23
+    - 62.164.204.0/23
+    - 62.231.53.0/24
+    - 63.116.158.0/24
+    - 64.4.0.0/18
+    - 64.5.116.0/24
+    - 64.13.130.0/23
+    - 64.15.23.0/24
+    - 64.15.25.0/24
+    - 64.40.128.0/24
+    - 64.40.142.0/23
+    - 64.52.107.0/24
+    - 64.72.206.0/24
+    - 64.112.188.0/24
+    - 64.236.0.0/16
+    - 65.18.198.0/24
+    - 65.52.0.0/14
     - 66.0.0.0/10
     - 66.64.0.0/12
     - 66.80.0.0/16
@@ -224,7 +363,45 @@
     - 66.144.0.0/12
     - 66.160.0.0/11
     - 66.192.0.0/10
+    - 66.9.47.0/24
+    - 66.45.92.0/24
+    - 66.94.52.0/23
+    - 66.119.144.0/20
+    - 66.132.204.0/22
+    - 66.178.148.0/23
+    - 66.185.137.0/24
+    - 66.185.138.0/23
+    - 66.185.140.0/23
+    - 66.185.142.0/24
+    - 66.235.159.0/24
+    - 67.210.82.0/24
+    - 67.210.98.0/23
+    - 67.210.128.0/23
+    - 68.154.0.0/15
+    - 68.210.0.0/15
+    - 68.218.0.0/15
+    - 68.220.0.0/15
     - 69.25.57.151/32
+    - 69.15.0.0/16
+    - 69.52.192.0/23
+    - 69.52.198.0/23
+    - 69.52.200.0/24
+    - 69.52.204.0/23
+    - 69.52.206.0/24
+    - 69.59.17.0/24
+    - 69.72.71.0/24
+    - 69.84.180.0/23
+    - 69.89.60.0/24
+    - 69.89.63.0/24
+    - 70.35.156.0/24
+    - 70.37.0.0/17
+    - 70.37.128.0/18
+    - 70.152.0.0/15
+    - 70.156.0.0/15
+    - 72.16.128.0/17
+    - 72.54.0.0/16
+    - 72.144.0.0/14
+    - 72.152.0.0/14
     - 73.0.0.0/9
     - 73.128.0.0/11
     - 73.160.0.0/15
@@ -235,6 +412,17 @@
     - 73.192.0.0/10
     - 74.235.192.0/18
     - 74.243.224.0/19
+    - 74.7.0.0/16
+    - 74.116.254.0/23
+    - 74.117.19.0/24
+    - 74.144.0.0/12
+    - 74.160.0.0/14
+    - 74.176.0.0/14
+    - 74.200.130.0/24
+    - 74.224.0.0/14
+    - 74.234.0.0/15
+    - 74.240.0.0/14
+    - 74.248.0.0/15
     - 76.0.0.0/8
     - 78.0.0.0/12
     - 78.16.0.0/14
@@ -243,6 +431,9 @@
     - 78.64.0.0/10
     - 78.128.0.0/9
     - 79.0.0.0/8
+    - 79.170.98.0/24
+    - 80.247.56.0/24
+    - 81.22.128.0/24
     - 82.0.0.0/10
     - 82.68.0.0/14
     - 82.72.0.0/13
@@ -255,7 +446,12 @@
     - 82.218.0.0/15
     - 82.220.0.0/14
     - 82.224.0.0/11
+    - 82.87.0.0/16
+    - 82.171.0.0/16
+    - 83.218.234.0/24
     - 84.0.0.0/8
+    - 84.81.0.0/16
+    - 84.222.0.0/15
     - 85.0.0.0/10
     - 85.64.0.0/12
     - 85.80.0.0/13
@@ -281,7 +477,11 @@
     - 85.152.0.0/13
     - 85.160.0.0/11
     - 85.192.0.0/10
+    - 85.204.120.0/23
+    - 85.210.0.0/15
+    - 85.212.0.0/16
     - 86.0.0.0/8
+    - 86.91.0.0/16
     - 88.0.0.0/8
     - 89.0.0.0/13
     - 89.8.0.0/14
@@ -301,6 +501,7 @@
     - 89.242.0.0/15
     - 89.244.0.0/14
     - 89.248.0.0/13
+    - 89.28.187.0/24
     - 90.0.0.0/10
     - 90.64.0.0/12
     - 90.80.0.0/13
@@ -320,9 +521,19 @@
     - 91.183.0.0/16
     - 91.184.0.0/13
     - 91.192.0.0/10
+    - 91.203.230.0/24
+    - 91.216.184.0/24
+    - 91.234.239.0/24
+    - 91.242.222.0/24
+    - 91.244.201.0/24
+    - 91.245.215.0/24
     - 92.0.0.0/8
+    - 92.112.9.0/24
+    - 92.118.22.0/23
     - 93.0.0.0/9
     - 93.128.0.0/10
+    - 93.94.32.0/21
+    - 93.174.64.0/21
     - 94.0.0.0/14
     - 94.8.0.0/13
     - 94.16.0.0/13
@@ -362,12 +573,21 @@
     - 94.235.0.0/16
     - 94.236.0.0/14
     - 94.240.0.0/12
+    - 94.140.16.0/24
+    - 94.143.105.0/24
+    - 94.143.106.0/23
+    - 94.143.108.0/23
+    - 94.143.110.0/24
+    - 94.245.64.0/18
     - 95.0.0.0/10
     - 95.64.0.0/12
     - 95.80.0.0/13
     - 95.92.0.0/14
     - 95.96.0.0/11
     - 95.128.0.0/9
+    - 98.64.0.0/14
+    - 98.70.0.0/15
+    - 98.142.122.0/23
     - 101.0.0.0/8
     - 102.0.0.0/9
     - 102.128.0.0/11
@@ -377,6 +597,10 @@
     - 102.183.0.0/16
     - 102.184.0.0/13
     - 102.192.0.0/10
+    - 102.37.0.0/16
+    - 102.133.0.0/16
+    - 102.211.187.0/24
+    - 102.223.214.0/24
     - 103.0.0.0/10
     - 103.64.0.0/11
     - 103.96.0.0/15
@@ -420,6 +644,25 @@
     - 103.200.0.0/13
     - 103.208.0.0/12
     - 103.224.0.0/11
+    - 103.8.80.0/24
+    - 103.36.129.0/24
+    - 103.58.119.0/24
+    - 103.61.61.0/24
+    - 103.61.62.0/23
+    - 103.103.44.0/22
+    - 103.131.148.0/23
+    - 103.164.237.0/24
+    - 103.204.21.0/24
+    - 103.204.22.0/24
+    - 103.249.62.0/23
+    - 103.254.7.0/24
+    - 104.40.0.0/13
+    - 104.146.128.0/17
+    - 104.193.12.0/23
+    - 104.193.126.0/24
+    - 104.204.254.0/24
+    - 104.208.0.0/13
+    - 104.237.40.0/24
     - 105.0.0.0/9
     - 105.128.0.0/11
     - 105.160.0.0/15
@@ -436,12 +679,20 @@
     - 105.168.0.0/13
     - 105.176.0.0/12
     - 105.192.0.0/10
+    - 108.140.0.0/14
+    - 108.161.110.0/24
     - 109.0.0.0/9
     - 109.128.0.0/12
     - 109.160.0.0/11
     - 109.192.0.0/10
+    - 109.246.0.0/16
     - 110.0.0.0/8
     - 111.0.0.0/8
+    - 111.221.16.0/21
+    - 111.221.24.0/22
+    - 111.221.29.0/24
+    - 111.221.30.0/23
+    - 111.221.64.0/18
     - 113.0.0.0/12
     - 113.16.0.0/14
     - 113.20.0.0/15
@@ -458,6 +709,9 @@
     - 113.32.0.0/11
     - 113.64.0.0/10
     - 113.128.0.0/9
+    - 113.197.64.0/22
+    - 115.42.47.0/24
+    - 117.121.247.0/24
     - 119.0.0.0/8
     - 122.0.0.0/9
     - 122.128.0.0/11
@@ -469,6 +723,7 @@
     - 122.187.128.0/17
     - 122.188.0.0/14
     - 122.192.0.0/10
+    - 122.149.0.0/16
     - 123.0.0.0/8
     - 124.0.0.0/9
     - 124.128.0.0/11
@@ -483,9 +738,94 @@
     - 124.172.0.0/14
     - 124.176.0.0/12
     - 124.192.0.0/10
+    - 124.252.0.0/16
     - 128.241.0.0/16
+    - 128.24.0.0/16
+    - 128.36.15.0/24
+    - 128.85.0.0/16
+    - 128.94.0.0/16
+    - 128.127.76.0/24
+    - 128.127.79.0/24
+    - 128.203.0.0/16
+    - 128.251.0.0/16
+    - 129.35.19.0/24
+    - 129.135.0.0/16
+    - 130.17.64.0/22
+    - 130.17.73.0/24
+    - 130.17.80.0/23
+    - 130.17.84.0/24
+    - 130.17.86.0/24
+    - 130.17.88.0/22
+    - 130.17.94.0/24
+    - 130.33.0.0/16
+    - 130.107.0.0/16
+    - 130.115.248.0/21
+    - 130.131.0.0/16
+    - 130.213.0.0/16
     - 131.0.0.0/8
+    - 131.145.0.0/16
+    - 131.163.0.0/16
+    - 131.176.228.0/22
+    - 131.189.0.0/16
+    - 131.253.1.0/24
+    - 131.253.5.0/24
+    - 131.253.6.0/24
+    - 131.253.8.0/24
+    - 131.253.12.0/22
+    - 131.253.24.0/21
+    - 131.253.32.0/20
+    - 131.253.61.0/24
+    - 131.253.62.0/23
+    - 131.253.128.0/17
+    - 132.164.0.0/16
+    - 132.196.0.0/16
+    - 132.220.0.0/16
+    - 132.245.0.0/16
+    - 134.33.0.0/16
+    - 134.112.0.0/16
+    - 134.138.0.0/16
+    - 134.149.0.0/16
+    - 134.170.0.0/16
+    - 134.217.7.0/24
+    - 135.4.0.0/16
+    - 135.6.0.0/15
+    - 135.13.0.0/16
+    - 135.18.0.0/16
+    - 135.56.0.0/24
+    - 135.56.2.0/24
+    - 135.56.4.0/23
+    - 135.56.6.0/24
+    - 135.56.8.0/24
+    - 135.85.0.0/16
+    - 135.86.0.0/16
+    - 135.88.0.0/16
+    - 135.93.0.0/16
+    - 135.116.0.0/16
+    - 135.119.0.0/16
+    - 135.130.0.0/16
+    - 135.149.0.0/16
+    - 135.171.0.0/16
+    - 135.183.0.0/16
+    - 135.185.0.0/16
+    - 135.220.0.0/15
+    - 135.222.0.0/16
+    - 135.224.0.0/15
+    - 135.226.0.0/16
+    - 135.228.0.0/15
+    - 135.230.0.0/16
+    - 135.232.0.0/14
+    - 135.236.0.0/15
+    - 135.240.0.0/15
+    - 135.244.0.0/16
+    - 135.246.0.0/15
+    - 135.254.0.0/15
     - 136.243.220.0/24
+    - 137.70.152.0/24
+    - 137.70.154.0/24
+    - 137.116.0.0/15
+    - 137.132.48.0/23
+    - 137.135.0.0/16
+    - 137.162.0.0/16
     - 138.0.0.0/9
     - 138.128.0.0/10
     - 138.192.0.0/13
@@ -494,6 +834,27 @@
     - 138.204.0.0/14
     - 138.208.0.0/12
     - 138.224.0.0/11
+    - 138.32.108.0/22
+    - 138.32.112.0/24
+    - 138.32.114.0/24
+    - 138.91.0.0/16
+    - 138.105.0.0/16
+    - 138.128.250.0/23
+    - 138.213.0.0/16
+    - 138.239.0.0/16
+    - 139.188.0.0/16
+    - 140.150.226.0/23
+    - 141.11.196.0/24
+    - 141.59.240.0/21
+    - 141.193.248.0/24
+    - 141.206.200.0/24
+    - 142.0.188.0/23
+    - 142.55.22.0/24
+    - 142.147.54.0/24
+    - 142.147.61.0/24
+    - 142.225.36.0/23
+    - 142.229.194.0/23
+    - 142.249.95.0/24
     - 143.0.0.0/11
     - 143.32.0.0/12
     - 143.48.0.0/13
@@ -509,8 +870,56 @@
     - 143.180.0.0/14
     - 143.184.0.0/13
     - 143.192.0.0/10
+    - 143.188.148.0/22
+    - 143.188.152.0/21
+    - 143.188.160.0/23
+    - 143.209.0.0/16
+    - 143.226.0.0/16
+    - 143.241.0.0/16
+    - 144.77.160.0/23
+    - 144.77.162.0/24
+    - 144.203.152.0/23
+    - 145.46.160.0/22
+    - 145.46.164.0/23
+    - 145.69.64.0/20
+    - 145.83.4.0/22
+    - 145.129.0.0/16
+    - 145.130.0.0/16
+    - 145.132.0.0/15
+    - 145.190.0.0/15
+    - 145.219.18.0/23
+    - 145.219.20.0/22
+    - 145.219.24.0/21
+    - 145.219.32.0/23
     - 146.174.160.0/19
+    - 146.116.249.0/24
+    - 146.225.0.0/21
+    - 147.79.26.0/24
+    - 147.140.72.0/23
+    - 147.140.74.0/24
+    - 147.145.0.0/16
+    - 147.160.48.0/24
+    - 147.214.0.0/16
+    - 147.243.0.0/16
+    - 148.7.0.0/16
+    - 148.53.0.0/16
+    - 148.59.127.0/24
+    - 149.1.0.0/16
+    - 149.175.0.0/16
+    - 149.198.0.0/16
+    - 149.204.0.0/16
     - 150.0.0.0/8
+    - 150.171.0.0/16
+    - 150.212.0.0/16
+    - 151.129.0.0/16
+    - 151.206.0.0/16
+    - 151.241.36.0/23
+    - 151.242.202.0/23
+    - 151.242.205.0/24
+    - 151.243.36.0/24
+    - 151.243.47.0/24
+    - 151.244.14.0/24
+    - 151.244.66.0/23
     - 152.0.0.0/16
     - 152.2.0.0/15
     - 152.4.0.0/15
@@ -529,6 +938,40 @@
     - 152.60.0.0/14
     - 152.64.0.0/10
     - 152.128.0.0/9
+    - 152.38.156.0/24
+    - 152.138.0.0/16
+    - 153.65.20.0/24
+    - 153.65.71.0/24
+    - 155.55.192.0/22
+    - 155.60.224.0/23
+    - 155.60.226.0/24
+    - 155.60.232.0/23
+    - 155.60.234.0/24
+    - 155.60.240.0/23
+    - 155.60.242.0/24
+    - 155.62.0.0/16
+    - 155.103.184.0/24
+    - 155.231.208.0/23
+    - 156.20.22.0/23
+    - 156.23.0.0/16
+    - 156.29.16.0/24
+    - 157.31.0.0/16
+    - 157.55.0.0/16
+    - 157.56.0.0/16
+    - 157.81.0.0/16
+    - 157.95.0.0/16
+    - 157.172.0.0/16
+    - 157.176.0.0/16
+    - 157.207.4.0/23
+    - 157.252.0.0/16
+    - 158.23.0.0/16
+    - 158.24.0.0/16
+    - 158.53.0.0/16
+    - 158.58.136.0/21
+    - 158.86.31.0/24
+    - 158.86.95.0/24
+    - 158.116.129.0/24
+    - 158.158.0.0/16
     - 159.0.0.0/9
     - 159.128.0.0/10
     - 159.192.0.0/14
@@ -545,8 +988,47 @@
     - 159.200.0.0/13
     - 159.208.0.0/12
     - 159.224.0.0/11
+    - 159.14.174.0/24
+    - 159.14.176.0/24
+    - 159.105.18.0/24
+    - 159.128.0.0/16
+    - 159.222.162.0/24
     - 160.0.0.0/8
+    - 160.4.0.0/16
+    - 160.92.192.0/24
+    - 160.187.9.0/24
+    - 160.207.0.0/16
+    - 160.234.0.0/16
+    - 161.131.104.0/24
+    - 161.131.217.0/24
+    - 161.157.0.0/16
+    - 161.195.96.0/21
+    - 161.195.152.0/24
+    - 161.195.155.0/24
+    - 161.195.188.0/23
+    - 161.195.248.0/24
+    - 161.220.0.0/16
+    - 161.221.0.0/22
+    - 161.221.4.0/23
+    - 161.221.6.0/24
     - 162.0.0.0/8
+    - 162.131.1.0/24
+    - 162.131.2.0/24
+    - 162.253.106.0/24
+    - 162.254.245.0/24
+    - 162.254.247.0/24
+    - 163.57.0.0/16
+    - 163.61.89.0/24
+    - 163.199.84.0/23
+    - 164.47.80.0/23
+    - 164.75.128.0/22
+    - 164.75.132.0/23
+    - 165.15.0.0/16
+    - 165.17.0.0/16
+    - 165.99.141.0/24
+    - 165.125.48.0/21
+    - 165.130.253.0/24
+    - 165.224.64.0/22
     - 166.0.0.0/11
     - 166.32.0.0/12
     - 166.48.0.0/17
@@ -559,7 +1041,17 @@
     - 166.56.0.0/13
     - 166.64.0.0/10
     - 166.128.0.0/9
+    - 166.98.6.0/23
     - 167.0.0.0/8
+    - 167.21.32.0/21
+    - 167.21.152.0/21
+    - 167.92.211.0/24
+    - 167.92.212.0/24
+    - 167.105.0.0/16
+    - 167.162.0.0/16
+    - 167.186.0.0/16
+    - 167.231.0.0/16
+    - 167.253.12.0/22
     - 168.0.0.0/9
     - 168.128.0.0/11
     - 168.160.0.0/12
@@ -576,10 +1068,37 @@
     - 168.186.0.0/15
     - 168.188.0.0/14
     - 168.192.0.0/10
+    - 168.61.0.0/16
+    - 168.62.0.0/15
+    - 168.98.112.0/21
+    - 168.137.64.0/20
     - 169.224.0.0/16
+    - 169.138.0.0/16
     - 170.0.0.0/8
+    - 170.22.154.0/23
+    - 170.22.156.0/23
+    - 170.39.84.0/23
+    - 170.39.86.0/24
+    - 170.62.37.0/24
+    - 170.75.176.0/24
+    - 170.110.250.0/24
+    - 170.114.56.0/23
+    - 170.165.0.0/16
+    - 170.170.120.0/22
+    - 170.170.127.0/24
+    - 170.176.133.0/24
+    - 170.176.140.0/24
+    - 170.176.147.0/24
+    - 170.176.205.0/24
     - 172.0.0.0/8
+    - 172.128.0.0/10
+    - 172.192.0.0/12
+    - 172.208.0.0/13
+    - 173.200.0.0/16
+    - 173.241.37.0/24
+    - 173.241.44.0/24
     - 176.0.0.0/8
+    - 176.116.123.0/24
     - 177.0.0.0/10
     - 177.64.0.0/11
     - 177.96.0.0/14
@@ -587,6 +1106,10 @@
     - 177.104.0.0/13
     - 177.112.0.0/12
     - 177.128.0.0/9
+    - 178.93.137.0/24
+    - 178.93.138.0/24
+    - 178.251.21.0/24
+    - 178.255.242.0/24
     - 179.0.0.0/9
     - 179.128.0.0/15
     - 179.130.0.0/23
@@ -679,6 +1202,39 @@
     - 185.234.0.0/15
     - 185.236.0.0/14
     - 185.240.0.0/12
+    - 185.12.10.0/24
+    - 185.34.84.0/22
+    - 185.34.250.0/24
+    - 185.42.70.0/24
+    - 185.45.122.0/24
+    - 185.52.203.0/24
+    - 185.70.160.0/23
+    - 185.76.37.0/24
+    - 185.76.182.0/23
+    - 185.87.134.0/24
+    - 185.89.200.0/22
+    - 185.90.154.0/24
+    - 185.113.189.0/24
+    - 185.115.155.0/24
+    - 185.116.168.0/23
+    - 185.117.95.0/24
+    - 185.117.183.0/24
+    - 185.117.230.0/24
+    - 185.135.58.0/24
+    - 185.149.54.0/24
+    - 185.154.80.0/22
+    - 185.157.16.0/22
+    - 185.187.104.0/23
+    - 185.187.123.0/24
+    - 185.192.184.0/23
+    - 185.195.244.0/23
+    - 185.209.208.0/23
+    - 185.211.208.0/24
+    - 185.222.140.0/22
+    - 185.236.221.0/24
+    - 185.238.165.0/24
+    - 185.242.137.0/24
+    - 185.242.139.0/24
     - 186.0.0.0/11
     - 186.32.0.0/13
     - 186.40.0.0/17
@@ -742,6 +1298,8 @@
     - 188.253.220.0/22
     - 188.253.224.0/19
     - 188.254.0.0/15
+    - 188.94.190.0/24
+    - 188.255.174.0/23
     - 189.0.0.0/9
     - 189.128.0.0/10
     - 189.192.0.0/11
@@ -810,7 +1368,71 @@
     - 191.180.0.0/14
     - 191.184.0.0/13
     - 191.192.0.0/10
+    - 191.232.0.0/13
     - 192.0.0.0/8
+    - 192.0.58.0/23
+    - 192.28.24.0/23
+    - 192.34.34.0/24
+    - 192.40.76.0/22
+    - 192.48.225.0/24
+    - 192.64.201.0/24
+    - 192.71.41.0/24
+    - 192.80.239.0/24
+    - 192.80.246.0/23
+    - 192.84.160.0/23
+    - 192.94.1.0/24
+    - 192.100.104.0/21
+    - 192.100.112.0/20
+    - 192.100.128.0/22
+    - 192.102.165.0/24
+    - 192.131.254.0/24
+    - 192.132.27.0/24
+    - 192.133.85.0/24
+    - 192.160.145.0/24
+    - 192.160.194.0/24
+    - 192.195.98.0/24
+    - 192.197.157.0/24
+    - 192.243.86.0/23
+    - 193.8.43.0/24
+    - 193.17.104.0/23
+    - 193.17.106.0/24
+    - 193.17.108.0/23
+    - 193.17.110.0/24
+    - 193.17.112.0/23
+    - 193.17.116.0/23
+    - 193.17.120.0/23
+    - 193.17.124.0/23
+    - 193.29.158.0/24
+    - 193.53.155.0/24
+    - 193.134.96.0/24
+    - 193.135.41.0/24
+    - 193.149.64.0/19
+    - 193.187.72.0/24
+    - 193.189.114.0/23
+    - 193.221.113.0/24
+    - 194.33.109.0/24
+    - 194.41.19.0/24
+    - 194.44.235.0/24
+    - 194.45.24.0/23
+    - 194.49.206.0/24
+    - 194.50.21.0/24
+    - 194.50.22.0/23
+    - 194.61.106.0/24
+    - 194.76.102.0/24
+    - 194.110.197.0/24
+    - 194.113.230.0/24
+    - 194.150.192.0/24
+    - 194.180.131.0/24
+    - 195.8.43.0/24
+    - 195.43.44.0/24
+    - 195.43.46.0/24
+    - 195.85.113.0/24
+    - 195.88.21.0/24
+    - 195.104.228.0/22
+    - 195.105.26.0/24
+    - 195.114.140.0/24
+    - 195.191.211.0/24
+    - 195.222.105.0/24
     - 196.0.0.0/8
     - 197.0.0.0/13
     - 197.8.0.0/14
@@ -838,6 +1460,55 @@
     - 197.234.0.0/15
     - 197.236.0.0/14
     - 197.240.0.0/12
+    - 198.8.73.0/24
+    - 198.22.19.0/24
+    - 198.47.13.0/24
+    - 198.49.8.0/24
+    - 198.49.178.0/24
+    - 198.51.0.0/24
+    - 198.52.0.0/24
+    - 198.91.39.0/24
+    - 198.99.146.0/24
+    - 198.167.255.0/24
+    - 198.185.5.0/24
+    - 198.186.138.0/24
+    - 198.186.172.0/24
+    - 198.200.130.0/24
+    - 198.202.23.0/24
+    - 198.202.27.0/24
+    - 198.206.164.0/24
+    - 198.245.174.0/24
+    - 198.252.211.0/24
+    - 199.0.185.0/24
+    - 199.21.128.0/24
+    - 199.26.109.0/24
+    - 199.27.231.0/24
+    - 199.30.16.0/20
+    - 199.34.122.0/24
+    - 199.38.175.0/24
+    - 199.50.0.0/16
+    - 199.60.28.0/24
+    - 199.65.27.0/24
+    - 199.65.29.0/24
+    - 199.65.30.0/23
+    - 199.65.40.0/24
+    - 199.65.243.0/24
+    - 199.65.247.0/24
+    - 199.65.251.0/24
+    - 199.103.90.0/23
+    - 199.103.122.0/24
+    - 199.118.0.0/16
+    - 199.181.32.0/24
+    - 199.181.34.0/24
+    - 199.181.39.0/24
+    - 199.189.36.0/23
+    - 199.189.38.0/24
+    - 199.229.48.0/21
+    - 199.233.162.0/24
+    - 199.242.32.0/20
+    - 199.242.48.0/21
+    - 199.247.205.0/24
+    - 199.248.124.0/24
     - 200.0.0.0/8
     - 201.0.0.0/9
     - 201.128.0.0/13
@@ -854,9 +1525,121 @@
     - 201.160.0.0/11
     - 201.192.0.0/10
     - 202.0.0.0/8
+    - 202.12.70.0/24
+    - 202.14.81.0/24
+    - 202.22.173.0/24
+    - 202.27.49.0/24
+    - 202.89.224.0/21
+    - 202.90.34.0/23
+    - 202.130.203.0/24
+    - 202.155.124.0/22
+    - 203.2.220.0/22
+    - 203.10.24.0/23
+    - 203.10.27.0/24
+    - 203.10.28.0/22
+    - 203.17.220.0/23
+    - 203.20.128.0/23
+    - 203.21.152.0/23
+    - 203.28.214.0/23
+    - 203.32.8.0/24
+    - 203.32.10.0/23
+    - 203.84.134.0/23
+    - 204.14.180.0/22
+    - 204.76.110.0/23
+    - 204.79.135.0/24
+    - 204.79.179.0/24
+    - 204.79.195.0/24
+    - 204.79.252.0/24
+    - 204.80.33.0/24
+    - 204.80.58.0/24
+    - 204.87.58.0/24
+    - 204.95.96.0/20
+    - 204.152.95.0/24
+    - 204.152.140.0/23
+    - 204.154.190.0/24
+    - 204.194.56.0/22
+    - 204.209.219.0/24
+    - 204.239.11.0/24
+    - 204.239.54.0/24
+    - 205.132.192.0/23
+    - 205.135.211.0/24
+    - 205.135.212.0/23
+    - 205.143.44.0/23
+    - 206.109.76.0/24
+    - 206.130.81.0/24
+    - 206.138.168.0/21
+    - 206.191.224.0/19
+    - 206.195.220.0/24
+    - 206.203.88.0/22
+    - 206.203.92.0/23
+    - 206.203.123.0/24
+    - 206.251.32.0/23
+    - 207.46.0.0/19
+    - 207.46.36.0/22
+    - 207.46.40.0/21
+    - 207.46.48.0/20
+    - 207.46.64.0/18
+    - 207.46.128.0/17
+    - 207.68.128.0/18
+    - 207.103.0.0/16
+    - 207.174.51.0/24
+    - 208.56.2.0/24
+    - 208.64.87.0/24
+    - 208.66.231.0/24
+    - 208.68.136.0/21
+    - 208.74.229.0/24
+    - 208.74.231.0/24
+    - 208.76.45.0/24
+    - 208.76.46.0/24
+    - 208.80.20.0/23
+    - 208.80.134.0/23
+    - 208.83.106.0/24
+    - 208.84.0.0/21
+    - 208.88.70.0/24
+    - 208.90.118.0/24
+    - 208.90.236.0/23
+    - 208.91.245.0/24
+    - 208.103.174.0/24
+    - 208.122.60.0/24
+    - 209.51.100.0/24
+    - 209.94.81.0/24
+    - 209.127.190.0/24
+    - 209.143.212.0/23
+    - 209.151.121.0/24
+    - 209.199.0.0/16
+    - 209.240.192.0/19
+    - 209.251.248.0/24
     - 212.237.120.0/22
+    - 212.1.218.0/23
+    - 212.1.222.0/23
+    - 212.46.57.0/24
+    - 212.103.45.0/24
+    - 212.207.0.0/16
+    - 213.54.0.0/16
+    - 213.109.150.0/24
+    - 213.156.248.0/23
+    - 213.199.128.0/18
+    - 213.218.36.0/23
+    - 213.218.48.0/22
     - 216.0.0.0/8
+    - 216.9.196.0/24
+    - 216.32.180.0/22
+    - 216.73.183.0/24
+    - 216.83.54.0/23
+    - 216.99.143.0/24
+    - 216.126.214.0/24
+    - 216.183.110.0/23
+    - 216.220.203.0/24
+    - 216.220.204.0/24
+    - 216.220.208.0/20
+    - 216.238.128.0/24
+    - 216.238.135.0/24
+    - 216.238.141.0/24
     - 217.147.172.0/24
+    - 217.11.173.0/24
+    - 217.71.189.0/24
+    - 217.169.202.0/24
+    - 217.176.0.0/16
 acmechief_host: deployment-acme-chief05.deployment-prep.eqiad1.wikimedia.cloud
 apt::use_experimental: true
 aptly::group: wikidev

The sorting there is not ideal. Let's fix that too.
https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/15f9af81e4fa587e0cb9a06d03cc91230fd5f634%5E%21/#F0

diff --git a/deployment-prep/_.yaml b/deployment-prep/_.yaml
index 423c57f..1c5c6f9 100644
--- a/deployment-prep/_.yaml
+++ b/deployment-prep/_.yaml

@@ -8,13 +8,13 @@
     - 2.52.0.0/15
     - 2.55.0.0/16
     - 2.56.0.0/13
+    - 2.58.103.0/24
     - 2.64.0.0/10
     - 2.128.0.0/9
-    - 2.58.103.0/24
-    - 4.192.0.0/12
     - 4.144.0.0/12
     - 4.160.0.0/11
     - 4.192.0.0/10
+    - 4.192.0.0/12
     - 5.0.0.0/8
     - 5.23.34.0/24
     - 5.102.37.0/24
@@ -45,31 +45,31 @@
     - 14.191.224.0/19
     - 14.192.0.0/10
     - 17.0.0.0/8
+    - 20.0.0.0/11
+    - 20.33.0.0/16
     - 20.33.0.0/16
     - 20.34.0.0/15
     - 20.36.0.0/14
-    - 20.40.0.0/13
-    - 20.48.0.0/12
-    - 20.64.0.0/10
-    - 20.128.0.0/16
-    - 20.150.0.0/15
-    - 20.152.0.0/15
-    - 20.192.0.0/10
-    - 20.0.0.0/11
-    - 20.33.0.0/16
     - 20.36.0.0/14
     - 20.40.0.0/13
+    - 20.40.0.0/13
+    - 20.48.0.0/12
     - 20.48.0.0/12
     - 20.64.0.0/10
+    - 20.64.0.0/10
+    - 20.128.0.0/16
     - 20.135.0.0/16
     - 20.136.0.0/17
     - 20.143.0.0/16
     - 20.150.0.0/15
+    - 20.150.0.0/15
+    - 20.152.0.0/15
     - 20.152.0.0/15
     - 20.157.0.0/16
     - 20.160.0.0/12
     - 20.184.0.0/13
     - 20.192.0.0/10
+    - 20.192.0.0/10
     - 23.96.0.0/14
     - 23.100.0.0/15
     - 23.102.0.0/16
@@ -83,11 +83,11 @@
     - 31.20.0.0/14
     - 31.24.0.0/13
     - 31.32.0.0/11
-    - 31.64.0.0/10
-    - 31.128.0.0/9
     - 31.56.106.0/24
     - 31.58.221.0/24
     - 31.59.230.0/24
+    - 31.64.0.0/10
+    - 31.128.0.0/9
     - 37.0.0.0/10
     - 37.64.0.0/11
     - 37.96.0.0/12
@@ -110,6 +110,8 @@
     - 37.190.0.0/15
     - 37.192.0.0/11
     - 37.224.0.0/12
+    - 37.228.121.0/24
+    - 37.228.126.0/24
     - 37.240.0.0/14
     - 37.244.0.0/17
     - 37.245.0.0/16
@@ -121,8 +123,6 @@
     - 37.247.64.0/18
     - 37.247.128.0/17
     - 37.248.0.0/13
-    - 37.228.121.0/24
-    - 37.228.126.0/24
     - 38.128.0.0/10
     - 38.192.0.0/11
     - 38.224.0.0/12
@@ -183,6 +183,7 @@
     - 41.128.0.0/9
     - 43.0.0.0/8
     - 45.0.0.0/12
+    - 45.8.43.0/24
     - 45.32.0.0/12
     - 45.48.0.0/13
     - 45.56.0.0/14
@@ -194,7 +195,9 @@
     - 45.62.128.0/17
     - 45.63.0.0/16
     - 45.64.0.0/10
+    - 45.66.80.0/24
     - 45.128.0.0/12
+    - 45.143.224.0/23
     - 45.144.0.0/14
     - 45.148.0.0/17
     - 45.148.128.0/19
@@ -206,13 +209,10 @@
     - 45.149.0.0/16
     - 45.150.0.0/15
     - 45.152.0.0/13
-    - 45.160.0.0/11
-    - 45.192.0.0/10
-    - 45.8.43.0/24
-    - 45.66.80.0/24
-    - 45.143.224.0/23
     - 45.156.164.0/23
     - 45.157.47.0/24
+    - 45.160.0.0/11
+    - 45.192.0.0/10
     - 46.0.0.0/12
     - 46.16.0.0/13
     - 46.24.0.0/14
@@ -222,6 +222,7 @@
     - 46.28.96.0/19
     - 46.28.128.0/17
     - 46.29.0.0/16
+    - 46.29.242.0/24
     - 46.30.0.0/15
     - 46.32.0.0/12
     - 46.48.0.0/13
@@ -252,7 +253,6 @@
     - 46.116.0.0/14
     - 46.120.0.0/13
     - 46.128.0.0/9
-    - 46.29.242.0/24
     - 46.254.176.0/21
     - 47.0.0.0/9
     - 47.128.0.0/12
@@ -304,21 +304,21 @@
     - 51.138.0.0/16
     - 51.140.0.0/14
     - 51.144.0.0/15
-    - 52.132.0.0/14
-    - 52.136.0.0/13
-    - 52.145.0.0/16
-    - 52.146.0.0/15
-    - 52.148.0.0/14
-    - 52.152.0.0/13
-    - 52.160.0.0/11
     - 52.96.0.0/12
     - 52.112.0.0/14
     - 52.120.0.0/14
     - 52.125.0.0/16
+    - 52.132.0.0/14
     - 52.136.0.0/13
+    - 52.136.0.0/13
+    - 52.145.0.0/16
+    - 52.146.0.0/15
     - 52.146.0.0/15
     - 52.148.0.0/14
+    - 52.148.0.0/14
     - 52.152.0.0/13
+    - 52.152.0.0/13
+    - 52.160.0.0/11
     - 52.160.0.0/11
     - 52.224.0.0/11
     - 57.141.0.0/16
@@ -346,6 +346,8 @@
     - 65.18.198.0/24
     - 65.52.0.0/14
     - 66.0.0.0/10
+    - 66.9.47.0/24
+    - 66.45.92.0/24
     - 66.64.0.0/12
     - 66.80.0.0/16
     - 66.81.0.0/17
@@ -356,23 +358,21 @@
     - 66.82.0.0/15
     - 66.84.0.0/14
     - 66.88.0.0/13
+    - 66.94.52.0/23
     - 66.96.0.0/11
+    - 66.119.144.0/20
     - 66.128.0.0/15
     - 66.132.0.0/14
+    - 66.132.204.0/22
     - 66.136.0.0/13
     - 66.144.0.0/12
     - 66.160.0.0/11
-    - 66.192.0.0/10
-    - 66.9.47.0/24
-    - 66.45.92.0/24
-    - 66.94.52.0/23
-    - 66.119.144.0/20
-    - 66.132.204.0/22
     - 66.178.148.0/23
     - 66.185.137.0/24
     - 66.185.138.0/23
     - 66.185.140.0/23
     - 66.185.142.0/24
+    - 66.192.0.0/10
     - 66.235.159.0/24
     - 67.210.82.0/24
     - 67.210.98.0/23
@@ -381,8 +381,8 @@
     - 68.210.0.0/15
     - 68.218.0.0/15
     - 68.220.0.0/15
-    - 69.25.57.151/32
     - 69.15.0.0/16
+    - 69.25.57.151/32
     - 69.52.192.0/23
     - 69.52.198.0/23
     - 69.52.200.0/24
@@ -410,8 +410,6 @@
     - 73.168.0.0/13
     - 73.176.0.0/12
     - 73.192.0.0/10
-    - 74.235.192.0/18
-    - 74.243.224.0/19
     - 74.7.0.0/16
     - 74.116.254.0/23
     - 74.117.19.0/24
@@ -421,7 +419,9 @@
     - 74.200.130.0/24
     - 74.224.0.0/14
     - 74.234.0.0/15
+    - 74.235.192.0/18
     - 74.240.0.0/14
+    - 74.243.224.0/19
     - 74.248.0.0/15
     - 76.0.0.0/8
     - 78.0.0.0/12
@@ -438,16 +438,16 @@
     - 82.68.0.0/14
     - 82.72.0.0/13
     - 82.80.0.0/12
+    - 82.87.0.0/16
     - 82.96.0.0/11
     - 82.128.0.0/10
+    - 82.171.0.0/16
     - 82.192.0.0/12
     - 82.208.0.0/13
     - 82.217.0.0/16
     - 82.218.0.0/15
     - 82.220.0.0/14
     - 82.224.0.0/11
-    - 82.87.0.0/16
-    - 82.171.0.0/16
     - 83.218.234.0/24
     - 84.0.0.0/8
     - 84.81.0.0/16
@@ -486,6 +486,7 @@
     - 89.0.0.0/13
     - 89.8.0.0/14
     - 89.16.0.0/12
+    - 89.28.187.0/24
     - 89.32.0.0/11
     - 89.64.0.0/12
     - 89.96.0.0/11
@@ -501,7 +502,6 @@
     - 89.242.0.0/15
     - 89.244.0.0/14
     - 89.248.0.0/13
-    - 89.28.187.0/24
     - 90.0.0.0/10
     - 90.64.0.0/12
     - 90.80.0.0/13
@@ -531,8 +531,8 @@
     - 92.112.9.0/24
     - 92.118.22.0/23
     - 93.0.0.0/9
-    - 93.128.0.0/10
     - 93.94.32.0/21
+    - 93.128.0.0/10
     - 93.174.64.0/21
     - 94.0.0.0/14
     - 94.8.0.0/13
@@ -559,12 +559,17 @@
     - 94.139.64.0/18
     - 94.139.128.0/17
     - 94.140.0.0/15
+    - 94.140.16.0/24
     - 94.142.0.0/17
     - 94.142.128.0/18
     - 94.142.192.0/20
     - 94.142.216.0/21
     - 94.142.224.0/19
     - 94.143.0.0/16
+    - 94.143.105.0/24
+    - 94.143.106.0/23
+    - 94.143.108.0/23
+    - 94.143.110.0/24
     - 94.144.0.0/12
     - 94.160.0.0/11
     - 94.192.0.0/11
@@ -573,11 +578,6 @@
     - 94.235.0.0/16
     - 94.236.0.0/14
     - 94.240.0.0/12
-    - 94.140.16.0/24
-    - 94.143.105.0/24
-    - 94.143.106.0/23
-    - 94.143.108.0/23
-    - 94.143.110.0/24
     - 94.245.64.0/18
     - 95.0.0.0/10
     - 95.64.0.0/12
@@ -590,18 +590,23 @@
     - 98.142.122.0/23
     - 101.0.0.0/8
     - 102.0.0.0/9
+    - 102.37.0.0/16
     - 102.128.0.0/11
+    - 102.133.0.0/16
     - 102.160.0.0/12
     - 102.176.0.0/14
     - 102.180.0.0/15
     - 102.183.0.0/16
     - 102.184.0.0/13
     - 102.192.0.0/10
-    - 102.37.0.0/16
-    - 102.133.0.0/16
     - 102.211.187.0/24
     - 102.223.214.0/24
     - 103.0.0.0/10
+    - 103.8.80.0/24
+    - 103.36.129.0/24
+    - 103.58.119.0/24
+    - 103.61.61.0/24
+    - 103.61.62.0/23
     - 103.64.0.0/11
     - 103.96.0.0/15
     - 103.98.0.0/16
@@ -614,9 +619,11 @@
     - 103.99.220.0/22
     - 103.99.224.0/19
     - 103.100.0.0/14
+    - 103.103.44.0/22
     - 103.104.0.0/13
     - 103.112.0.0/12
     - 103.128.0.0/12
+    - 103.131.148.0/23
     - 103.144.0.0/13
     - 103.152.0.0/14
     - 103.156.0.0/15
@@ -630,6 +637,7 @@
     - 103.159.220.0/22
     - 103.159.224.0/19
     - 103.160.0.0/11
+    - 103.164.237.0/24
     - 103.192.0.0/14
     - 103.196.0.0/16
     - 103.197.0.0/17
@@ -642,18 +650,10 @@
     - 103.197.192.0/18
     - 103.198.0.0/15
     - 103.200.0.0/13
-    - 103.208.0.0/12
-    - 103.224.0.0/11
-    - 103.8.80.0/24
-    - 103.36.129.0/24
-    - 103.58.119.0/24
-    - 103.61.61.0/24
-    - 103.61.62.0/23
-    - 103.103.44.0/22
-    - 103.131.148.0/23
-    - 103.164.237.0/24
     - 103.204.21.0/24
     - 103.204.22.0/24
+    - 103.208.0.0/12
+    - 103.224.0.0/11
     - 103.249.62.0/23
     - 103.254.7.0/24
     - 104.40.0.0/13
@@ -715,6 +715,7 @@
     - 119.0.0.0/8
     - 122.0.0.0/9
     - 122.128.0.0/11
+    - 122.149.0.0/16
     - 122.160.0.0/12
     - 122.176.0.0/13
     - 122.184.0.0/15
@@ -723,7 +724,6 @@
     - 122.187.128.0/17
     - 122.188.0.0/14
     - 122.192.0.0/10
-    - 122.149.0.0/16
     - 123.0.0.0/8
     - 124.0.0.0/9
     - 124.128.0.0/11
@@ -739,7 +739,6 @@
     - 124.176.0.0/12
     - 124.192.0.0/10
     - 124.252.0.0/16
-    - 128.241.0.0/16
     - 128.24.0.0/16
     - 128.36.15.0/24
     - 128.85.0.0/16
@@ -747,6 +746,7 @@
     - 128.127.76.0/24
     - 128.127.79.0/24
     - 128.203.0.0/16
+    - 128.241.0.0/16
     - 128.251.0.0/16
     - 129.35.19.0/24
     - 129.135.0.0/16
@@ -827,20 +827,20 @@
     - 137.135.0.0/16
     - 137.162.0.0/16
     - 138.0.0.0/9
-    - 138.128.0.0/10
-    - 138.192.0.0/13
-    - 138.200.0.0/16
-    - 138.202.0.0/15
-    - 138.204.0.0/14
-    - 138.208.0.0/12
-    - 138.224.0.0/11
     - 138.32.108.0/22
     - 138.32.112.0/24
     - 138.32.114.0/24
     - 138.91.0.0/16
     - 138.105.0.0/16
+    - 138.128.0.0/10
     - 138.128.250.0/23
+    - 138.192.0.0/13
+    - 138.200.0.0/16
+    - 138.202.0.0/15
+    - 138.204.0.0/14
+    - 138.208.0.0/12
     - 138.213.0.0/16
+    - 138.224.0.0/11
     - 138.239.0.0/16
     - 139.188.0.0/16
     - 140.150.226.0/23
@@ -869,10 +869,10 @@
     - 143.178.0.0/16
     - 143.180.0.0/14
     - 143.184.0.0/13
-    - 143.192.0.0/10
     - 143.188.148.0/22
     - 143.188.152.0/21
     - 143.188.160.0/23
+    - 143.192.0.0/10
     - 143.209.0.0/16
     - 143.226.0.0/16
     - 143.241.0.0/16
@@ -891,8 +891,8 @@
     - 145.219.20.0/22
     - 145.219.24.0/21
     - 145.219.32.0/23
-    - 146.174.160.0/19
     - 146.116.249.0/24
+    - 146.174.160.0/19
     - 146.225.0.0/21
     - 147.79.26.0/24
     - 147.140.72.0/23
@@ -933,12 +933,12 @@
     - 152.37.0.0/18
     - 152.37.128.0/17
     - 152.38.0.0/15
+    - 152.38.156.0/24
     - 152.40.0.0/13
     - 152.48.0.0/13
     - 152.60.0.0/14
     - 152.64.0.0/10
     - 152.128.0.0/9
-    - 152.38.156.0/24
     - 152.138.0.0/16
     - 153.65.20.0/24
     - 153.65.71.0/24
@@ -973,7 +973,11 @@
     - 158.116.129.0/24
     - 158.158.0.0/16
     - 159.0.0.0/9
+    - 159.14.174.0/24
+    - 159.14.176.0/24
+    - 159.105.18.0/24
     - 159.128.0.0/10
+    - 159.128.0.0/16
     - 159.192.0.0/14
     - 159.196.0.0/17
     - 159.196.128.0/22
@@ -987,12 +991,8 @@
     - 159.198.0.0/15
     - 159.200.0.0/13
     - 159.208.0.0/12
-    - 159.224.0.0/11
-    - 159.14.174.0/24
-    - 159.14.176.0/24
-    - 159.105.18.0/24
-    - 159.128.0.0/16
     - 159.222.162.0/24
+    - 159.224.0.0/11
     - 160.0.0.0/8
     - 160.4.0.0/16
     - 160.92.192.0/24
@@ -1040,8 +1040,8 @@
     - 166.52.0.0/14
     - 166.56.0.0/13
     - 166.64.0.0/10
-    - 166.128.0.0/9
     - 166.98.6.0/23
+    - 166.128.0.0/9
     - 167.0.0.0/8
     - 167.21.32.0/21
     - 167.21.152.0/21
@@ -1053,7 +1053,11 @@
     - 167.231.0.0/16
     - 167.253.12.0/22
     - 168.0.0.0/9
+    - 168.61.0.0/16
+    - 168.62.0.0/15
+    - 168.98.112.0/21
     - 168.128.0.0/11
+    - 168.137.64.0/20
     - 168.160.0.0/12
     - 168.176.0.0/13
     - 168.184.0.0/18
@@ -1068,12 +1072,8 @@
     - 168.186.0.0/15
     - 168.188.0.0/14
     - 168.192.0.0/10
-    - 168.61.0.0/16
-    - 168.62.0.0/15
-    - 168.98.112.0/21
-    - 168.137.64.0/20
-    - 169.224.0.0/16
     - 169.138.0.0/16
+    - 169.224.0.0/16
     - 170.0.0.0/8
     - 170.22.154.0/23
     - 170.22.156.0/23
@@ -1140,7 +1140,14 @@
     - 181.176.0.0/12
     - 181.192.0.0/10
     - 185.0.0.0/10
+    - 185.12.10.0/24
+    - 185.34.84.0/22
+    - 185.34.250.0/24
+    - 185.42.70.0/24
+    - 185.45.122.0/24
+    - 185.52.203.0/24
     - 185.64.0.0/13
+    - 185.70.160.0/23
     - 185.72.0.0/16
     - 185.73.0.0/17
     - 185.73.128.0/19
@@ -1150,7 +1157,12 @@
     - 185.73.192.0/18
     - 185.74.0.0/15
     - 185.76.0.0/14
+    - 185.76.37.0/24
+    - 185.76.182.0/23
     - 185.80.0.0/12
+    - 185.87.134.0/24
+    - 185.89.200.0/22
+    - 185.90.154.0/24
     - 185.96.0.0/13
     - 185.104.0.0/17
     - 185.104.128.0/21
@@ -1164,21 +1176,35 @@
     - 185.106.0.0/15
     - 185.108.0.0/14
     - 185.112.0.0/12
+    - 185.113.189.0/24
+    - 185.115.155.0/24
+    - 185.116.168.0/23
+    - 185.117.95.0/24
+    - 185.117.183.0/24
+    - 185.117.230.0/24
     - 185.128.0.0/12
+    - 185.135.58.0/24
     - 185.144.0.0/13
+    - 185.149.54.0/24
     - 185.152.0.0/14
+    - 185.154.80.0/22
     - 185.156.0.0/16
     - 185.157.0.0/21
     - 185.157.8.0/22
     - 185.157.12.0/23
     - 185.157.15.0/24
     - 185.157.16.0/20
+    - 185.157.16.0/22
     - 185.157.32.0/19
     - 185.157.64.0/18
     - 185.157.128.0/17
     - 185.158.0.0/15
     - 185.160.0.0/11
+    - 185.187.104.0/23
+    - 185.187.123.0/24
     - 185.192.0.0/13
+    - 185.192.184.0/23
+    - 185.195.244.0/23
     - 185.200.0.0/18
     - 185.200.64.0/19
     - 185.200.96.0/22
@@ -1191,6 +1217,9 @@
     - 185.202.0.0/15
     - 185.204.0.0/14
     - 185.208.0.0/12
+    - 185.209.208.0/23
+    - 185.211.208.0/24
+    - 185.222.140.0/22
     - 185.224.0.0/13
     - 185.232.0.0/16
     - 185.233.0.0/18
@@ -1201,38 +1230,9 @@
     - 185.233.128.0/17
     - 185.234.0.0/15
     - 185.236.0.0/14
-    - 185.240.0.0/12
-    - 185.12.10.0/24
-    - 185.34.84.0/22
-    - 185.34.250.0/24
-    - 185.42.70.0/24
-    - 185.45.122.0/24
-    - 185.52.203.0/24
-    - 185.70.160.0/23
-    - 185.76.37.0/24
-    - 185.76.182.0/23
-    - 185.87.134.0/24
-    - 185.89.200.0/22
-    - 185.90.154.0/24
-    - 185.113.189.0/24
-    - 185.115.155.0/24
-    - 185.116.168.0/23
-    - 185.117.95.0/24
-    - 185.117.183.0/24
-    - 185.117.230.0/24
-    - 185.135.58.0/24
-    - 185.149.54.0/24
-    - 185.154.80.0/22
-    - 185.157.16.0/22
-    - 185.187.104.0/23
-    - 185.187.123.0/24
-    - 185.192.184.0/23
-    - 185.195.244.0/23
-    - 185.209.208.0/23
-    - 185.211.208.0/24
-    - 185.222.140.0/22
     - 185.236.221.0/24
     - 185.238.165.0/24
+    - 185.240.0.0/12
     - 185.242.137.0/24
     - 185.242.139.0/24
     - 186.0.0.0/11
@@ -1257,6 +1257,7 @@
     - 188.28.0.0/14
     - 188.32.0.0/11
     - 188.64.0.0/11
+    - 188.94.190.0/24
     - 188.96.0.0/12
     - 188.112.0.0/13
     - 188.120.0.0/15
@@ -1298,7 +1299,6 @@
     - 188.253.220.0/22
     - 188.253.224.0/19
     - 188.254.0.0/15
-    - 188.94.190.0/24
     - 188.255.174.0/23
     - 189.0.0.0/9
     - 189.128.0.0/10
@@ -1609,12 +1609,12 @@
     - 209.199.0.0/16
     - 209.240.192.0/19
     - 209.251.248.0/24
-    - 212.237.120.0/22
     - 212.1.218.0/23
     - 212.1.222.0/23
     - 212.46.57.0/24
     - 212.103.45.0/24
     - 212.207.0.0/16
+    - 212.237.120.0/22
     - 213.54.0.0/16
     - 213.109.150.0/24
     - 213.156.248.0/23
@@ -1635,9 +1635,9 @@
     - 216.238.128.0/24
     - 216.238.135.0/24
     - 216.238.141.0/24
-    - 217.147.172.0/24
     - 217.11.173.0/24
     - 217.71.189.0/24
+    - 217.147.172.0/24
     - 217.169.202.0/24
     - 217.176.0.0/16
 acmechief_host: deployment-acme-chief05.deployment-prep.eqiad1.wikimedia.cloud

• bd808 mentioned this in T399415: Unable to generate family for wpbeta:zh with github action (ClientError: (403) Request forbidden).Apr 7 2026, 3:39 PM

AlexisJazz mentioned this in T425600: "Nahui Ollin" is enwiki's top-viewed article in April 2026.May 7 2026, 2:11 AM

rae subscribed.May 7 2026, 3:00 AM

Count_Count subscribed.May 19 2026, 2:55 PM

The bots are back.

Screenshot 2026-05-20 at 19.53.50.png (2,054×1,414 px, 472 KB)

• bd808 merged a task: T426831: Beta cluster down: Error: 502, Backend fetch failed.May 21 2026, 1:55 AM

• bd808 added a subscriber: Viktoria_Hillerud_WMSE.

• bd808 triaged this task as High priority.May 21 2026, 1:58 AM

The current traffic pattern is full of the "2 requests per /24 network" pattern. Range blocks will need to be vast again to make much difference. We really need to get requestctl and more fine grained pattern matching tech into the Beta Cluster CDN edge.

root@deployment-mediawiki14:~# ./small-ban-hammer.sh 1 +1 | sort -n -k 4 | grep '# 2 hits' | wc -l
11118

That is 11,118 distinct /24 networks each sending 2 requests in the last ~2 hours.

Nikerabbit subscribed.May 21 2026, 11:19 AM

In T420833#11943406, @bd808 wrote:

That is 11,118 distinct /24 networks each sending 2 requests in the last ~2 hours.

19,091 now after ~16 hours. The load has fallen off a bit too, but is still quite elevated from our typical baseline. No new blocks applied yet.

• bd808 mentioned this in T428989: Beta cluster access via open proxies (due to internet censorship).Jun 26 2026, 4:54 PM

Request served via deployment-cache-text08 deployment-cache-text08, Varnish XID 644382721
Error: 503, Backend fetch failed at Thu, 02 Jul 2026 04:03:14 GMT

What are these stupid crawlers even requesting? There's not that much "content" on beta.

What would happen if https://en.wikipedia.org/wiki/Basic_access_authentication was enabled? Even if the password was "password", perhaps bots would just give up?

What would happen if you'd serve a small simple site for 24 hours, would that trick the crawlers into thinking they're done?

atsuko mentioned this in T430882: Beta cluster is not working due to error in Translate\TtmServer\ServiceCreationFailure.Jul 2 2026, 4:58 PM

In T420833#12078788, @AlexisJazz wrote:

What would happen if https://en.wikipedia.org/wiki/Basic_access_authentication was enabled? Even if the password was "password", perhaps bots would just give up?

Every existing automation against Beta cluster would fail.

What would happen if you'd serve a small simple site for 24 hours, would that trick the crawlers into thinking they're done?

Every existing automation against Beta cluster would fail. Bot wave would probably die off until the next time they come around again.

We only keep 30 days of load data so it is hard to see some of the longer trends, but the shape of this graph feels pretty typical for an amplified crawler wave:

Screenshot 2026-07-02 at 11.40.00.png (2,208×1,350 px, 740 KB)

This graph is a drill down of the load graph on https://grafana.wmcloud.org/goto/afqx02dhiv4sgd?orgId=1 with only 15m averages shown.

root@deployment-mediawiki14:~# ./small-ban-hammer.sh 1 +1 | sort -n -k 4 | grep '# 2 hits' | wc -l
53136

2 hits per /24 network across 53,136 in the last ~18 hours. We are still under attack from ~~bot nets~~ residential proxies.

In T420833#12082393, @bd808 wrote:

Every existing automation against Beta cluster would fail. Bot wave would probably die off until the next time they come around again.

Assuming these crawlers are permanently present, what if you served a small site for just one hour? They must be badly configured (why would anyone crawl beta cluster to begin with?), who knows for how long they'll back off.

You could even copy a "this domain is for sale!" page from a domain squatter, if there's anything that makes these bots leave for an extended period of time I'd expect it to be that.

Xqt added a parent task: T431173: site_tests.TestProductionAndTestSite fails with TimeoutError.Jul 4 2026, 10:06 AM

what if you served a small site for just one hour?

Please read the previous answers again.

You could even copy a "this domain is for sale!" page from a domain squatter

Please read the previous answers again.

I'd expect it to be that.

We don't.

In T420833#12087315, @Aklapper wrote:

what if you served a small site for just one hour?

Please read the previous answers again.

You could even copy a "this domain is for sale!" page from a domain squatter

Please read the previous answers again.

If it's only one hour at least some of the "automations against beta cluster" could possibly be paused, or the hour could be scheduled for a time of day when few or no automations are running, and others (that actually target beta cluster and aren't random crawlers) should resume after the hour is over. Maybe some errors get reported erroneously, on the other hand, the regular downtime should be causing logspam too.

I'd expect it to be that.

We don't.

You either know too much or too little about how these crawlers work. Either way, neither of us will ever know if we never try.

Instead of what you wrote, which comes across as passive-aggressive, you could have said "Unfortunately, even just an hour would cause equally much disruption because $REASON. And due to $MORE_REASON this would be a severe problem that can't ever outweigh beta cluster being crawled to death. I'm unsure if these crawlers would ever back off for a substantial amount of time." Same message without coming across as disrespectful.

Unlike you, nobody pays me to be here. Your passive-aggressive style doesn't sit well with me, and I hope it wasn't intended to come across like that.

Re-reading what I wrote before, I see that it can be interpreted as passive-aggressive. I apologize.

In T420833#12087356, @Aklapper wrote:

Re-reading what I wrote before, I see that it can be interpreted as passive-aggressive. I apologize.

Apology accepted. :-) It's inevitable something occasionally doesn't come across as intended when using a text-only medium.

Is there no possible solution? More bandwidth/servers might help, but cost money and it's questionable if it would even work. Since they use residential proxies I assume they're effectively undetectable, using a common user agent and not requesting URLs that could be easily matched with a pattern. Using some DDOS protection service would possibly help, but also costs money and may create accessibility problems.

Maybe if the first few requests from an IP in an N-day period would have an 80% chance of being dropped. The reason for the 80% is to prevent tipping off the bot operators how it works. Or respond very slowly to the first N requests in an N-day period. Any normal user would get through after a few tries, crawlers with disposable IPs would be at a disadvantage. Maybe there's something about the requests that could be leveraged to reduce their impact, for example reducing the default "Number of edits to show in recent changes, page histories, and in logs" if that's how they crawl. One more odd idea: unblock everything, wait for the residential proxies to subside (they will presumably return to using data centers when allowed as that's likely cheaper) and instead of blocking the data centers outright just slow them down/throttle them, if the software has any option to do that.

If none of this is usable my apologies, I'm only trying to help. I use beta cluster on a semi-regular basis to test stuff to avoid disrupting production. Testwiki is sometimes an alternative, but not always.

• bd808 mentioned this in T432878: Automated traffic affects paulina.toolforge.org performance.Jul 22 2026, 8:47 PM

Wellverywell subscribed.Aug 1 2026, 8:03 AM

"Fun" new developments. :/

In T435408#12234663, @bd808 wrote:

I think the crawler pressure is so high that the CDN edge is getting swamped out and missing metrics reporting periods.

Maybe it's time to put a honeypot - add a hidden link on Beta Cluster output (somwhere on top, maybe banner ? ) - link should be hidden or by CSS or by JS - so regular users cannot see this. But bots/crawlers do not care - if they visit that link, then block that IP for some time.

In T420833#12238031, @pmiazga wrote:

Maybe it's time to put a honeypot - add a hidden link on Beta Cluster output (somwhere on top, maybe banner ? ) - link should be hidden or by CSS or by JS - so regular users cannot see this. But bots/crawlers do not care - if they visit that link, then block that IP for some time.

The bulk of the unwanted traffic is coming in via the "2 requests per /24 network" pattern. This makes IP blocking close to useless.

[23:20]  <    bd808> *sigh* Beta Cluster is functionally down because of crawlers. They have used 51,606 distinct /24 networks today to make 1,431,324 requests. so. tiring.
[23:23]  <    bd808> that "51,606 distinct /24 networks" stat is a undercount. Those are the unique /24 networks that have each been used to make *2* requests. There are other /24s carrying more traffic.
[23:23]  <    bd808> 151,920 is the total number of distinct /24s seen today
[23:26]  <    bd808> !log deployment-mediawiki14: `systemctl stop php8.3-fpm; sleep 5m; systemctl start php8.3-fpm` -- maybe the bot storm will break if we give fast 500 responses for 5 minutes.

maybe the bot storm will break if we give fast 500 responses for 5 minutes.

Spoiler, it did not.

I asked ChatGPT 5.6 Sol Extra High, and it seems to recommend adding admission challenge (CAPTCHA, Wikimedia/OAuth-based verification, scoped bearer tokens etc), creating separate verified and anonymous capacity pools, block or strongly limit gate expensive URL classes, bring request-pattern controls (like Wikimedia’s production requestctl system) into beta clusters, and disallow the verified traffic to overload the single MediaWiki backend.

I have attached the conversation file as below, not sure if it helps though:

Solution to beta cluster being crawled to death.pdf1 MBDownload

It's probably harder than it looks to solve this due to wanting to keep beta cluster pretty similar to prod, and due to bots being able to disguise themselves so well.

But if we're having non-SRE's submit ideas, here's a couple of mine:

Htpasswd (or whatever the non apache equivalent is) comes to mind as a dead simple solution. Could even publish the password publicly, since the bots probably arent smart enough to read the docs and use it.

Anubis worked well for chasing away bots from xtools. It has the browser do some calculations before finishing the request.

In T420833#12240092, @Novem_Linguae wrote:

It's probably harder than it looks to solve this due to wanting to keep beta cluster pretty similar to prod, and due to bots being able to disguise themselves so well.

I think the bigger problem is that it's a target for automated tests without human intervention, they can't do Anubis.

I guess we could get some of those automations to add HTTP basic auth (aka your htpasswd suggestion) but I had the impression some of the automation operators would be harder to contact. Was also thinking about modifying UA string for automations as another possible way to bypass checks. We don't necessarily need a single unified solution. It could be Anubis for humans and UA string or basic auth for automations.

karapayneWMDE subscribed.Aug 21 2026, 9:54 AM

In T420833#12239970, @Hakimi97 wrote:

I asked ChatGPT 5.6 Sol Extra High

This is basically the same as going “I asked the local arsonist how to prevent house fires”. LLM vendors desperately scraping the web for any source of training input they can get their hands on is the cause of this miserable problem; it’s not going to be the solution.

Also, for the HTTP basic auth idea, see T420833#12082393.

In T420833#12240963, @LucasWerkmeister wrote:

In T420833#12239970, @Hakimi97 wrote:

I asked ChatGPT 5.6 Sol Extra High

This is basically the same as going “I asked the local arsonist how to prevent house fires”. LLM vendors desperately scraping the web for any source of training input they can get their hands on is the cause of this miserable problem; it’s not going to be the solution.

I mean, people have been using LLM to identify security vulnerabilities to either take advantage of the situation or to fix the code vulnerabities. I don't see how LLM should be disallowed or be discriminated if it is used to identify problems/possible solutions that we could not even answer confidently.

Okay here is another question: Can someone here guarantee that the beta cluster to have the exact same environment as production wikis' one? If indeed the same why do the beta clusters are more vulnerable to be overwhelmed/attacked by crawlers compared to production wikis?

In T420833#12240963, @LucasWerkmeister wrote:

Also, for the HTTP basic auth idea, see T420833#12082393.

I'm sure I saw that before. they also fail when the whole site is down. we can give an error page with documentation instead of just blocking. static page that doesn't hit PHP or DB.

how important can an automation be if no one noticed it stopped working?

In T420833#12241167, @Hakimi97 wrote:

have been using LLM to identify security vulnerabilities

this isn't a security vulnerability we're discussing. this isn't the right forum to debate LLM usage.

In T420833#12241167, @Hakimi97 wrote:

Okay here is another question: Can someone here guarantee that the beta cluster to have the exact same environment as production wikis' one? If indeed the same why do the beta clusters are more vulnerable to be overwhelmed/attacked by crawlers compared to production wikis?

Your chatbot session actually covered this in pretty deep detail. You prompted it on page 4 with:

What are the differences between Wikimedia production wikis and Wikimedia beta clusters that causing this issue occur mainly on beta clusters?

The generative pre-trained transformer using RAG + summary algorithms presented you with ~4 printed pages of output that describe how Beta is a smaller scale system in both hardware and human resources and some of the follow on effects of that difference.

With additional token inputs from other writings, it may have also summarized content explaining that a number of traffic management tools which are used in the WMF production network are encumbered by commercial licenses or trade secrets. These solutions are not present in the Beta Cluster environment today because it is an open project which makes complying with embargoed information protections too difficult to have been attempted.

TL;DR: Beta Cluster has:

fewer computers to handle traffic
fewer people to monitor and respond to issues
fewer tools to use when responding to traffic overload issues

In T420833#12241539, @jeremyb wrote:

how important can an automation be if no one noticed it stopped working?

I imagine plenty of people have noticed, but what are they supposed to do about it? (Other than try to get the LLM bubble to pop already, I suppose.)

I for one started replacing my beta-targeting automations with local wikis instead (including in AC/DC, and I thought there was a second repo where I did something similar but I can’t find it now so maybe I’m mistaken).

In T420833#12239970, @Hakimi97 wrote:

I asked ChatGPT 5.6 Sol Extra High, and it seems to recommend adding admission challenge (CAPTCHA, Wikimedia/OAuth-based verification, scoped bearer tokens etc), creating separate verified and anonymous capacity pools, block or strongly limit gate expensive URL classes, bring request-pattern controls (like Wikimedia’s production requestctl system) into beta clusters, and disallow the verified traffic to overload the single MediaWiki backend.

I have attached the conversation file as below, not sure if it helps though:

Solution to beta cluster being crawled to death.pdf1 MBDownload

As usual, AI is virtually useless. CAPTCHAs etc are not acceptable and anonymous access is supposed to work.

In T420833#12240092, @Novem_Linguae wrote:

It's probably harder than it looks to solve this due to wanting to keep beta cluster pretty similar to prod, and due to bots being able to disguise themselves so well.

But if we're having non-SRE's submit ideas, here's a couple of mine:

Htpasswd (or whatever the non apache equivalent is) comes to mind as a dead simple solution. Could even publish the password publicly, since the bots probably arent smart enough to read the docs and use it.

Anubis worked well for chasing away bots from xtools. It has the browser do some calculations before finishing the request.

I already suggested that: T420833#12078788 and that idea was shut down because "Every existing automation against Beta cluster would fail."

Would it be okay to create, say, a few dozen or hundred articles on beta cluster with LLM jailbreak attempts or information written to disrupt/poison the training process? This probably isn't going to suddenly solve the problem at hand unless the crawlers detect we are screwing with them, but it can't really hurt either?

Oink oink oink oink oink oink oink oink oink?

In T420833#12242859, @LucasWerkmeister wrote:

I imagine plenty of people have noticed, but what are they supposed to do about it? (Other than try to get the LLM bubble to pop already, I suppose.)

I'm not just talking about outages but also in case we announce:

starting in one month you need to do xyz to bypass Anubis or CAPTCHA.

and then some operators adopt the bypass method in time and some do it after they notice one month later their thing stopped working.

we have a cloud announcements list. do we have or need a beta announcements list?

In T420833#12242907, @AlexisJazz wrote:

Would it be okay to create, say, a few dozen or hundred articles on beta cluster with LLM jailbreak attempts or information written to disrupt/poison the training process? This probably isn't going to suddenly solve the problem at hand unless the crawlers detect we are screwing with them, but it can't really hurt either?

I love the subversive energy of this, but I do not think it would be a good use of our time. "Wikipedia deliberately poisons AI training data" is not a headline I want our organizations and communities to have to deal with. The clickbait power of that for all sides of the debate would be too strong for the average news organization to resist instead of reporting "Wikimedia fights back against commercial bot nets destroying their testing systems." (Yes, I believe residential proxies are exactly the same as dark web bot nets.)

I deployed Produnto to beta, but for the last 24 hours it's been too slow to verify whether it's working or not. I suggest switching to an allowlist of IP networks, allowing only WMF. No need to add staff IPs, we can use ssh -D. Volunteers can use ssh -D via Toolforge.

Ideally there would be some kind of web-based authentication or tunnel but just taking it off the public internet is a simple interim solution.

In T420833#12243359, @tstarling wrote:

I deployed Produnto to beta, but for the last 24 hours it's been too slow to verify whether it's working or not. I suggest switching to an allowlist of IP networks, allowing only WMF. No need to add staff IPs, we can use ssh -D. Volunteers can use ssh -D via Toolforge.

Ideally there would be some kind of web-based authentication or tunnel but just taking it off the public internet is a simple interim solution.

Wait how would I access beta cluster if this happens? I have no Toolforge account? And what happened to "Every existing automation against Beta cluster would fail"? Wouldn't HTTP basic auth be simpler and more accessible?

In T420833#12244804, @AlexisJazz wrote:

And what happened to "Every existing automation against Beta cluster would fail"?

They would use whitelisted IPs, so there would be no change for them.

Wouldn't HTTP basic auth be simpler

Definitely not. I'm just saying block all traffic from the public internet as a network rule. No need to even touch puppet.

more accessible?

I don't really understand why people use beta or why you care about it.

In T420833#12245118, @tstarling wrote:

In T420833#12244804, @AlexisJazz wrote:

And what happened to "Every existing automation against Beta cluster would fail"?

They would use whitelisted IPs, so there would be no change for them.

Wouldn't HTTP basic auth be simpler

Definitely not. I'm just saying block all traffic from the public internet as a network rule. No need to even touch puppet.

more accessible?

I don't really understand why people use beta or why you care about it.

I test gadgets, scripts, template changes and other things on beta to avoid disrupting production. When something strange happens on production, I try to reproduce it on beta cluster for the same reason. I have filed many bug reports based on testing on beta.

The Commons app is also tested on beta cluster.

In T420833#12245119, @AlexisJazz wrote:

I test gadgets, scripts, template changes and other things on beta to avoid disrupting production. When something strange happens on production, I try to reproduce it on beta cluster for the same reason. I have filed many bug reports based on testing on beta.

You can use test.wikipedia.org for that kind of thing.

The Commons app is also tested on beta cluster.

There's test-commons.wikimedia.org, which is supposedly closed per T213295, but it's easier to reopen it than to maintain a whole parallel infrastructure.

bking subscribed.Aug 24 2026, 12:57 PM

In T420833#12246071, @tstarling wrote:

In T420833#12245119, @AlexisJazz wrote:

I test gadgets, scripts, template changes and other things on beta to avoid disrupting production. When something strange happens on production, I try to reproduce it on beta cluster for the same reason. I have filed many bug reports based on testing on beta.

You can use test.wikipedia.org for that kind of thing.

Sometimes, but often not. testwiki has no shared file repository like Commons, doesn't enable testing behavior on a RTL wiki and has no associated testwikidata, to name a few things.

And beta cluster is ahead of production, on a few occasions I filed bug reports for stuff before it could production.

Viktoria_Hillerud_WMSE mentioned this in T435926: We want to be able to test the tool by logging in.Aug 25 2026, 12:15 PM

DLynch subscribed.Aug 25 2026, 5:21 PM

@tstarling Since it was decided that the current solution is to block all public internet traffic except for those within the IP whitelist, should this task be boldly "resolved"?

In T420833#12254869, @Hakimi97 wrote:

@tstarling Since it was decided that the current solution is to block all public internet traffic except for those within the IP whitelist, should this task be boldly "resolved"?

Actually blocking the traffic would resolve the task. But I'm not going to do that on my own authority, I don't own Beta. It's a suggestion for @bd808 and the relevant teams to consider.

(adding some Commons app contributors to inform them about possible changes)

Bugreporter2 awarded a token.Aug 29 2026, 11:50 AM

Tgr subscribed.Sep 1 2026, 10:07 AM

I understand that there are huge issues with bot traffic. However, beta-dewiki in particular has been established for well over a decade as a testing ground for interface adjustments, scripts, important templates etc. in an environment rather close to production, even anticipating the effects of configuration changes in a playground environment. Testwiki can not adequately replace these use cases, especially as it serves too many communities in parallel. See e.g. our onwiki documentation at
https://de.wikipedia.org/wiki/Wikipedia:Technik/Cloud/Beta

In my experience, the current ip blocks render beta mostly unusable, because more often than not access is blocked for me and others. Above you say "captchas etc are not acceptable". Surely, it would be better for blacklisted ips to employ captchas than blocking access entirely. Or are there issues with technical implementability of such a captcha requirement?

See also T401839#12175319 and the comments in T215217.

Beta cluster is being crawled to death by bot traffic coming from residential proxiesOpen, HighPublicBUG REPORTActions

Description

Related ObjectsSearch...

Event Timeline

Beta cluster is being crawled to death by bot traffic coming from residential proxies
Open, HighPublicBUG REPORT
Actions

Related Objects
Search...