Page MenuHomePhabricator

High load on deployment-mediawiki14 and slow responses
Closed, ResolvedPublicBUG REPORT

Description

Same deal as T392003: High load on deployment-mediawiki14 and slow responses . The load average on deployment-mediawiki14 is up around 7 again which functionally means all of the PHP workers are busy serving content to bots.

See T392534#10763059 for some hopefully helpful tips on finding things to block.

Event Timeline

bd808 changed the task status from Open to In Progress.Apr 23 2025, 8:32 PM
bd808 claimed this task.
bd808 triaged this task as High priority.

I am again using variations of tail -50000 /var/log/apache2/other_vhosts_access-json.log | grep -oP '"X-Client-IP": "\d+\.\d+\.\d+\.' |sort|uniq -c|sort -nr|head -n40 to find clusters of requests that look suspicious.

https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/d8b94389e713db63d2964c42e5c1591e15a303f8

diff --git a/deployment-prep/_.yaml b/deployment-prep/_.yaml
index 0486816..a23ebe8 100644
--- a/deployment-prep/_.yaml
+++ b/deployment-prep/_.yaml

@@ -40,6 +40,13 @@
     - 154.16.246.0/24
     - 193.43.72.0/24
     - 14.160.0.0/11
+    - 138.186.108.0/22
+    - 168.121.96.0/22
+    - 170.80.36.0/22
+    - 177.39.120.0/21
+    - 185.244.152.0/24
+    - 187.95.16.0/20
+    - 189.84.176.0/20
 acmechief_host: deployment-acme-chief05.deployment-prep.eqiad1.wikimedia.cloud
 apt::use_experimental: true
 aptly::group: wikidev

Mentioned in SAL (#wikimedia-releng) [2025-04-23T20:46:59Z] <bd808> Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534)

Reedy renamed this task from HIgh load on deployment-mediawiki14 and slow responses to High load on deployment-mediawiki14 and slow responses.Apr 23 2025, 8:47 PM

https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/ef2212ca1aeee50d3277b4813d60acb6742aec73

diff --git a/deployment-prep/_.yaml b/deployment-prep/_.yaml
index a23ebe8..6dae330 100644
--- a/deployment-prep/_.yaml
+++ b/deployment-prep/_.yaml

@@ -47,6 +47,16 @@
     - 185.244.152.0/24
     - 187.95.16.0/20
     - 189.84.176.0/20
+    - 179.125.128.0/17
+    - 177.37.128.0/17
+    - 187.19.128.0/17
+    - 191.5.80.0/20
+    - 186.249.208.0/21
+    - 170.239.72.0/22
+    - 45.226.116.0/22
+    - 189.60.0.0/14
+    - 177.22.176.0/21
+    - 138.0.64.0/22
 acmechief_host: deployment-acme-chief05.deployment-prep.eqiad1.wikimedia.cloud
 apt::use_experimental: true
 aptly::group: wikidev

I made some quick helper scripts in deployment-mediawiki14/root for this stuff:

find-top-class-b.sh
#!/usr/bin/env bash
exec tail -50000 /var/log/apache2/other_vhosts_access-json.log |
        grep -oP '"X-Client-IP": "\d+\.\d+\.' |
        sort |
        uniq -c |
        sort -nr |
        head -n20
class-c-drilldown.sh
#!/usr/bin/env bash
exec tail -50000 /var/log/apache2/other_vhosts_access-json.log |
        grep "$1" |
        grep -oP '"X-Client-IP": "\d+\.\d+\.\d+\.' |
        sort |
        uniq -c |
        sort -nr |
        head -n40

I am using them something like:

root@deployment-mediawiki14:~# ./find-top-class-b.sh
    332 "X-Client-IP": "177.37.
    304 "X-Client-IP": "179.125.
    252 "X-Client-IP": "187.19.
    213 "X-Client-IP": "45.226.
    205 "X-Client-IP": "186.249.
    190 "X-Client-IP": "191.5.
    186 "X-Client-IP": "170.239.
    179 "X-Client-IP": "138.0.
    176 "X-Client-IP": "45.190.
    176 "X-Client-IP": "177.22.
    175 "X-Client-IP": "168.197.
    173 "X-Client-IP": "191.37.
    171 "X-Client-IP": "45.70.
    171 "X-Client-IP": "138.97.
    169 "X-Client-IP": "189.6.
    169 "X-Client-IP": "177.137.
    169 "X-Client-IP": "170.83.
    166 "X-Client-IP": "168.232.
    164 "X-Client-IP": "177.221.
    164 "X-Client-IP": "170.247.
root@deployment-mediawiki14:~# ./class-c-drilldown.sh 177.37.
     13 "X-Client-IP": "177.37.198.
     12 "X-Client-IP": "177.37.235.
     12 "X-Client-IP": "177.37.232.
     11 "X-Client-IP": "177.37.233.
     10 "X-Client-IP": "177.37.132.
      9 "X-Client-IP": "177.37.181.
      9 "X-Client-IP": "177.37.152.
      9 "X-Client-IP": "177.37.133.
      8 "X-Client-IP": "177.37.237.
      8 "X-Client-IP": "177.37.149.
      8 "X-Client-IP": "177.37.148.
      7 "X-Client-IP": "177.37.186.
      7 "X-Client-IP": "177.37.184.
      7 "X-Client-IP": "177.37.183.
      7 "X-Client-IP": "177.37.182.
      6 "X-Client-IP": "177.37.236.
      6 "X-Client-IP": "177.37.185.
      6 "X-Client-IP": "177.37.177.
      6 "X-Client-IP": "177.37.174.
      6 "X-Client-IP": "177.37.173.
      6 "X-Client-IP": "177.37.155.
      6 "X-Client-IP": "177.37.145.
      6 "X-Client-IP": "177.37.144.
      6 "X-Client-IP": "177.37.136.
      5 "X-Client-IP": "177.37.85.
      5 "X-Client-IP": "177.37.248.
      5 "X-Client-IP": "177.37.241.
      5 "X-Client-IP": "177.37.187.
      5 "X-Client-IP": "177.37.180.
      5 "X-Client-IP": "177.37.178.
      5 "X-Client-IP": "177.37.154.
      5 "X-Client-IP": "177.37.138.
      4 "X-Client-IP": "177.37.87.
      4 "X-Client-IP": "177.37.84.
      4 "X-Client-IP": "177.37.251.
      4 "X-Client-IP": "177.37.249.
      4 "X-Client-IP": "177.37.197.
      4 "X-Client-IP": "177.37.137.
      4 "X-Client-IP": "177.37.103.
      3 "X-Client-IP": "177.37.250.

Then I take a class C like 177.37.198. and run a whois 177.37.198.1 from my laptop:

bd808@mbp03:~$ whois 177.37.198.1
inetnum:     177.37.128.0/17
aut-num:     AS28126
abuse-c:     JOSNO18
owner:       BRISANET SERVICOS DE TELECOMUNICACOES S.A
ownerid:     04.601.397/0001-28
responsible: JO�O PAULO ESTEVAM
country:     BR
owner-c:     JOSNO18
tech-c:      JOSNO18
created:     20110302
changed:     20211208

nic-hdl-br:  JOSNO18
person:      JOSE SELDOMAR NOGUEIRA
e-mail:      seldomar@grupobrisanet.com.br
country:     BR
created:     20150216
changed:     20240220

This is how 177.37.128.0/17 made it onto the block list in T392534#10763046.

https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/80718740d397e7a27c90e9749c65527c9c75ec4e

diff --git a/deployment-prep/_.yaml b/deployment-prep/_.yaml
index 6dae330..e63c9b6 100644
--- a/deployment-prep/_.yaml
+++ b/deployment-prep/_.yaml

@@ -57,6 +57,12 @@
     - 189.60.0.0/14
     - 177.22.176.0/21
     - 138.0.64.0/22
+    - 143.0.48.0/22
+    - 177.37.96.0/20
+    - 177.37.80.0/21
+    - 186.249.128.0/19
+    - 186.249.24.0/21
+    - 186.249.216.0/22
 acmechief_host: deployment-acme-chief05.deployment-prep.eqiad1.wikimedia.cloud
 apt::use_experimental: true
 aptly::group: wikidev

Mentioned in SAL (#wikimedia-releng) [2025-04-23T21:29:01Z] <bd808> Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534)

Getting more aggressive. This list was made by comparing ./find-top-class-b.sh output with ranges listed at https://ipnetinfo.com/country/BR.

https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/00fdf2ff43807c2080f97aba95733f5e39840b1d

diff --git a/deployment-prep/_.yaml b/deployment-prep/_.yaml
index e63c9b6..05ab2f7 100644
--- a/deployment-prep/_.yaml
+++ b/deployment-prep/_.yaml

@@ -63,6 +63,23 @@
     - 186.249.128.0/19
     - 186.249.24.0/21
     - 186.249.216.0/22
+    - 143.0.0.0/16
+    - 168.196.0.0/16
+    - 168.197.0.0/16
+    - 168.232.0.0/16
+    - 170.239.0.0/16
+    - 170.244.0.0/16
+    - 170.247.0.0/16
+    - 170.84.0.0/16
+    - 177.137.0.0/16
+    - 177.22.0.0/16
+    - 177.221.0.0/16
+    - 179.108.0.0/16
+    - 191.5.0.0/16
+    - 45.168.0.0/16
+    - 45.172.0.0/16
+    - 45.190.0.0/16
+    - 45.70.0.0/16
 acmechief_host: deployment-acme-chief05.deployment-prep.eqiad1.wikimedia.cloud
 apt::use_experimental: true
 aptly::group: wikidev

Mentioned in SAL (#wikimedia-releng) [2025-04-23T21:47:10Z] <bd808> Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534)

https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/daee6f5f7d94c4952db8bf7e5753aca584c90102

diff --git a/deployment-prep/_.yaml b/deployment-prep/_.yaml
index 05ab2f7..6522552 100644
--- a/deployment-prep/_.yaml
+++ b/deployment-prep/_.yaml

@@ -80,6 +80,142 @@
     - 45.172.0.0/16
     - 45.190.0.0/16
     - 45.70.0.0/16
+    - 111.119.0.0/16
+    - 124.243.0.0/16
+    - 128.201.0.0/16
+    - 131.161.0.0/16
+    - 131.196.0.0/16
+    - 131.221.0.0/16
+    - 131.255.0.0/16
+    - 131.72.0.0/16
+    - 138.0.0.0/16
+    - 138.117.0.0/16
+    - 138.122.0.0/16
+    - 138.185.0.0/16
+    - 138.204.0.0/16
+    - 138.255.0.0/16
+    - 138.59.0.0/16
+    - 138.94.0.0/16
+    - 138.97.0.0/16
+    - 143.137.0.0/16
+    - 143.202.0.0/16
+    - 143.255.0.0/16
+    - 164.163.0.0/16
+    - 167.249.0.0/16
+    - 167.250.0.0/16
+    - 168.0.0.0/16
+    - 168.121.0.0/16
+    - 168.181.0.0/16
+    - 168.195.0.0/16
+    - 168.205.0.0/16
+    - 168.228.0.0/16
+    - 168.90.0.0/16
+    - 170.0.0.0/16
+    - 170.150.0.0/16
+    - 170.231.0.0/16
+    - 170.233.0.0/16
+    - 170.238.0.0/16
+    - 170.245.0.0/16
+    - 170.246.0.0/16
+    - 170.254.0.0/16
+    - 170.78.0.0/16
+    - 170.79.0.0/16
+    - 170.80.0.0/16
+    - 170.81.0.0/16
+    - 170.82.0.0/16
+    - 170.83.0.0/16
+    - 177.10.0.0/16
+    - 177.124.0.0/16
+    - 177.125.0.0/16
+    - 177.128.0.0/16
+    - 177.129.0.0/16
+    - 177.131.0.0/16
+    - 177.152.0.0/16
+    - 177.185.0.0/16
+    - 177.223.0.0/16
+    - 177.23.0.0/16
+    - 177.47.0.0/16
+    - 177.54.0.0/16
+    - 177.55.0.0/16
+    - 177.66.0.0/16
+    - 177.67.0.0/16
+    - 177.72.0.0/16
+    - 177.73.0.0/16
+    - 177.74.0.0/16
+    - 177.84.0.0/16
+    - 177.86.0.0/16
+    - 177.87.0.0/16
+    - 179.0.0.0/16
+    - 179.189.0.0/16
+    - 179.191.0.0/16
+    - 179.42.0.0/16
+    - 179.48.0.0/16
+    - 179.63.0.0/16
+    - 186.193.0.0/16
+    - 186.194.0.0/16
+    - 186.195.0.0/16
+    - 186.219.0.0/16
+    - 186.224.0.0/16
+    - 186.233.0.0/16
+    - 186.235.0.0/16
+    - 186.250.0.0/16
+    - 187.180.0.0/16
+    - 187.183.0.0/16
+    - 187.62.0.0/16
+    - 187.84.0.0/16
+    - 187.85.0.0/16
+    - 189.113.0.0/16
+    - 189.6.0.0/16
+    - 190.89.0.0/16
+    - 191.177.0.0/16
+    - 191.242.0.0/16
+    - 191.243.0.0/16
+    - 191.37.0.0/16
+    - 191.7.0.0/16
+    - 192.141.0.0/16
+    - 201.182.0.0/16
+    - 201.71.0.0/16
+    - 45.160.0.0/16
+    - 45.161.0.0/16
+    - 45.162.0.0/16
+    - 45.163.0.0/16
+    - 45.164.0.0/16
+    - 45.165.0.0/16
+    - 45.166.0.0/16
+    - 45.167.0.0/16
+    - 45.169.0.0/16
+    - 45.170.0.0/16
+    - 45.173.0.0/16
+    - 45.176.0.0/16
+    - 45.177.0.0/16
+    - 45.178.0.0/16
+    - 45.179.0.0/16
+    - 45.180.0.0/16
+    - 45.181.0.0/16
+    - 45.182.0.0/16
+    - 45.183.0.0/16
+    - 45.184.0.0/16
+    - 45.185.0.0/16
+    - 45.186.0.0/16
+    - 45.187.0.0/16
+    - 45.188.0.0/16
+    - 45.191.0.0/16
+    - 45.225.0.0/16
+    - 45.226.0.0/16
+    - 45.228.0.0/16
+    - 45.229.0.0/16
+    - 45.230.0.0/16
+    - 45.231.0.0/16
+    - 45.233.0.0/16
+    - 45.234.0.0/16
+    - 45.235.0.0/16
+    - 45.237.0.0/16
+    - 45.238.0.0/16
+    - 45.239.0.0/16
+    - 45.4.0.0/16
+    - 45.5.0.0/16
+    - 45.6.0.0/16
+    - 45.71.0.0/16
 acmechief_host: deployment-acme-chief05.deployment-prep.eqiad1.wikimedia.cloud
 apt::use_experimental: true
 aptly::group: wikidev

Mentioned in SAL (#wikimedia-releng) [2025-04-23T22:15:07Z] <bd808> Forced puppet run and restarted varnish on deployment-cache-text08 to pick up a huge pile of new blocks (T392534)

Even blocking by class B is not making much of a dent. I am going to try an even bigger stick and block all class A CIDR blocks (/8) that had more than 500 requests in the last 50,000 requests.

I generated the list with this script:

big-ban-hammer.sh
#!/usr/bin/env bash
CUTOFF=${1:-500}
exec tail -50000 /var/log/apache2/other_vhosts_access-json.log |
        grep -oP '"X-Client-IP": "\d+\.' |
        grep -oP '\d+\.' |
        sort |
        uniq -c |
        sort -nr |
        awk -v cutoff=$CUTOFF '$1 > cutoff' |
        awk '{print $2}' |
        sort -n |
        awk '{print "    - " $1 "0.0.0/8"}'

I am also going to clear out the giant list of blocks that has been built up to this point when I apply this new set of ridiculously wide blocks. We will be able to get the historical blocks back via the history at https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+log/master/deployment-prep/_.yaml if needed.

https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/5666f32ebf5e2c6f4a62999ef6911d71081c1b48

diff --git a/deployment-prep/_.yaml b/deployment-prep/_.yaml
index 6522552..ba8f88c 100644
--- a/deployment-prep/_.yaml
+++ b/deployment-prep/_.yaml

@@ -1,221 +1,22 @@
 abuse_networks:
   blocked_nets:
     networks:
-    - 8.208.0.0/12
-    - 13.104.0.0/14
-    - 13.64.0.0/11
-    - 13.96.0.0/13
-    - 23.96.0.0/13
-    - 40.112.0.0/13
-    - 40.120.0.0/14
-    - 40.124.0.0/16
-    - 40.125.0.0/17
-    - 40.74.0.0/15
-    - 40.76.0.0/14
-    - 40.80.0.0/12
-    - 40.96.0.0/12
-    - 45.89.148.0/23
-    - 45.93.184.0/23
-    - 47.235.0.0/16
-    - 47.236.0.0/14
-    - 47.240.0.0/14
-    - 47.244.0.0/15
-    - 47.246.0.0/16
-    - 47.74.0.0/15
-    - 47.76.0.0/14
-    - 47.80.0.0/13
-    - 52.145.0.0/16
-    - 52.146.0.0/15
-    - 52.148.0.0/14
-    - 52.152.0.0/13
-    - 52.160.0.0/11
-    - 91.124.117.0/24
-    - 96.62.0.0/16
-    - 102.129.130.0/24
-    - 140.228.23.0/24
-    - 146.100.0.0/14
-    - 146.104.0.0/14
-    - 146.108.0.0/15
-    - 146.110.0.0/16
-    - 154.16.246.0/24
-    - 193.43.72.0/24
-    - 14.160.0.0/11
-    - 138.186.108.0/22
-    - 168.121.96.0/22
-    - 170.80.36.0/22
-    - 177.39.120.0/21
-    - 185.244.152.0/24
-    - 187.95.16.0/20
-    - 189.84.176.0/20
-    - 179.125.128.0/17
-    - 177.37.128.0/17
-    - 187.19.128.0/17
-    - 191.5.80.0/20
-    - 186.249.208.0/21
-    - 170.239.72.0/22
-    - 45.226.116.0/22
-    - 189.60.0.0/14
-    - 177.22.176.0/21
-    - 138.0.64.0/22
-    - 143.0.48.0/22
-    - 177.37.96.0/20
-    - 177.37.80.0/21
-    - 186.249.128.0/19
-    - 186.249.24.0/21
-    - 186.249.216.0/22
-    - 143.0.0.0/16
-    - 168.196.0.0/16
-    - 168.197.0.0/16
-    - 168.232.0.0/16
-    - 170.239.0.0/16
-    - 170.244.0.0/16
-    - 170.247.0.0/16
-    - 170.84.0.0/16
-    - 177.137.0.0/16
-    - 177.22.0.0/16
-    - 177.221.0.0/16
-    - 179.108.0.0/16
-    - 191.5.0.0/16
-    - 45.168.0.0/16
-    - 45.172.0.0/16
-    - 45.190.0.0/16
-    - 45.70.0.0/16
-    - 111.119.0.0/16
-    - 124.243.0.0/16
-    - 128.201.0.0/16
-    - 131.161.0.0/16
-    - 131.196.0.0/16
-    - 131.221.0.0/16
-    - 131.255.0.0/16
-    - 131.72.0.0/16
-    - 138.0.0.0/16
-    - 138.117.0.0/16
-    - 138.122.0.0/16
-    - 138.185.0.0/16
-    - 138.204.0.0/16
-    - 138.255.0.0/16
-    - 138.59.0.0/16
-    - 138.94.0.0/16
-    - 138.97.0.0/16
-    - 143.137.0.0/16
-    - 143.202.0.0/16
-    - 143.255.0.0/16
-    - 164.163.0.0/16
-    - 167.249.0.0/16
-    - 167.250.0.0/16
-    - 168.0.0.0/16
-    - 168.121.0.0/16
-    - 168.181.0.0/16
-    - 168.195.0.0/16
-    - 168.205.0.0/16
-    - 168.228.0.0/16
-    - 168.90.0.0/16
-    - 170.0.0.0/16
-    - 170.150.0.0/16
-    - 170.231.0.0/16
-    - 170.233.0.0/16
-    - 170.238.0.0/16
-    - 170.245.0.0/16
-    - 170.246.0.0/16
-    - 170.254.0.0/16
-    - 170.78.0.0/16
-    - 170.79.0.0/16
-    - 170.80.0.0/16
-    - 170.81.0.0/16
-    - 170.82.0.0/16
-    - 170.83.0.0/16
-    - 177.10.0.0/16
-    - 177.124.0.0/16
-    - 177.125.0.0/16
-    - 177.128.0.0/16
-    - 177.129.0.0/16
-    - 177.131.0.0/16
-    - 177.152.0.0/16
-    - 177.185.0.0/16
-    - 177.223.0.0/16
-    - 177.23.0.0/16
-    - 177.47.0.0/16
-    - 177.54.0.0/16
-    - 177.55.0.0/16
-    - 177.66.0.0/16
-    - 177.67.0.0/16
-    - 177.72.0.0/16
-    - 177.73.0.0/16
-    - 177.74.0.0/16
-    - 177.84.0.0/16
-    - 177.86.0.0/16
-    - 177.87.0.0/16
-    - 179.0.0.0/16
-    - 179.189.0.0/16
-    - 179.191.0.0/16
-    - 179.42.0.0/16
-    - 179.48.0.0/16
-    - 179.63.0.0/16
-    - 186.193.0.0/16
-    - 186.194.0.0/16
-    - 186.195.0.0/16
-    - 186.219.0.0/16
-    - 186.224.0.0/16
-    - 186.233.0.0/16
-    - 186.235.0.0/16
-    - 186.250.0.0/16
-    - 187.180.0.0/16
-    - 187.183.0.0/16
-    - 187.62.0.0/16
-    - 187.84.0.0/16
-    - 187.85.0.0/16
-    - 189.113.0.0/16
-    - 189.6.0.0/16
-    - 190.89.0.0/16
-    - 191.177.0.0/16
-    - 191.242.0.0/16
-    - 191.243.0.0/16
-    - 191.37.0.0/16
-    - 191.7.0.0/16
-    - 192.141.0.0/16
-    - 201.182.0.0/16
-    - 201.71.0.0/16
-    - 45.160.0.0/16
-    - 45.161.0.0/16
-    - 45.162.0.0/16
-    - 45.163.0.0/16
-    - 45.164.0.0/16
-    - 45.165.0.0/16
-    - 45.166.0.0/16
-    - 45.167.0.0/16
-    - 45.169.0.0/16
-    - 45.170.0.0/16
-    - 45.173.0.0/16
-    - 45.176.0.0/16
-    - 45.177.0.0/16
-    - 45.178.0.0/16
-    - 45.179.0.0/16
-    - 45.180.0.0/16
-    - 45.181.0.0/16
-    - 45.182.0.0/16
-    - 45.183.0.0/16
-    - 45.184.0.0/16
-    - 45.185.0.0/16
-    - 45.186.0.0/16
-    - 45.187.0.0/16
-    - 45.188.0.0/16
-    - 45.191.0.0/16
-    - 45.225.0.0/16
-    - 45.226.0.0/16
-    - 45.228.0.0/16
-    - 45.229.0.0/16
-    - 45.230.0.0/16
-    - 45.231.0.0/16
-    - 45.233.0.0/16
-    - 45.234.0.0/16
-    - 45.235.0.0/16
-    - 45.237.0.0/16
-    - 45.238.0.0/16
-    - 45.239.0.0/16
-    - 45.4.0.0/16
-    - 45.5.0.0/16
-    - 45.6.0.0/16
-    - 45.71.0.0/16
+    - 38.0.0.0/8
+    - 45.0.0.0/8
+    - 131.0.0.0/8
+    - 138.0.0.0/8
+    - 168.0.0.0/8
+    - 170.0.0.0/8
+    - 177.0.0.0/8
+    - 179.0.0.0/8
+    - 181.0.0.0/8
+    - 186.0.0.0/8
+    - 187.0.0.0/8
+    - 189.0.0.0/8
+    - 190.0.0.0/8
+    - 191.0.0.0/8
+    - 200.0.0.0/8
+    - 201.0.0.0/8

Mentioned in SAL (#wikimedia-releng) [2025-04-23T22:43:38Z] <bd808> Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534)

https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/f36af87c3ae9bcb2f7c0a78ecf6bdd21586687e9

diff --git a/deployment-prep/_.yaml b/deployment-prep/_.yaml
index ba8f88c..15b05b1 100644
--- a/deployment-prep/_.yaml
+++ b/deployment-prep/_.yaml

@@ -1,20 +1,28 @@
 abuse_networks:
   blocked_nets:
     networks:
+    - 14.0.0.0/8
     - 38.0.0.0/8
+    - 41.0.0.0/8
     - 45.0.0.0/8
+    - 47.0.0.0/8
+    - 102.0.0.0/8
     - 131.0.0.0/8
     - 138.0.0.0/8
+    - 143.0.0.0/8
+    - 152.0.0.0/8
     - 168.0.0.0/8
     - 170.0.0.0/8
     - 177.0.0.0/8
     - 179.0.0.0/8
     - 181.0.0.0/8
+    - 185.0.0.0/8
     - 186.0.0.0/8
     - 187.0.0.0/8
     - 189.0.0.0/8
     - 190.0.0.0/8
     - 191.0.0.0/8
+    - 197.0.0.0/8
     - 200.0.0.0/8
     - 201.0.0.0/8
 acmechief_host: deployment-acme-chief05.deployment-prep.eqiad1.wikimedia.cloud

Mentioned in SAL (#wikimedia-releng) [2025-04-23T22:59:51Z] <bd808> Forced puppet run and restarted varnish on deployment-cache-text08 to pick up new blocks (T392534)

https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/d9f0e372ce622f8e53a84b2dae9c99069ef8b54e

diff --git a/deployment-prep/_.yaml b/deployment-prep/_.yaml
index 15b05b1..01ac1d8 100644
--- a/deployment-prep/_.yaml
+++ b/deployment-prep/_.yaml

@@ -2,15 +2,21 @@
   blocked_nets:
     networks:
     - 14.0.0.0/8
+    - 37.0.0.0/8
     - 38.0.0.0/8
     - 41.0.0.0/8
     - 45.0.0.0/8
+    - 46.0.0.0/8
     - 47.0.0.0/8
     - 102.0.0.0/8
+    - 103.0.0.0/8
+    - 105.0.0.0/8
     - 131.0.0.0/8
     - 138.0.0.0/8
     - 143.0.0.0/8
     - 152.0.0.0/8
+    - 160.0.0.0/8
+    - 167.0.0.0/8
     - 168.0.0.0/8
     - 170.0.0.0/8
     - 177.0.0.0/8
@@ -19,9 +25,12 @@
     - 185.0.0.0/8
     - 186.0.0.0/8
     - 187.0.0.0/8
+    - 188.0.0.0/8
     - 189.0.0.0/8
     - 190.0.0.0/8
     - 191.0.0.0/8
+    - 192.0.0.0/8
+    - 196.0.0.0/8
     - 197.0.0.0/8
     - 200.0.0.0/8
     - 201.0.0.0/8

I think using new tasks each time will be better than keeping one open one in the longer term. Maybe after Wikimedia-Hackathon-2025 I will try to setup some nicer tools.

Thanks a lot for getting Beta back up, @bd808! I wanted to say this exercise is nearly identical to what I've done for XTools for years.

I am again using variations of tail -50000 /var/log/apache2/other_vhosts_access-json.log | grep -oP '"X-Client-IP": "\d+\.\d+\.\d+\.' |sort|uniq -c|sort -nr|head -n40 to find clusters of requests that look suspicious.

Wowweee! This makes it sooo much easier :)

Maybe after Wikimedia-Hackathon-2025 I will try to setup some nicer tools.

Please share those tools! Also, I'd be happy to work on it at the Hackathon, if you're interested.

Mentioned in SAL (#wikimedia-releng) [2025-04-24T15:32:22Z] <bd808> Punched a hole in the beta cluster network blocks to allow 47.144.0.0/12 through. (T392534)

bd808 closed subtask Restricted Task as Resolved.Apr 26 2025, 12:06 AM