Page MenuHomePhabricator

Sudden surge of requests to https://wikipedia.org/ from Telus customers
Open, LowPublic

Description

At 09:36 this morning, @elukey noticed a sudden surge in the requests to text-varnish in ulsfo

Looking at the sampled-1000 data, I found ~ 120k requests with the following characteristics:

  • All for https://wikipedia.org
  • All from IPs from telus.net dynamic ip pools, see https://w.wiki/33T9 (restricted to NDA)
  • All followed the redirect to https://www.wikipedia.org

*All with user-agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36

Of note: Chrome 29, according to https://en.wikipedia.org/wiki/Google_Chrome_version_history was released in august 2013.

It looks like some coordinated request, possibly non-malicious, like a connection test from routers. There is a similar spike at around the same time during the Pacific night, between 8 and 9 UTC on January 7th, but I'm not sure if we can still verify what was in those requests.

Event Timeline

Joe triaged this task as Low priority.Mar 2 2021, 11:41 AM
Joe created this task.

Over 30 days there were some other related spikes (use a fine grained 5 minutes aggregate instead of daily ones) https://w.wiki/33TA Apparently hitting same URLs. Last was on Feb 23 around 19:00 UTC.

Did this cause any actual issue?

Did this cause any actual issue?

No actual issue, hence the low priority. I anyhow thought it would be to have a task opened with what we found so if it re-happens we know what's going on. I don't think it's worth reaching out to their abuse right now, just to be clear, although this is clearly useless traffic it's a tiny html page that is easily cached.

There is a similar spike at around the same time during the Pacific night, between 8 and 9 UTC on January 7th, but I'm not sure if we can still verify what was in those request

Used the following spark sql script from sta1004:

scala> spark.sql("SELECT count(*) as count, uri_host, uri_path, user_agent FROM wmf.webrequest where webrequest_source='text' and year=2021 and month=1 and day=7 and hour=8 and hostname LIKE 'cp%.ulsfo.wmnet' GROUP BY uri_host, uri_path, user_agent  ORDER BY count DESC LIMIT 10").show(false)

But I didn't notice the signature outlined in the description (I double checked the same sql query with 2021-03-21T09:36 and I did confirm what Joe posted on the description). The top talkers for January were iPhone-related UAs for ja/en mobile wikipedias.