In T373630#10226978, @Hghani wrote:
Problem Statement
Based on further analysis, we've identified that certain actor signatures identified in the findings below are generating highly suspicious traffic. Specifically, they are sending requests every few seconds and to a different domain with each request. Upon reviewing our automated filtering logic, it appears that we apply automata labelling at the actor_signature level. This means our current heuristics may fail to label traffic accurately when users frequently switch domains, as each new domain visit generates a different actor signature, resetting metrics like pages per minute or total pageviews.
This oversight is likely a major factor behind the nearly 2000% increase in unique devices from Singapore, as these actor signatures are bypassing our automata filters. In addition to the redirect problem discussed earlier in this ticket, we believe this pattern accounts for a significant portion of the overall increase in unique devices we've observed.
To address this, we propose applying automata labelling at the actor_signature_project_family level. This would continue to capture automata behavior at the domain level while also including actors who evade detection by switching domains. After discussing this with @JAllemandou , he suggested this approach sounds reasonable, but it would require a thorough impact analysis before implementation.
@JAllemandou I'd like to add some new findings from looking at Singapore's unique device increase which extends from a discussion with traffic.
For September 2024 we see about 1800% increase in Unique Devices from Singapore, YoY which puts total unique devices counted in Singapore at 125,911,935 for a population of 5 million, which is about 36% of total new unique devices year-over-year (even larger contribution to total uniques than USA uniques). The majority (at peak 99% of unique device: see chart below) of these uniques, similar to last month and the month before, are coming from Cloud based ISPs such as Huawei Cloud (specifically: looking at Huawei Cloud (AS136907)). Many of these are actor signatures with thousands of uniques attributed to them.
Consulting with @Vgutierrez, we believe a large chunk of this traffic is likely automated because:
a) Lack of referrer header.
b) UA looks like it's being faked.
c) The provider is Cloud infrastructure.
d) the volume of unique devices is basically impossible to be generated legitimately from Singapore.This increase is occurring on pageviews and so the increase appears in both the project domain and project family tables which means if this traffic is automated, it is not being captured by our automata filter. It also means it is likely separate issue from the redirect problem identified above, and given the increases we are seeing in Singapore every month since Jan 2024, this number may continue to explode higher.
September will be another month where we will see about 18% YoY increase in Uniques and it seems like a significant portion is coming from Singapore this month. I’m not sure if there’s an easy fix, given that this traffic is bypassing our automata filter and we’re uncertain of its exact nature. However, the significant impact this suspicious traffic is having on the increase in unique devices adds another piece to the puzzle.
What needs to be done
- In the Automata pipeline, starting with webrequest_actor_metrics_hourly, replace line#39 get_actor_signature to get project family actor signature.
- This will ensure that automata logic is applied at the project family level instead of domain level. Project family level is a super set that will filter automated bots at both project and domain levels. So this logic change works for existing and desired use case
- Next the pageview actor table should use the actor_signature_per_project_family to join with automata label table to filter out automated actors at the project level pageview_actor.hql#74