In response the findings in this ticket we have a couple of follow up questions that hopefully Traffic can help provide some insight to.
Summary of findings:
We have experienced a large (20%) increase in Unique Devices Year over Year for July 2024 and August 2024 in our monthly metrics which are computed from the unique_devices_per_project_family_monthly table. We noticed through our investigation that these increases were much less in the corresponding unique_devices_per_domain_monthly. Looking at the daily versions of these tables we noticed large spikes in unique devices on particular days such as August 14th 2024 July 14th. We narrowed down a large portion of the increase to apparently automated traffic flagged as 'is_redirect_to_pageview' (specifically status code 301) that made into the project_family table because our automata labelling heuristic is only applied to is_pageview= TRUE web requests. However, analysis revealed several anomalies/questions that we would like to explore as we continue our investigation:
- Is there an explanation for why there are users that apparently WMF-Last-Access-Global is set but not WMF-Last-Access and vice versa? It seems that Cookies should be set simultaneously such that any user should have a minimum a single pair of cookies.
select if(x_analytics_map['nocookies'] is null, 'accepts cookies', 'does not accept cookies') cookies,
length(x_analytics_map['WMF-Last-Access-Global']) global_length,
length(x_analytics_map['WMF-Last-Access']) local_length,
count(*)
FROM wmf.pageview_actor
WHERE x_analytics_map IS NOT NULL
AND agent_type = 'user'
AND is_pageview = TRUE
AND year = 2024 AND month = 8 AND day = 14 and hour = 10
group by x_analytics_map['nocookies'] is null,
length(x_analytics_map['WMF-Last-Access-Global']),
length(x_analytics_map['WMF-Last-Access'])
cookies global_length local_length count(1)
accepts cookies NULL NULL 141697
accepts cookies 11 11 15651640
does not accept cookies NULL NULL 2496307
accepts cookies 11 NULL 1810474
accepts cookies NULL 11 162134- Should every redirect correspond to a http_status code 200? The bulk of the automated requests that I mentioned ended as status 301 with no corresponding status 200. Our data shows they are being redirected to /wiki/a uri_path (probably from /wiki/A/) we did not see any 200 status that followed or preceded these requests in the webrequest data.
- Looking at the isp_data, or any other field, is there any way to determine the origin of the request? Even if it's a general determination of vpn, data center request, etc. We would appreciate help deciphering some of the fields in isp_data:
example:
ip isp organization autonomous_system_organization autonomous_system_number IP xxx.xx.xx SPTEL PTE. LTD. SPTEL PTE. LTD. SPTEL PTE. LTD. 23856
- What is the purpose of counting redirects? If a user is redirected from a desktop site (302) to the mobile site (200), then what's the problem of counting them only when they resolve as 200 on the correct mobile set? Global cookie being set there would still capture the domain and the project family since the mobile version of the site is still in the same project family. I can think of 2 answers: sometimes redirect fails and so we don't want to lose the original unique device as part of our count and the other reason is maybe a redirect can occur from one project family to another?
