Update:

Data retention has been changed from 90 days to 180 days:
Some data loss occurred already so data starts at 2024-07-03T08:00.

The proposed plan by DPE is to apply the automata labelling logic to is_redirect_to_pageview web traffic without altering our automata heuristics or altering the heuristics applied to pageviews. The resulting list of automated actor signatures will be Unioned with the original list of bot actors and then the unique devices will be repopulated. The heuristics will be applied to the redirect traffic separately so that the additional page requests that are added to an actor signature from the redirects does not trigger heuristics accidentally.

The next steps will be to conduct Initial analysis to be done by @JAllemandou to assess the impact of the proposed change using 2-3 days of test data and will try to provide an update sometime next week. Ideally, the spike in Unique Devices should flatten out.

DPE assessed that we will not be able to re-run the pipeline for the last 3 months because it would be too expensive. More likely we will be applying the fix for Nov 1st onward if we don't notice a spike in October, otherwise the patch will be applied to the current month. In any case, the previously effected months will be asterisked.

Update:

• Testing on 2 days of data: July 15 and Aug 6 was completed.
  • Joseph: observed that unique devices inflation was reduced and unique devices for US enwiki was back to normal.
  • The data generated is available here :
    • joal.webrequest_actor_metrics_hourly_redirects - The hourly actor-metrics computed for redirect-to-pageviews only (2 days for each day of analysis, the actual day and the previous - 2024-07-[14/15] and 2024-08-[05/06]).
    • joal.webrequest_actor_metrics_rollup_hourly_redirects - The hourly 24h rollup of actor metrics for redirect-to-pageviews only (the actual 2 days of analysis 2024-07-15 and 2024-08-06).
    • joal.webrequest_actor_label_hourly_redirects - The labels computed on redirect-to-pageviews actors (2 days of analysis 2024-07-15 and 2024-08-06)
    • joal.pageview_actor The pageview-actors with the new labelled agents: same automated actors as production (from pageview), augmented with actors from redirects (for 2 days of analysis).
    • joal.unique_devices_per_project_family_daily - Same exact unique computation as production, except computed on the previously generated pageview_actor data. This table shows the corrected number (2 days of analysis).
• Changes will also include a New column added to all 3 actor_label tables: is_redirect.
  • This means each actor will have 2 rows one for pageview automated and one for redirect automated
  • this could increase data volume by 1.5x or 2x

• @JAllemandou and @mforns have committed to implementing the final change on Nov 1, 2024

• Meanwhile in T376882 , DPE are also fixing the issues caused due to missing webrequest logs which has been affecting pageviews since May 2023. This impacts <0.5% of overall pageviews per month.