Page MenuHomePhabricator
Create Task
Maniphest T415874

Extract bot classification into new private repo
Closed, ResolvedPublic
Actions

Description

In our revamping of bot classification, we would like to extract and centralize the new process into its own repo, private.

Linked with:

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Move bot detection pipeline into new repoanalytics/refinerymaster+0 -608
Bump Airflow image to include missing jarsoperations/deployment-chartsmaster+1 -1
dse-k8s-services Airflow: Bump imageoperations/deployment-chartsmaster+1 -5
dse-k8s airflow-analytics-test: Bump imageoperations/deployment-chartsmaster+4 -0
dse-k8s-services Blunderbuss: Bump imageoperations/deployment-chartsmaster+1 -1
Bump Blunderbuss imageoperations/deployment-chartsmaster+1 -1
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Use main image in canary events dagrepos/data-engineering/airflow-dags!2063aquT415874_use_main_airflow_image_in_canary_events_dagmain
Fix Airflow image with missing refinery jarsrepos/data-engineering/airflow-dags!2062aquT415874_fix_airflow_imagemain
bot-detection pipeline fixrepos/data-engineering/airflow-dags!2059aquT415874_switch_bot_pipeline_repository_fix2main
Hotfix: Remove optimization on webrequest_actor pipelinerepos/data-engineering/airflow-dags!2058aquT415874_switch_bot_pipeline_repository_fixmain
analytics-test: Fix a dag of test using new Airflow imagerepos/data-engineering/airflow-dags!2054aquT415874_fix_analytics_test_test_dagmain
Bump workflow-utils from 0.30.0 to 0.32.0repos/data-engineering/airflow-dags!2051aquT415874_bump_prod_image_with_workflow_utilsmain
Update blunderbuss with last version of workflow_utilsrepos/data-engineering/blunderbuss!15aquT415874_workflow_utils_bumpmain
Fix fsspec_options passingrepos/data-engineering/workflow_utils!64aquT415874_pass_fsspec_options_to_fsmain
Bump blunderbuss bugler versionrepos/data-engineering/airflow-dags!2044aquT415874_bump_blunderbuss_bugler_1.5.2main
Print debug lines about env varsrepos/data-engineering/blunderbuss-bugler!24aquT415874_add_debug_printingmain
Bump blunderbuss Bugler to support extra parameterrepos/data-engineering/airflow-dags!2043aquT415874_bump_blunderbuss_buglermain
Analytics-test: Test new Airflow image with bump in workflow-utilsrepos/data-engineering/airflow-dags!2035aquT415874_test_new_docker_image_in_analytics_testmain
Bump workflow-utils from 0.13 to 0.30.0repos/data-engineering/airflow-dags!2029aquT415874_bump_workflow_utils_in_mainmain
Add Jinja2 rendering of artifact_sources config via blunderbuss_bugler_env_varsrepos/data-engineering/blunderbuss!13aquT415874_private_repo_supportmain
Add fsspec_options parameter to FsArtifactSourcerepos/data-engineering/workflow_utils!59aquT415874_pass_custom_headers_for_urlmain
Add blunderbuss_bugler_env_vars to request payloadrepos/data-engineering/blunderbuss-bugler!22aquT415874_private_repo_supportmain
Use refinery-private archive for webrequest actor HQL scriptsrepos/data-engineering/airflow-dags!1988aquT415874_switch_bot_pipeline_repositorymain
Show related patches Customize query in GitLab

Related Objects

Event Timeline

Antoine_Quhen created this task.Jan 29 2026, 8:35 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 29 2026, 8:35 AM
Antoine_Quhen moved this task from Next Up to In progress on the Data-Engineering (Q3 FY25/26 January 1st - March 31th) board.Jan 29 2026, 8:36 AM
CodeReviewBot added a project: Patch-For-Review.Feb 9 2026, 3:13 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1988

Use bot-detection archive for webrequest actor HQL scripts

gerritbot added a comment.Feb 9 2026, 3:17 PM

Change #1237928 had a related patch set uploaded (by Aqu; author: Aqu):

[analytics/refinery@master] Move bot detection pipeline into new repo

https://gerrit.wikimedia.org/r/1237928

xcollazo moved this task from In progress to In Review on the Data-Engineering (Q3 FY25/26 January 1st - March 31th) board.Feb 9 2026, 4:26 PM
CodeReviewBot added a comment.Feb 12 2026, 10:00 AM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/blunderbuss-bugler/-/merge_requests/22

Add blunderbuss_bugler_env_vars to request payload

CodeReviewBot added a comment.Feb 12 2026, 12:15 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils/-/merge_requests/59

Add fsspec_options parameter to FsArtifactSource

CodeReviewBot added a comment.Feb 12 2026, 3:14 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/blunderbuss/-/merge_requests/13

Add Jinja2 rendering of artifact_sources config via blunderbuss_bugler_env_vars

CodeReviewBot added a comment.Feb 23 2026, 9:30 PM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/blunderbuss-bugler/-/merge_requests/22

Add blunderbuss_bugler_env_vars to request payload

CodeReviewBot added a comment.Feb 24 2026, 10:25 AM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/blunderbuss/-/merge_requests/13

Add Jinja2 rendering of artifact_sources config via blunderbuss_bugler_env_vars

CodeReviewBot added a comment.Feb 24 2026, 1:16 PM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils/-/merge_requests/59

Add fsspec_options parameter to FsArtifactSource

gerritbot added a comment.Feb 24 2026, 8:53 PM

Change #1243221 had a related patch set uploaded (by Aqu; author: Aqu):

[operations/deployment-charts@master] Bump Blunderbuss image

https://gerrit.wikimedia.org/r/1243221

gerritbot added a comment.Feb 25 2026, 5:10 PM

Change #1243221 merged by jenkins-bot:

[operations/deployment-charts@master] Bump Blunderbuss image

https://gerrit.wikimedia.org/r/1243221

CodeReviewBot added a comment.Feb 27 2026, 10:07 AM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2029

Bump workflow-utils from 0.13 to 0.30.0

CodeReviewBot added a comment.Mar 3 2026, 10:15 AM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2029

Bump workflow-utils from 0.13 to 0.30.0

gerritbot added a comment.Mar 3 2026, 11:45 AM

Change #1247570 had a related patch set uploaded (by Aqu; author: Aqu):

[operations/deployment-charts@master] dse-k8s airflow-analytics-test: Bump image

https://gerrit.wikimedia.org/r/1247570

CodeReviewBot added a comment.Mar 3 2026, 2:43 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2035

Analytics-test: Test new Airflow image with bump in workflow-utils

CodeReviewBot added a comment.Mar 3 2026, 4:00 PM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2035

Analytics-test: Test new Airflow image with bump in workflow-utils

CodeReviewBot added a comment.Mar 3 2026, 4:15 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2043

Bump blunderbuss Bugler to support extra parameter

CodeReviewBot added a comment.Mar 3 2026, 4:41 PM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2043

Bump blunderbuss Bugler to support extra parameter

CodeReviewBot added a comment.Mar 3 2026, 5:57 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/blunderbuss-bugler/-/merge_requests/24

Print debug lines about env vars

CodeReviewBot added a comment.Mar 3 2026, 6:11 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2044

Bump blunderbuss bugler version

CodeReviewBot added a comment.Mar 3 2026, 7:55 PM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2044

Bump blunderbuss bugler version

CodeReviewBot added a comment.Mar 3 2026, 10:31 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils/-/merge_requests/64

Fix fsspec_options passing

CodeReviewBot added a comment.Mar 4 2026, 1:54 PM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils/-/merge_requests/64

Fix fsspec_options passing

CodeReviewBot added a comment.Mar 4 2026, 2:13 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/blunderbuss/-/merge_requests/15

Update blunderbuss with last version of workflow_utils

CodeReviewBot added a comment.Mar 4 2026, 2:29 PM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/blunderbuss/-/merge_requests/15

Update blunderbuss with last version of workflow_utils

gerritbot added a comment.Mar 4 2026, 2:37 PM

Change #1248015 had a related patch set uploaded (by Aqu; author: Aqu):

[operations/deployment-charts@master] dse-k8s-services Blunderbuss: Bump image

https://gerrit.wikimedia.org/r/1248015

gerritbot added a comment.Mar 4 2026, 2:42 PM

Change #1248015 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s-services Blunderbuss: Bump image

https://gerrit.wikimedia.org/r/1248015

CodeReviewBot added a comment.Mar 5 2026, 9:22 AM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2051

Bump workflow-utils from 0.30.0 to 0.32.0

CodeReviewBot added a comment.Mar 5 2026, 9:48 AM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2051

Bump workflow-utils from 0.30.0 to 0.32.0

gerritbot added a comment.Mar 5 2026, 10:52 AM

Change #1247570 merged by Brouberol:

[operations/deployment-charts@master] dse-k8s airflow-analytics-test: Bump image

https://gerrit.wikimedia.org/r/1247570

CodeReviewBot added a comment.Mar 5 2026, 4:04 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2054

analytics-test: test new Airflow image

CodeReviewBot added a comment.Mar 6 2026, 10:13 AM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2054

analytics-test: Fix a dag of test using new Airflow image

gerritbot added a comment.Mar 9 2026, 8:09 AM

Change #1249094 had a related patch set uploaded (by Aqu; author: Aqu):

[operations/deployment-charts@master] dse-k8s-services Airflow: Bump image

https://gerrit.wikimedia.org/r/1249094

gerritbot added a comment.Mar 9 2026, 8:19 AM

Change #1249094 merged by Brouberol:

[operations/deployment-charts@master] dse-k8s-services Airflow: Bump image

https://gerrit.wikimedia.org/r/1249094

Antoine_Quhen renamed this task from Extract bot classification into new repo to Extract bot classification into new private repo.Mar 9 2026, 8:59 AM
Antoine_Quhen updated the task description. (Show Details)
Antoine_Quhen added a comment.Mar 9 2026, 9:19 AM

Summary of the work done here.

CodeReviewBot added a comment.Mar 9 2026, 12:31 PM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1988

Use refinery-private archive for webrequest actor HQL scripts

CodeReviewBot added a comment.Mar 9 2026, 1:10 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2058

Hotfix: Remove optimization on webrequest_actor pipeline

CodeReviewBot added a comment.Mar 9 2026, 1:19 PM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2058

Hotfix: Remove optimization on webrequest_actor pipeline

CodeReviewBot added a comment.Mar 9 2026, 3:29 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2059

bot-detection pipeline fix

Ottomata updated the task description. (Show Details)Mar 9 2026, 3:58 PM
CodeReviewBot added a comment.Mar 9 2026, 9:13 PM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2062

Fix Airflow image

CodeReviewBot added a comment.Mar 9 2026, 9:33 PM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2062

Fix Airflow image with missing refinery jars

CodeReviewBot added a comment.Mar 10 2026, 11:39 AM

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2063

Use main image in canary events dag

gerritbot added a comment.Mar 10 2026, 11:44 AM

Change #1249936 had a related patch set uploaded (by Aqu; author: Aqu):

[operations/deployment-charts@master] Bump Airflow image to include missing jars

https://gerrit.wikimedia.org/r/1249936

gerritbot added a comment.Mar 11 2026, 8:11 AM

Change #1249936 merged by Brouberol:

[operations/deployment-charts@master] Bump Airflow image to include missing jars

https://gerrit.wikimedia.org/r/1249936

CodeReviewBot added a comment.Mar 11 2026, 3:19 PM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2063

Use main image in canary events dag

gerritbot added a comment.Mar 12 2026, 10:12 AM

Change #1237928 merged by Aqu:

[analytics/refinery@master] Move bot detection pipeline into new repo

https://gerrit.wikimedia.org/r/1237928

CodeReviewBot added a comment.Mar 18 2026, 9:16 AM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2059

bot-detection pipeline fix

Antoine_Quhen closed this task as Resolved.Mar 18 2026, 1:27 PM
Antoine_Quhen moved this task from In Review to Done on the Data-Engineering (Q3 FY25/26 January 1st - March 31th) board.

The legacy pipeline has been extracted, and removed from analytics-refinery.

Antoine_Quhen reopened this task as In Progress.Mar 18 2026, 1:30 PM
Ahoelzl edited projects, added Data-Engineering (Q4 FS25/26 April 1st - June 30st); removed Data-Engineering (Q3 FY25/26 January 1st - March 31th).Fri, Apr 3, 10:27 PM
Ahoelzl moved this task from Next Up to Done on the Data-Engineering (Q4 FS25/26 April 1st - June 30st) board.
Ahoelzl closed this task as Resolved.Fri, Apr 3, 10:29 PM
Ahoelzl edited projects, added Data-Engineering (Q3 FY25/26 January 1st - March 31th); removed Data-Engineering (Q4 FS25/26 April 1st - June 30st).Fri, Apr 3, 10:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits