Page MenuHomePhabricator
Create Task
Maniphest T273854

Automate regular WDQS query parsing and data-extraction
Closed, ResolvedPublic
Actions

Description

Once a CLI job is available to process wdqs-hourly, a job should be added to Airflow so that production event data is processed regularly (@dcausse we'll need your help for that one :).
Note: This task depends on T283255.

Details

SubjectRepoBranchLines +/-
Airflow dag to extract and process sparql querieswikimedia/discovery/analyticsmaster+221 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

JAllemandou created this task.Feb 4 2021, 9:32 AM
Restricted Application added a project: Wikidata. · View Herald TranscriptFeb 4 2021, 9:32 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
JAllemandou claimed this task.Feb 4 2021, 9:32 AM
• MPhamWMF moved this task from Incoming to Watching / Waiting on the Wikidata-Query-Service board.Feb 8 2021, 4:35 PM
• fdans triaged this task as High priority.Mar 1 2021, 5:03 PM
• fdans moved this task from Incoming to Smart Tools for Better Data on the Analytics board.
JAllemandou removed JAllemandou as the assignee of this task.May 20 2021, 4:24 PM
JAllemandou updated the task description. (Show Details)
JAllemandou added a parent task: T280640: [EPIC] Refine WDQS queries analysis.
JAllemandou updated the task description. (Show Details)
JAllemandou added a subscriber: dcausse.
JAllemandou mentioned this in T283258: Provide a job regularly deleting wdqs processed query after 90 days.May 20 2021, 4:28 PM
• AKhatun_WMF moved this task from Watching / Waiting to Analysis on the Wikidata-Query-Service board.May 27 2021, 2:06 PM
• AKhatun_WMF mentioned this in T283256: Extract operator/nodes/triples/paths/exprs list from queries.Jun 1 2021, 9:59 AM
• AKhatun_WMF claimed this task.Jun 4 2021, 7:21 AM
gerritbot added a comment.Jun 8 2021, 2:49 PM

Change 698799 had a related patch set uploaded (by AKhatun; author: AKhatun):

[wikimedia/discovery/analytics@master] Airflow dag to extract and process sparql queries

https://gerrit.wikimedia.org/r/698799

gerritbot added a project: Patch-For-Review.Jun 8 2021, 2:49 PM
dcausse mentioned this in T284575: Add akhatun to analytics-search-users group.Jun 8 2021, 3:26 PM
• MPhamWMF moved this task from Analysis to Current work on the Wikidata-Query-Service board.Jun 10 2021, 1:41 PM
• MPhamWMF added a project: Discovery-Search (Current work).
• MPhamWMF moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.Jun 10 2021, 3:07 PM
dcausse moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Jun 14 2021, 3:31 PM
gerritbot added a comment.Jun 21 2021, 5:42 PM

Change 698799 merged by jenkins-bot:

[wikimedia/discovery/analytics@master] Airflow dag to extract and process sparql queries

https://gerrit.wikimedia.org/r/698799

• AKhatun_WMF mentioned this in rWDAN58fc22bef150: Airflow dag to extract and process sparql queries.Jun 21 2021, 5:43 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 21 2021, 6:11 PM
Stashbot added a comment.Jun 21 2021, 6:22 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-21T18:22:57Z] <ebernhardson@deploy1002> Started deploy [wikimedia/discovery/analytics@40b4b2f]: T273854 Airflow dag to extract and process sparql queries

Stashbot added a comment.Jun 21 2021, 6:30 PM

Mentioned in SAL (#wikimedia-operations) [2021-06-21T18:30:08Z] <ebernhardson@deploy1002> Finished deploy [wikimedia/discovery/analytics@40b4b2f]: T273854 Airflow dag to extract and process sparql queries (duration: 07m 11s)

EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Jun 21 2021, 8:45 PM
• AKhatun_WMF moved this task from Current work to Analysis on the Wikidata-Query-Service board.Jun 22 2021, 7:46 AM
dcausse moved this task from Analysis to Current work on the Wikidata-Query-Service board.Jun 24 2021, 1:44 PM
Gehel closed this task as Resolved.Jul 26 2021, 12:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits