Page MenuHomePhabricator
Create Task
Maniphest T270103

Import commons mediainfo RDF dumps to hive
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

As a search engineer I want to have access to mediainfo RDF dumps in HDFS so that I can manipulate this data with hadoop and join with do joins with the wikidata RDF dumps.

There will be 3 steps to this task

AC:

  • mediainfo TTL dumps are imported and munged weekly in hdfs

Details

SubjectRepoBranchLines +/-
fix import commons ttl dagwikimedia/discovery/analyticsmaster+2 -2
airflow dag for commons dumpwikimedia/discovery/analyticsmaster+289 -132
create init dag for import ttlwikimedia/discovery/analyticsmaster+81 -4
Add import_commons_mediainfo_dumps to role::analytics_cluster::launcheroperations/puppetproduction+30 -7
Customize query in gerrit

Related Objects
Search...

Event Timeline

dcausse created this task.Dec 14 2020, 1:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 14 2020, 1:44 PM
CBogen triaged this task as High priority.Dec 14 2020, 4:15 PM
CBogen moved this task from needs triage to ML & Data Pipeline on the Discovery-Search board.
CBogen moved this task from ML & Data Pipeline to Current work on the Discovery-Search board.Jan 11 2021, 4:28 PM
CBogen edited projects, added Discovery-Search (Current work); removed Discovery-Search.
CBogen set the point value for this task to 8.Jan 11 2021, 4:56 PM
CBogen moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
Mstyles moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.Jan 15 2021, 11:27 PM
dcausse assigned this task to Mstyles.Jan 18 2021, 2:12 PM
gerritbot added a comment.Jan 26 2021, 12:14 AM

Change 658465 had a related patch set uploaded (by Mstyles; owner: Mstyles):
[wikimedia/discovery/analytics@master] airflow dag for commons dump

https://gerrit.wikimedia.org/r/658465

gerritbot added a project: Patch-For-Review.Jan 26 2021, 12:14 AM
Mstyles added a subscriber: elukey.Jan 27 2021, 6:11 PM

The rdf spark tools changes have been merged. The airflow work is in progress and waiting on @elukey to merge the puppet patch.

gerritbot added a comment.Jan 28 2021, 8:24 AM

Change 642411 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/puppet@production] Add import_commons_mediainfo_dumps to role::analytics_cluster::launcher

https://gerrit.wikimedia.org/r/642411

gerritbot added a comment.Jan 28 2021, 8:37 AM

Change 642411 merged by Elukey:
[operations/puppet@production] Add import_commons_mediainfo_dumps to role::analytics_cluster::launcher

https://gerrit.wikimedia.org/r/642411

gerritbot added a comment.Feb 17 2021, 2:57 AM

Change 664707 had a related patch set uploaded (by Mstyles; owner: Mstyles):
[wikimedia/discovery/analytics@master] create init dag for import ttl

https://gerrit.wikimedia.org/r/664707

Mstyles moved this task from In Progress to Waiting on the Discovery-Search (Current work) board.Feb 18 2021, 4:34 PM
gerritbot added a comment.Feb 18 2021, 6:19 PM

Change 664707 abandoned by Mstyles:
[wikimedia/discovery/analytics@master] create init dag for import ttl

Reason:
changes now a part of https://gerrit.wikimedia.org/r/c/wikimedia/discovery/analytics/ /658465

https://gerrit.wikimedia.org/r/664707

gerritbot added a comment.Feb 24 2021, 7:49 PM

Change 658465 merged by jenkins-bot:
[wikimedia/discovery/analytics@master] airflow dag for commons dump

https://gerrit.wikimedia.org/r/658465

Mstyles mentioned this in rWDAN403493430b5e: airflow dag for commons dump.Feb 24 2021, 7:57 PM
Stashbot added a comment.Feb 24 2021, 8:40 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-24T20:40:25Z] <mstyles@deploy1001> Started deploy [wikimedia/discovery/analytics@44fba51]: add import ttl dags - T270103

Stashbot added a comment.Feb 24 2021, 8:42 PM

Mentioned in SAL (#wikimedia-operations) [2021-02-24T20:42:59Z] <mstyles@deploy1001> Finished deploy [wikimedia/discovery/analytics@44fba51]: add import ttl dags - T270103 (duration: 02m 33s)

Mstyles moved this task from Waiting to To Be Deployed on the Discovery-Search (Current work) board.Feb 24 2021, 9:11 PM
Mstyles added a comment.Feb 24 2021, 9:13 PM

last step is to make sure that the airflow dags work next week and then this ticket should be good to be closed

Mstyles moved this task from To Be Deployed to Needs review on the Discovery-Search (Current work) board.Mar 1 2021, 5:50 PM
gerritbot added a comment.Mar 1 2021, 6:32 PM

Change 667683 had a related patch set uploaded (by Mstyles; owner: Mstyles):
[wikimedia/discovery/analytics@master] fix import commons ttl dag

https://gerrit.wikimedia.org/r/667683

gerritbot added a comment.Mar 1 2021, 8:57 PM

Change 667683 merged by jenkins-bot:
[wikimedia/discovery/analytics@master] fix import commons ttl dag

https://gerrit.wikimedia.org/r/667683

Mstyles mentioned this in rWDANca2c5b5f8b80: fix import commons ttl dag.Mar 1 2021, 9:21 PM
Mstyles moved this task from Needs review to To Be Deployed on the Discovery-Search (Current work) board.Mar 1 2021, 9:44 PM
Stashbot added a comment.Mar 1 2021, 9:52 PM

Mentioned in SAL (#wikimedia-operations) [2021-03-01T21:52:02Z] <mstyles@deploy1002> Started deploy [wikimedia/discovery/analytics@ca2c5b5]: import commons ttl dag fix (T270103)

Stashbot added a comment.Mar 1 2021, 9:54 PM

Mentioned in SAL (#wikimedia-operations) [2021-03-01T21:54:36Z] <mstyles@deploy1002> Finished deploy [wikimedia/discovery/analytics@ca2c5b5]: import commons ttl dag fix (T270103) (duration: 02m 34s)

Stashbot added a comment.Mar 1 2021, 10:39 PM

Mentioned in SAL (#wikimedia-operations) [2021-03-01T22:39:10Z] <mstyles@deploy1002> Started deploy [wikimedia/discovery/analytics@ca2c5b5]: import commons ttl dag fix (T270103)

Stashbot added a comment.Mar 1 2021, 10:41 PM

Mentioned in SAL (#wikimedia-operations) [2021-03-01T22:41:14Z] <mstyles@deploy1002> Finished deploy [wikimedia/discovery/analytics@ca2c5b5]: import commons ttl dag fix (T270103) (duration: 02m 04s)

Mstyles moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Mar 2 2021, 1:11 AM
Gehel closed this task as Resolved.Mar 8 2021, 1:08 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 12 2021, 10:00 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits