Page MenuHomePhabricator
Create Task
Maniphest T222739

Move reportupdater queries from limn-* repositories to reportupdater-queries
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

See parent task and discussion.

List of reportupdater jobs and referenced repositories:

https://github.com/wikimedia/puppet/blob/0d0757f9836f893f6c4a1f2bf3f98b085c2a189d/modules/profile/manifests/reportupdater/jobs/mysql.pp

https://github.com/wikimedia/analytics-limn-flow-data/tree/master/flow
https://github.com/wikimedia/analytics-limn-flow-data/tree/master/flow-beta-features
https://github.com/wikimedia/analytics-limn-edit-data/tree/master/edit-beta-features
https://github.com/wikimedia/analytics-limn-language-data/tree/master/language
https://github.com/wikimedia/analytics-limn-language-data/tree/master/published_cx2_translations
https://github.com/wikimedia/analytics-limn-language-data/tree/master/mt_engines
https://github.com/wikimedia/analytics-limn-language-data/blob/master/cx
https://github.com/wikimedia/analytics-limn-ee-data/blob/master/ee
https://github.com/wikimedia/analytics-limn-ee-data/blob/master/ee-beta-features

Once done, add Cleanup task to clean up any deprecated repositories.

Details

SubjectRepoBranchLines +/-
Delete limn-language-data queries in favor of reportupdater-queriesanalytics/limn-language-datamaster+0 -494
Delete limn-ee-data queries in favor of reportupdater-queriesanalytics/limn-ee-datamaster+0 -204
Delete limn-edit-data queries in favor of reportupdater-queriesanalytics/limn-edit-datamaster+0 -39
Delete limn-flow-data queries in favor of reportupdater-queriesanalytics/limn-flow-datamaster+0 -215
ReportUpdater: change repo of all queries to reportupdater-queriesoperations/puppetproduction+1 -14
Add interlanguage hadoop queries to queries directoryanalytics/reportupdater-queriesmaster+75 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Milimetric triaged this task as Medium priority.May 7 2019, 4:22 PM
Milimetric created this task.
Milimetric moved this task from Incoming to Ops Week on the Analytics board.
• Nuria reassigned this task from Milimetric to • fdans.May 14 2019, 7:45 PM
• Nuria updated the task description. (Show Details)
• Nuria subscribed.

Assigning to @fdans, if we can do away with limn-* repositories we are set to move reportupdater queries away from accessing eventlogging data in hadoop entirely.

• Nuria added a project: Analytics-Kanban.May 15 2019, 10:13 PM
• Nuria added a parent task: T159170: Sunset MySQL data store for eventlogging.
Milimetric updated the task description. (Show Details)May 16 2019, 4:56 PM
• fdans moved this task from Next Up to In Progress on the Analytics-Kanban board.May 17 2019, 1:28 PM
• Nuria moved this task from In Progress to Paused on the Analytics-Kanban board.Jun 7 2019, 4:04 PM
• fdans moved this task from Paused to In Progress on the Analytics-Kanban board.Jun 14 2019, 2:57 PM
• fdans moved this task from In Progress to In Code Review on the Analytics-Kanban board.Jun 17 2019, 9:01 AM
• fdans moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Jun 19 2019, 4:00 PM
• Nuria moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Jun 21 2019, 4:02 PM
• Nuria moved this task from Done to Ready to Deploy on the Analytics-Kanban board.Jun 21 2019, 4:21 PM
• Nuria added a comment.Jun 27 2019, 6:51 PM

Ping @fdans should this ticket be in ready-to-deploy? I think is in-progress right? The one ready to deploy seems this one: T221064?

• Nuria added a comment.Jun 27 2019, 6:53 PM

I think queries have been moved but for this work to be ready-to-deploy more chnages are needed like changing the puppet of the reportupdater jobs?

• Nuria moved this task from Ready to Deploy to In Code Review on the Analytics-Kanban board.Jun 27 2019, 6:54 PM
• Nuria added a comment.Jun 28 2019, 4:37 PM

See code change: https://gerrit.wikimedia.org/r/#/c/analytics/reportupdater-queries/+/517084/ (merged on other task)

• Nuria added a comment.Jun 28 2019, 4:37 PM

Puppet code : https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/517085/

gerritbot added a comment.Jul 1 2019, 2:51 PM

Change 517085 had a related patch set uploaded (by Fdans; owner: Fdans):
[operations/puppet@production] ReportUpdater: change repo of all queries to reportupdater-queries

https://gerrit.wikimedia.org/r/517085

gerritbot added a project: Patch-For-Review.Jul 1 2019, 2:51 PM
gerritbot added a comment.Jul 3 2019, 1:10 PM

Change 520432 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/reportupdater-queries@master] Add interlanguage hadoop queries to queries directory

https://gerrit.wikimedia.org/r/520432

gerritbot added a comment.Jul 15 2019, 2:08 PM

Change 520432 merged by Ottomata:
[analytics/reportupdater-queries@master] Add interlanguage hadoop queries to queries directory

https://gerrit.wikimedia.org/r/520432

gerritbot added a comment.Jul 16 2019, 2:46 PM

Change 523731 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/limn-flow-data@master] Delete limn-flow-data queries in favor of reportupdater-queries

https://gerrit.wikimedia.org/r/523731

gerritbot added a comment.Jul 16 2019, 2:49 PM

Change 523732 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/limn-edit-data@master] Delete limn-edit-data queries in favor of reportupdater-queries

https://gerrit.wikimedia.org/r/523732

gerritbot added a comment.Jul 16 2019, 2:51 PM

Change 523733 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/limn-language-data@master] Delete limn-language-data queries in favor of reportupdater-queries

https://gerrit.wikimedia.org/r/523733

gerritbot added a comment.Jul 16 2019, 2:53 PM

Change 523734 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/limn-ee-data@master] Delete limn-ee-data queries in favor of reportupdater-queries

https://gerrit.wikimedia.org/r/523734

• Nuria added a comment.Jul 17 2019, 3:30 PM

I think that before we delete queries from repos we probably want to make sure this puppet change is deployed and working well, right? : https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/517085/

gerritbot added a comment.Jul 18 2019, 2:38 PM

Change 517085 merged by Ottomata:
[operations/puppet@production] ReportUpdater: change repo of all queries to reportupdater-queries

https://gerrit.wikimedia.org/r/517085

gerritbot added a comment.Jul 24 2019, 4:34 PM

Change 523731 merged by Ottomata:
[analytics/limn-flow-data@master] Delete limn-flow-data queries in favor of reportupdater-queries

https://gerrit.wikimedia.org/r/523731

gerritbot added a comment.Jul 24 2019, 4:34 PM

Change 523732 merged by Ottomata:
[analytics/limn-edit-data@master] Delete limn-edit-data queries in favor of reportupdater-queries

https://gerrit.wikimedia.org/r/523732

gerritbot added a comment.Jul 24 2019, 4:35 PM

Change 523734 merged by Ottomata:
[analytics/limn-ee-data@master] Delete limn-ee-data queries in favor of reportupdater-queries

https://gerrit.wikimedia.org/r/523734

gerritbot added a comment.Jul 24 2019, 4:35 PM

Change 523733 merged by jenkins-bot:
[analytics/limn-language-data@master] Delete limn-language-data queries in favor of reportupdater-queries

https://gerrit.wikimedia.org/r/523733

Stashbot added a comment.Jul 24 2019, 4:40 PM

Mentioned in SAL (#wikimedia-analytics) [2019-07-24T16:40:33Z] <ottomata> removed all non reportupdater-queries job repositories from /srv/reportupdater/jobs/ - T222739

Ottomata moved this task from Ops Week to Operational Excellence on the Analytics board.Aug 5 2019, 3:45 PM
• Nuria added a comment.Aug 21 2019, 5:43 PM

Ping @fdans what does remain to be done in this task?

• fdans added a comment.Aug 22 2019, 2:40 PM

@Nuria just the removal of repositories, but since there's tasks open for each of them, I think we can mark this one as done?

• Nuria added a comment.Aug 22 2019, 2:42 PM

@fdans sounds good, closing.

• Nuria closed this task as Resolved.Aug 22 2019, 2:42 PM
• Nuria set the point value for this task to 8.
phuedx mentioned this in T228975: Deletion of limn-language-data repository.Sep 17 2022, 6:57 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits