Page MenuHomePhabricator

Create a repository and user for Product Analytics Oozie jobs
Closed, ResolvedPublic

Description

Event Timeline

mforns created this task.Aug 19 2019, 5:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 19 2019, 5:56 PM
fdans added a subscriber: fdans.

Tagging Release Engineering to create the repo.

example https://github.com/wikimedia/wikimedia-discovery-analytics

greg added a subscriber: greg.

Release Engineering recommends and supports Gerrit code review, is that what you wanted?

https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests

@mforns: Can you please reply to the previous comment? Thanks! (Also, what is a "user" in which specific system?)

@Aklapper @greg

Can you please reply to the previous comment? Thanks! (Also, what is a "user" in which specific system?)

Sorry for the ambiguity.
Yes, we'd like a new repo in Gerrit. I'll tag and ping ProductAnalytics for them to Request one in:
https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests

Regarding the user, I meant a Hadoop user, not to be done by Release Engineering.

Thanks!

@kzimmerman
Can you Product Analytics please, as Greg suggests, request a repository in Gerrit to store your team's Oozie jobs?
https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests
I can do that as well, but I thought you'd like to chose a repo name, owner and type.
Cheers

kzimmerman triaged this task as Medium priority.
kzimmerman reassigned this task from kzimmerman to mpopov.Mar 17 2020, 5:43 PM
mpopov added a comment.EditedMar 19 2020, 1:17 PM

Requested analytics/wmf-product on the new repos page

mpopov added a comment.EditedApr 13 2020, 4:50 PM

Update: Gerrit repo analytics/wmf-product has been created for us to use. I've renamed the owner group to "wmf-product-analytics" and added those of us on the team who have Gerrit accounts.

Change 589307 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[analytics/wmf-product@master] Add README

https://gerrit.wikimedia.org/r/589307

Change 589307 merged by Bearloga:
[analytics/wmf-product@master] Add README

https://gerrit.wikimedia.org/r/589307

Change 589320 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[operations/puppet@production] Add analytics-product system user

https://gerrit.wikimedia.org/r/589320

Change 589320 abandoned by Bearloga:
Add analytics-product system user

Reason:
Going to upload a different patch based on Luca's recent refactoring work

https://gerrit.wikimedia.org/r/589320

@mpopov should we not use a repo like analytics/wmf-product/jobs? We'll want repos for other things as well; I'll want analytics/wmf-product/wmfdata-python, for example 🙂

Change 595540 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[operations/puppet@production] Add analytics-product system user

https://gerrit.wikimedia.org/r/595540

mpopov moved this task from Triage to Upcoming Quarter on the Product-Analytics board.
kzimmerman renamed this task from Create a repository and user for Product Analytics Oozie jobs? to Create a repository and user for Product Analytics Oozie jobs.Jul 20 2020, 4:30 PM
mpopov reassigned this task from mpopov to elukey.EditedJul 21 2020, 12:25 PM
mpopov added a project: Analytics.
mpopov added subscribers: elukey, mpopov.

@elukey: Is it okay if I assign you for the user creation part? (The repository part was done months ago.)

Note: we will use analytics-privatedata user for now

mpopov updated the task description. (Show Details)Jul 21 2020, 12:26 PM
mpopov updated the task description. (Show Details)

it is yes, I'll bring this up during the next SRE meeting! Thanks for the patience :)

fdans edited projects, added Analytics-Radar; removed Analytics.Jul 27 2020, 3:41 PM

Change 595540 merged by Elukey:
[operations/puppet@production] Add analytics-product system user

https://gerrit.wikimedia.org/r/595540

Just completed the creation of the analytics-product system user + kerberos keytab, you should now be able to use it :)

mpopov closed this task as Resolved.Jul 29 2020, 3:05 PM
mpopov updated the task description. (Show Details)

Thanks, Luca! Just ran sudo -u analytics-product kerberos-run-command analytics-product hdfs dfs -ls / on all the stat nodes and no problems there :)

nshahquinn-wmf reopened this task as Open.EditedAug 4 2020, 7:11 PM

@mpopov should we not use a repo like analytics/wmf-product/jobs? We'll want repos for other things as well; I'll want analytics/wmf-product/wmfdata-python, for example 🙂

@mpopov, I'm still wondering about this.

Technically, it looks like Gerrit allows us to use analytics/wmf-product as a repo and still nest other repos within it. But it doesn't seem like it would work in GitLab (which treats "groups" and "projects" separately). And even if worked, it would still be pretty confusing for analytics/wmf-product to be a repo for jobs while, say, analytics/wmf-product/wmfdata-python is a totally separate project!

mpopov added a comment.Aug 5 2020, 2:49 PM

@nshahquinn-wmf: That's an excellent point! I updated the task description with the analytics/wmf-product/jobs repo idea. I like that. Should we just use the analytics/wmf-product root repo as a convenient index of "sub-repos" like analytics/wmf-product/jobs and analytics/wmf-product/wmfdata-python?

@nshahquinn-wmf: That's an excellent point! I updated the task description with the analytics/wmf-product/jobs repo idea. I like that. Should we just use the analytics/wmf-product root repo as a convenient index of "sub-repos" like analytics/wmf-product/jobs and analytics/wmf-product/wmfdata-python?

Using it as an index definitely makes sense. I wish Gerrit did that automatically, but since it doesn't, manual it is 😁 For now, there won't be anything to index other than the jobs repo, but no reason not to get started!

mpopov updated the task description. (Show Details)Aug 5 2020, 3:20 PM
mpopov added a comment.Aug 5 2020, 3:26 PM

Requested analytics/wmf-product/jobs Gerrit repo

QChris added a subscriber: QChris.Aug 6 2020, 5:53 AM

Requested analytics/wmf-product/jobs Gerrit repo

Done. Created at https://gerrit.wikimedia.org/r/#/admin/projects/analytics/wmf-product/jobs

mpopov closed this task as Resolved.Aug 6 2020, 1:59 PM
mpopov updated the task description. (Show Details)
mpopov updated the task description. (Show Details)

Thank you @QChris!