Page MenuHomePhabricator

Productionize Pivot UI
Closed, ResolvedPublic13 Story Points

Description

ASSUMPTION: this task should be very similar to productionizing AQS

  • puppetize Pivot UI (maybe service-runner?)
  • setup scap3 deployment (with repository, etc.)
  • maybe debianize stuff? We could also just deploy npm packages like we do with aqs
  • put behind LDAP
  • run on stat1001 (already has connection to Druid cluster because it's all part of the analytics cluster) or bohrium (easier to test?)

Details

Related Gerrit Patches:

Event Timeline

Nuria created this task.Jun 20 2016, 8:49 PM
Nuria added a comment.Jun 20 2016, 8:53 PM

Access should be restricted by LDAP

Milimetric triaged this task as Medium priority.Jul 7 2016, 5:33 PM
Milimetric moved this task from Incoming to Dashiki on the Analytics board.
Milimetric renamed this task from Puppetize pivot UI to Productionize Pivot UI.Aug 18 2016, 3:58 PM
Milimetric updated the task description. (Show Details)
Milimetric updated the task description. (Show Details)Aug 18 2016, 4:06 PM
Milimetric set the point value for this task to 13.
Nuria assigned this task to elukey.Aug 25 2016, 4:52 PM
Nuria edited projects, added Analytics-Kanban; removed Analytics.
JAllemandou moved this task from Next Up to Paused on the Analytics-Kanban board.Aug 30 2016, 10:07 AM
JAllemandou moved this task from Paused to Next Up on the Analytics-Kanban board.
elukey added a comment.EditedSep 2 2016, 1:04 PM

Marko gave me some reference of puppet code for a new service introduced: https://gerrit.wikimedia.org/r/#/c/305256/7 (deployed via scap)

Main problem now: https://github.com/github/dmca/blob/master/2016/2016-08-31-Facet.md

I am writing to request that Github take down the project at the following link: https://github.com/implydata/pivot.

This project infringes the copyright of Metamarkets Group, Inc. in the proprietary, 
interactive visualization software called Facet, described at https://metamarkets.com/2015/introducing-facet/. 
Facet was developed internally at Metamarkets and Metamarkets holds the copyright thereto. Facet is only available to Metamarkets customers who purchase a license to use Facet.

Change 309301 had a related patch set uploaded (by Elukey):
Add the pivot.wikimedia.org VHost to stat1001

https://gerrit.wikimedia.org/r/309301

Change 309301 merged by Elukey:
Add the pivot.wikimedia.org VHost to stat1001

https://gerrit.wikimedia.org/r/309301

Change 309312 had a related patch set uploaded (by Elukey):
Add the pivot.wikimedia.org domain

https://gerrit.wikimedia.org/r/309312

Change 309315 had a related patch set uploaded (by Elukey):
Add the pivot.w.o domain to the stat1001 misc Varnish director

https://gerrit.wikimedia.org/r/309315

Change 309315 merged by Elukey:
Add the pivot.w.o domain to the stat1001 misc Varnish director

https://gerrit.wikimedia.org/r/309315

Change 309312 merged by Elukey:
Add the pivot.wikimedia.org domain

https://gerrit.wikimedia.org/r/309312

Mentioned in SAL [2016-09-09T12:45:28Z] <elukey> running authdns-update on ns0.w.o to pick up the new domain pivot.wikimedia.org (T138262)

elukey added a comment.Sep 9 2016, 1:10 PM

https://pivot.wikimedia.org/ is up and running, at the moment this is the status:

Varnish --> stat1001 VHost (Basic Auth + LDAP) --proxied--> stat1001 (nodejs run using screen)

Next steps:

  1. set up gerrit repos (scap + code)
  2. puppet code to manage scap and systemd units on stat1001
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.Sep 14 2016, 3:06 PM

OOOk, Dan and I spent some time understanding how pivot initializes itself, and what would be required to integrate it with service-runner well. Its a little nasty. pivot is written in typescript, and the init code that it has doesn't have any built in way of requireing the files without having them manually inspect things like process.argv. We could write our own version that is able to configure pivot without doing that, and/or we could upstream those changes to imply-data (but there repo is down?).

I'm inclined to drop the service-runner integration for now. If/when pivot is back up, and if we really need service-runner for something slick (like easy node-clustering), then we can revisit this. For now, I'm for just doing a systemd unit around the pivot CLI like Luca was suggesting.

For future reference, this is how Andrew worked around the problems in Pivot's current code: https://gist.github.com/ottomata/b0b85d88f7f9dd0be5524adaf1e63253

If we go that way, I'd rather upstream it to Pivot itself. And we can't / shouldn't do that until their repo is back up on github. For now, yeah, I agree a simple systemd setup will meet our goal for this quarter.

Change 312495 had a related patch set uploaded (by Elukey):
First draft of the Pivot UI's puppetization

https://gerrit.wikimedia.org/r/312495

Change 312495 had a related patch set uploaded (by Elukey):
Imply Pivot UI's puppetization

https://gerrit.wikimedia.org/r/312495

Change 312495 merged by Elukey:
Introduce the Imply Pivot UI's module and role

https://gerrit.wikimedia.org/r/312495

Change 315047 had a related patch set uploaded (by Elukey):
Fix wrong Pivot dependency in its puppet class

https://gerrit.wikimedia.org/r/315047

Change 315047 merged by Elukey:
Fix wrong Pivot dependency in its puppet class

https://gerrit.wikimedia.org/r/315047

Change 315050 had a related patch set uploaded (by Elukey):
Updated scap configuration

https://gerrit.wikimedia.org/r/315050

Change 315050 merged by Elukey:
Updated scap configuration

https://gerrit.wikimedia.org/r/315050

Change 315056 had a related patch set uploaded (by Elukey):
Fix Scap name for the Analytics Pivot repository

https://gerrit.wikimedia.org/r/315056

Change 315056 merged by Elukey:
Fix Scap name for the Analytics Pivot repository

https://gerrit.wikimedia.org/r/315056

Change 315073 had a related patch set uploaded (by Elukey):
Fix the Pivot's systemd unit with the right exec path

https://gerrit.wikimedia.org/r/315073

Change 315073 merged by Elukey:
Fix the Pivot's systemd unit with the right exec path

https://gerrit.wikimedia.org/r/315073

Change 315081 had a related patch set uploaded (by Elukey):
Add the Druid's default port to Pivot's hieradata config

https://gerrit.wikimedia.org/r/315081

Change 315081 merged by Elukey:
Add the Druid's default port to Pivot's hieradata config

https://gerrit.wikimedia.org/r/315081

Change 315083 had a related patch set uploaded (by Elukey):
Move the httpd Proxy rules from stat1002 to localhost for Pivot

https://gerrit.wikimedia.org/r/315083

Change 315083 merged by Elukey:
Move the httpd Proxy rules from stat1002 to localhost for Pivot

https://gerrit.wikimedia.org/r/315083

Change 315222 had a related patch set uploaded (by Elukey):
Set SyslogIdentifier in Pivot's systemd unit

https://gerrit.wikimedia.org/r/315222

Change 315222 merged by Elukey:
Set SyslogIdentifier in Pivot's systemd unit

https://gerrit.wikimedia.org/r/315222

Pivot has been deployed via Scap to stat1001 and now it is serving pivot.wikimedia.org.

I updated https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Druid with basic admin info for pivot.

The only problem left is that version 0.9.39 seems to not visualize correctly data labels on the front page, meanwhile version 0.8.4 (the one previously running on stat1002 in screen) is not affected.

Change 315315 had a related patch set uploaded (by Mforns):
[WIP] Add data cube title to config

https://gerrit.wikimedia.org/r/315315

mforns added a subscriber: mforns.Oct 11 2016, 5:47 PM

@elukey @Milimetric
This ugly hack may solve the label problem, if nothing else does.
I haven't tested yet, but I thought it would be cool to have it here for you guys to see.
Feel free to abandon whenever we find a better solution.

it's not ugly, and should probably be upstreamed anyway if they haven't fixed it otherwise. I left a suggestion on the patch and we should totally do this if there's nothing easier in the druid config.

Change 315480 had a related patch set uploaded (by Elukey):
Add config file for the Pivot UI

https://gerrit.wikimedia.org/r/315480

Change 315480 merged by Elukey:
Add config file for the Pivot UI

https://gerrit.wikimedia.org/r/315480

Change 315481 had a related patch set uploaded (by Elukey):
Fix template render bug for the Pivot UI

https://gerrit.wikimedia.org/r/315481

Change 315481 merged by Elukey:
Fix template render bug for the Pivot UI

https://gerrit.wikimedia.org/r/315481

Change 315486 had a related patch set uploaded (by Elukey):
Fix remaining issues with the Pivot UI

https://gerrit.wikimedia.org/r/315486

Change 315486 merged by Elukey:
Fix remaining issues with the Pivot UI

https://gerrit.wikimedia.org/r/315486

@mforns: I added a simple config file to Pivot and now the labels are clearly visible, plus we can add a description to them (afaik it is not possible to add this data to druid).

I agree with @Milimetric though that your patch is good and should be upstreamed! Thanks a lot for the work :)

Change 315315 abandoned by Mforns:
[WIP] Add data cube title to config

Reason:
We solved this by adding a config file.

https://gerrit.wikimedia.org/r/315315

elukey moved this task from In Progress to Done on the Analytics-Kanban board.Oct 13 2016, 9:46 AM
Nuria closed this task as Resolved.Oct 19 2016, 7:23 PM