/srv/published should be structured similarly, have identical README across stat hosts describing said structure
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	mpopov
	Jun 1 2020, 6:44 PM

Description

stat1004:~$ ls /srv/published
datasets  README

stat1005:~$ ls /srv/published
datasets  notebooks  README

stat1007:~$ ls /srv/published
dashboards  datasets  notebooks  README  reports  test

stat1006:~$ ls /srv/published
datasets  README

stat1008:~$ ls /srv/published/
README

Perhaps we could have standardized /srv/published/[datasets|reports|notebooks|dashboards] on all hosts + README with hierarchy description & guidance, managed by Puppet?

A README with guidance on the different subdirectories would be helpful for users deciding, for example, whether to put their Jupyter-based report in .../notebooks or .../reports. Eventually, I'd love to have a nbviewer service for automatically rendering raw jupyter notebooks in published/notebooks and then published/reports could be for pre-rendered HTML reports (from Jupyter, R Markdown, or other)

Event Timeline

mpopov created this task.Jun 1 2020, 6:44 PM

mpopov updated the task description. (Show Details)

nshahquinn-wmf awarded a token.Jun 2 2020, 4:38 PM

nshahquinn-wmf unsubscribed.

nshahquinn-wmf subscribed.

This is not urgent and we also need to clarify the right approach; we're moving this to Upcoming Quarter to revisit in the new fiscal year.

kzimmerman moved this task from Triage to Upcoming Quarter on the Product-Analytics board.Jun 2 2020, 5:12 PM

we fully support the beautification of the folders there, let us know if we can help

I don't think this should be done manually by hand because we'd need to remember to set up the folders if a new stat node is added. So I think Puppet is the way to go here. I'm happy to make the necessary patches but I've found it really hard to keep track of / fully grasp which roles and profiles are applied to the stat hosts. I have some notes here: meta:User:MPopov (WMF)/Notes/DevOps#Puppet but they're not as detailed as I'd like.

In T254189#6202350, @Milimetric wrote:

we fully support the beautification of the folders there, let us know if we can help

Yay! I was wondering if @elukey would be able to put together a short guide to which puppet configs are responsible for which parts of the stat nodes. The only ones I have absolute clarity on are the ones related to packages/libraries. If you think that'd be a good idea I can request it via a Phab task so it can be properly triaged & prioritized.

Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:33 AM

Hm, if you want to do this in puppet, and not manually, then we can help, but I want to first double check something.

If all stat boxes have the same folders, the opportunity for collision goes up, overwriting files by putting them on the same path on two different boxes. When I thought you were all going to maintain this structure manually, I figured that would be an opportunity to make sure users understand this weird setup. But if it's done automatically, I wonder if it's more dangerous?

If you don't think that's a problem, then let us know what structure you'd like and we can make it happen in puppet.

LGoto moved this task from Upcoming Quarter to Backlog on the Product-Analytics board.Jun 22 2020, 4:44 PM

In the time since filing this task we have not encountered any problems that this would be a solution to, so this ask is no longer relevant.

/srv/published should be structured similarly, have identical README across stat hosts describing said structureClosed, DeclinedPublicActions

Description

Event Timeline

/srv/published should be structured similarly, have identical README across stat hosts describing said structure
Closed, DeclinedPublic
Actions