For https://phabricator.wikimedia.org/T100673, I would like to investigate the possibility of retrieving the data directly from the HTTP request logs.
It sounds like this data is available in HDFS (via stat1002), sourced from Varnish logs. Can I get access to this?
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| access: grant Jdouglas access toanalytics-privatedata-users group | Matanya | operations/puppet | production | +1 -1 |
If all I want to do is write Hive jobs, do I still need such high-level access, or can I get something more limited?
The webrequest logs are inherently 'private data' (they contain IP addresses etc.) so hence the name. But, ja, you will need that to access this data, and being in this group will let you log into stat1002 and run Hive queries.
Change 220990 had a related patch set uploaded (by Matanya):
access: grant Jdouglas access toanalytics-privatedata-users group
While the three days has passed and there is a patchset pending, this task hasn't yet received mgmt approval from his manager, which appears to be @Tfinc.
As such, I've assigned this to Tomasz for his approval.
Tomasz: Once you approve, please place the task up for grabs/unassign it from yourself. This will have it show up as unassigned and patchset pending for our ops clinic person to pick up and merge.
Change 220990 merged by RobH:
access: grant Jdouglas access toanalytics-privatedata-users group
With Wes's approval of rights escalation, I've gone ahead and merged the patchset.
@Jdouglas: once the hosts call in (can take up to 30 minutes or so), you should be all set. If things aren't working, simply re-open this task and we can investigate.