Page MenuHomePhabricator

[Needs grooming] Turnilo: include authentication status in request data cube
Open, Needs TriagePublic

Description

In order to filter logged-in vs logged-out access to APIs, the webrequests_sampled_128 cube (or equivalent) should have a dimension that indicates the authentication status (and ideally, authentication method) of each request.

Implementation ideas:

  • Dan said: the pipeline is varnish -> kafka -> hdfs -> druid -> turnilo where Turnilo is just a way to query the data but you can get to it straight from kafka, hdfs, or druid without Turnilo
  • Valentin pointed out: The X-Analytics header already has a loggedIn flag. This could be updated to indicate the auth method (cookie, auth header, or centrlauth token, etc).
  • Mikhail suggested: This flag could be processed by oozie to make this information available in Druid.
  • Gergö added: Note that at least for the action API, we have three auth methods (cookies, OAuth and CentralAuth tokens). The mediawiki_api_request table contains the raw data (headers + API parameters) but you'd have to post-process it a bit.

Event Timeline

JArguello-WMF renamed this task from Turnilo: include authentication status in request data cube to [Needs grooming] Turnilo: include authentication status in request data cube.Apr 20 2023, 3:14 PM
JArguello-WMF moved this task from Needs Grooming to Incoming on the API Platform board.
lmata subscribed.

Hi, we're moving to radar for now. Please let us know if there's anything we can do to assist.

This is currently in "Backlog" on the Data-Engineering board, along with over 100 other tasks. To me it feels important, and somewhat time sensitive. It's also very easy. I'm moving it back to Incoming so Andreas and Virginia can see it again.

So this work will basically enable better work on the upcoming Content Reuse work. Since that's high priority for the organization, maybe we should reconsider this task's priority. And if the decision is not to do it soon, I'd recommend even declining it so the folks requesting it can know to adjust.

Ahoelzl subscribed.

Moved to Next Up (in consideration for Q1)