Page MenuHomePhabricator

MPIC: Add stream name to forms/database/api
Open, MediumPublic3 Estimated Story Points

Description

T360731 T360707

Description

Pending T366325, stream name needs to be added as a field to the forms, database tables, and returned in the mpic api json response.

For web and app base instruments, we will know what the schema title will be (ticket TK for generating a monotable stream name for web and app):

  • user selects web:
    • stream name: TK
    • schema title: /analytics/product_metrics/web/base
  • user select app:
    • stream name: TK
    • schema title: /analytics/product_metrics/app/base

If we are supporting custom schemas (derived from MP base schemas), we need a text input field for this use case.

Technical notes

Once designs are settled, we should have a new form field that captures if the instrument is a web base, app base, or custom instrument, and a text field for capturing the stream name if it is a custom instrument.

Note that stream names must be available on production in order to query for corresponding schema titles.

Schema titles can be retrieved by querying for stream name in the meta api << only applicable for if/when we implement an autocomplete lookup field for capturing a custom schema title.

Re: TKs << see T366949#9914737

Acceptance Criteria

  • stream_name and schema_title are included in the database init script (Already done here)
  • Forms are updated with latest design for this field
  • API response returns corresponding schema title given the stream name
  • Update the MPIC design doc

Required

  • Unit/Integration tests?
  • Documentation?
  • Passed QA?

Details

Event Timeline

Just a few questions:

  • Is this new field optional? It seems we only need when using custom schemas
  • It seems we should update the Design Document according the work we are going to do here. Should we add that to the Acceptance Criteria? I'm not sure if there is a different task for this purpose
  • I didn't find any design with this new form field. Is there any? I'll add it temporary as a regular field close to the Name and Description ones. We can move it later to the right position and with the proper configuration
  • Do we have to call the meta api every time someone make a request to get details for a specific instrument? Why not creating an additional column in the database to save the stream_title when launching/modifying the instrument and using that value every time the details are requested? Is the public endpoint (api/v1/instruments) the only one that must return the stream_title?
cjming updated the task description. (Show Details)
  • Is this new field optional? It seems we only need when using custom schemas

We just decided in the meeting today that for alpha release, we'll keep the custom schema title as a text input (disabled by default until the user selects Custom radio option). For a future beta release, this will need to be an autocomplete lookup field that fetches schema title options from Gerrit's API.

3.png (938×1 px, 101 KB)

  • It seems we should update the Design Document according the work we are going to do here. Should we add that to the Acceptance Criteria? I'm not sure if there is a different task for this purpose

Yes - i am behind on updating the design doc (remember that we decided that canonical is now the MediaWiki page, not the google doc) - i'll add as AC for this ticket, and current/future tickets.

  • I didn't find any design with this new form field. Is there any? I'll add it temporary as a regular field close to the Name and Description ones. We can move it later to the right position and with the proper configuration

see screenshot above -- @mwilliams is updating Figma designs after today's meeting

  • Do we have to call the meta api every time someone make a request to get details for a specific instrument? Why not creating an additional column in the database to save the stream_title when launching/modifying the instrument and using that value every time the details are requested? Is the public endpoint (api/v1/instruments) the only one that must return the stream_title?

This is only applicable when we have to provide autocomplete options for the custom schema title. Since we're not doing that part for the alpha release (just collecting schema title in a simple text input field), we can skip this for now. I updated the ticket description to state this explicitly.

@Sfaci - let's discuss validation options for the custom schema title text input field. While we want to support custom schemas in the alpha release for end-to-end testing/dogfooding, we should figure out what, if anything, we want to do with incorrectly submitted custom schema titles.

Maybe it's easy to do the autocomplete lookup field whose options are populated by a Gerrit API call against the secondary schema repository but we'd need a key/token and probably a pinia store to manage its state as the options for the lookup field. I will spin up a ticket for this work

But in the meantime, a lo-fi version to deal with validation of the custom schema title could be... nothing? I'm not sure what happens if somehow a malformed schema title works its way into the meta api. I'll found out and post back here.

Turns out it's easy! we can hit public meta api to get production stream names and their mapped schema titles.

As we have discussed on Slack, we are going to push the following MR: https://gitlab.wikimedia.org/repos/data-engineering/mpic/-/merge_requests/46 where changes related to this ticket only include to add a TextField to type there the stream name to save it to the database.

On the meantime we are going to keep working, using this task, on the pending changes (using the Action API to get the stream_title from the given stream_name to populate a Lookup field)

One dependency of this task is that the monotable + base stream configs need to be in prod before we can properly populate the stream name for app base and web base radio button options.

The work is captured by the following tickets:

  • T367057: [SPIKE] Document decision to use a single table per base schema
  • T366802: Create a Metrics Platform schema fragment for instrument discriminating/filter fields
  • T366807: [EPIC] Update Metrics Platform Client Libraries to accept instrument name
  • T366827: Land MR !39
  • T367034: Make MetricsPlatform MediaWiki extension deliver JS and PHP Client Libraries

See https://wikimedia.slack.com/archives/C05ERLBF0E7/p1718031446788869

cjming triaged this task as Medium priority.Fri, Jun 21, 11:49 PM
cjming updated the task description. (Show Details)
cjming updated the task description. (Show Details)
phuedx subscribed.

One dependency of this task is that the monotable + base stream configs need to be in prod before we can properly populate the stream name for app base and web base radio button options.

I've added T366807 as a subtask (dependency) for this task.

The new MR to get stream data from the ActionAPI when user is working with a custom instrument is almost done, but we need some sample values for the stream name when user chooses app or web types.
We decided to pause this task because we need to find the way to publish to production some streams for web and app instrument so that we can use them as the default values when user selects this type of instrument.

Change #1050678 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[operations/mediawiki-config@master] Add test streams for Metrics Platform app + web base instruments

https://gerrit.wikimedia.org/r/1050678

@phuedx @Sfaci - maybe above config patch will cover us for testing until monotable work is finished?

The config patch looks fine.

We decided to pause this task because we need to find the way to publish to production some streams for web and app instrument so that we can use them as the default values when user selects this type of instrument.

@phuedx @Sfaci - maybe above config patch will cover us for testing until monotable work is finished?

Why do we need the streams to be in production for testing? Would it be sufficient to define them on the Toolforge instance? I suppose that this would require that we run our own instance of MPIC but that wouldn't necessarily be the end of the world 🤔

Why do we need the streams to be in production for testing? Would it be sufficient to define them on the Toolforge instance? I suppose that this would require that we run our own instance of MPIC but that wouldn't necessarily be the end of the world 🤔

ok - how is this instead?
https://gitlab.wikimedia.org/toolforge-repos/mwdemo/-/merge_requests/1

Change #1050678 abandoned by Clare Ming:

[operations/mediawiki-config@master] Add test streams for Metrics Platform app + web base instruments

Reason:

in lieu of https://gitlab.wikimedia.org/toolforge-repos/mwdemo/-/merge_requests/1

https://gerrit.wikimedia.org/r/1050678