The old convention included eventlogging as a prefix, eg: eventlogging_HomepageModule, eventlogging_HomepageVisit. The only usage I've found for the metrics platform client in ContentTranslation (useEventLogging.js#4) seems to use a different format mediawiki.product_metrics.mint_for_readers, also see T363685#9908791 where consistency reasons are mentioned.

I guess we still want to have two separated streams for visits to the homepage to keep track of non-JS capable users as we do now. The proposal would be to create two new streams to transition HomepageModule and HomepageVisit to. Naming proposal:

• mediawiki.product_analytics.growth_homepagevisit
• mediawiki.product_analytics.growth_homepagemodule

Thoughts? cc @nettrom_WMF @Urbanecm_WMF @Michael @Cyndymediawiksim

Note: we may want to take the opportunity to re-scope the purpose of the schemas here to not be homepage limited.

In T370907#10037961, @Sgs wrote:

The old convention included eventlogging as a prefix, eg: eventlogging_HomepageModule, eventlogging_HomepageVisit.

Note both of those schemas are legacy ones – they were originally a part of eventlogging (https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/EventLogging). Our newer schemas (which were created directly within Event Platform) do not have a clear pattern. For example, we have:

• mediawiki.mentor_dashboard.interaction
• mediawiki.mentor_dashboard.personalized_praise
• mediawiki.mentor_dashboard.visit
• mediawiki.editgrowthconfig
• mediawiki.structured_task.article. image_suggestion_interaction
• mediawiki.structured_task.article. link_suggestion_interaction

(and possibly few more)

The only usage I've found for the metrics platform client in ContentTranslation (useEventLogging.js#4) seems to use a different format mediawiki.product_metrics.mint_for_readers, also see T363685#9908791 where consistency reasons are mentioned.

Given we're using a new system, using a common pattern to "group" relevant schemas makes a lot of sense. Others indeed seem to be using mediawiki.product_metrics as a prefix, so doing that makes sense to me as well.

I guess we still want to have two separated streams for visits to the homepage to keep track of non-JS capable users as we do now. The proposal would be to create two new streams to transition HomepageModule and HomepageVisit to. Naming proposal:

• mediawiki.product_analytics.growth_homepagevisit
• mediawiki.product_analytics.growth_homepagemodule

Works for me. Wondering whether it is more appropriate to call it growth_ or growth., since essentially growth is yet another part of the path. But, this is a nice-to-have thing – we can definitely work with an underscore as well.

@Urbanecm_WMF Going by the recommendation that T363685#9908791 makes, I'm guessing that something like mediawiki.product_metrics.growth_homepage_community_updates would work. The last segment follows a pattern that reads <team_name>_<page>_<modulename>

After reading the How to Guide, I would want to raise the question whether Metrics Platform is the right tool to use. That documentation prominently states:

As important as creating and operating your instrument is ensuring that you have a complete lifecycle plan for your instrument. Instruments should not, by default, be long lived.

(bold in the original)

As I understand the examples of our other instrumentation brought up in previous comments here, all of those are intentionally permanent instrumentations. Based on the linked docs, that is explicitly not what Metrics Platform is for. So we should not use Metrics Platform for that.

If we are looking to run an experiment with Metrics Platform, then I think the proposed naming convention is plausible. Though I would appreciate some guidance from the team (looping in @Sfaci on that), as the documentation provides none (based on my glancing at it).
Given that we will be using the Metrics Platform for time-limited experiments, I would suggest including some time-reference in the name of the stream, so that it can be quickly differentiated from future experiments. So maybe mediawiki.product_metrics.growth_FY25_homepage_community_updates or something similar. For how long do we expect these experiments to usually run?

On a more general note, I would very much appreciate for there to be a Kickoff with us and the Metrics Platform team on our upcoming collaboration, and an intro to Metrics Platform, before we start making hard to reverse decisions.

I'm chiming in to make sure we're on the same page and don't spend efforts on possibly dead ends. Here's my understanding of some of the things we're discussing:

1. My understanding from the Grows team's adoption document is that we're only instrumenting the Community Updates module with Metrics Platform. The existing HomepageVisit and HomepageModule instrumentation will stay the way they are (except we're adding the community-updates module to HomepageModule per T365889). In other words, we're not migrating existing instrumentation to Metrics Platform. Please let me know if this plan has changed!

2. The user interactions with the Community Update module that we'll instrument are fairly straightforward: impressions and clicks. These interactions should be covered by the core interaction schemas and the API. There's currently a plan to use a single stream (and subsequent database table) for the core interaction schema, see T367057. If that's the case, there's already a stream defined.

3. @Michael is correct that Metrics Platform instruments are supposed to be short-lived. This is something I've brought up previously with the Metrics Platform team. From what I know, there's currently not a clear plan for what we do with long-term instrumentation (for what I now refer to as "product health metrics").

Either way, I think a kickoff meeting with Growth and Metrics Platform to start the conversations to make sure we're aligned on things is an excellent idea!

Hey @nettrom_WMF, thanks for the clarification.

@Michael Our use of Metrics Platform for instrumentation in this case stems from the fact that we're a participating team in the SDS 2.1 KR around eventually running an A/B test with the MPIC platform. This means that we have to instrument using Metrics Platform in order to eventually run the test.

With regards to the metrics being short lived, this is something that an in-person meeting would be helpful in clarifying and it generally applies to all instruments and not just Growths. A number of people will be out this week and for the next 3 weeks for summer break so having everyone available will be a bit of a challenge. Let's see how to work around this.

In T370907#10039545, @nettrom_WMF wrote:

I'm chiming in to make sure we're on the same page and don't spend efforts on possibly dead ends. Here's my understanding of some of the things we're discussing:

1. My understanding from the Grows team's adoption document is that we're only instrumenting the Community Updates module with Metrics Platform. The existing HomepageVisit and HomepageModule instrumentation will stay the way they are (except we're adding the community-updates module to HomepageModule per T365889). In other words, we're not migrating existing instrumentation to Metrics Platform. Please let me know if this plan has changed!

That is correct, however, in terms of code, instrumenting only the community updates module generates more inconsistency rather than consistency which is what the MP adoption is supposed to bring. That is because the Community Updates module is not just another Growth feature but a homepage module. Hence, the already existing instrumentation and the new MP instrumentation just for one homepage module need to coexist. I would strongly advocate to migrate all homepage instruments to the MP as follow-up plan from the adoption for Community Updayes. Otoh, although a bit late in the process, we could choose another feature or project from Growth team to adopt MP which does not have existing instrumentation in place, eg: CommunityConfiguration

2. The user interactions with the Community Update module that we'll instrument are fairly straightforward: impressions and clicks. These interactions should be covered by the core interaction schemas and the API. There's currently a plan to use a single stream (and subsequent database table) for the core interaction schema, see T367057. If that's the case, there's already a stream defined.

My understanding is that if we use that stream we will be locked to the upstream capabilities which are insufficient. eg: capture the update title, as mentioned in T365889#10039305. The idea to have a new stream is to re-use as much as we can from core schemas but also be able to add Growth specific interaction data.

3. @Michael is correct that Metrics Platform instruments are supposed to be short-lived. This is something I've brought up previously with the Metrics Platform team. From what I know, there's currently not a clear plan for what we do with long-term instrumentation (for what I now refer to as "product health metrics").

I think the decomission of instruments has always been a request from analytics teams and it's not bound to the used platform or technology. It is just more prominent in the MP documentation than in the Event Platform one: Event_Platform/Instrumentation_How_To#Decommissioning

Hi @Michael!

Sorry for the lack of information about how to choose the right stream name! Definitely it's something we should add there. Anyway, you are right about the proposed names, mediawiki.product_metrics should be used as prefix and, after that, you can use your instrument name. For example, the last three instruments that were created recently use the following names:

• mediawiki.product_metrics.wikifunctions_ui,
• mediawiki.product_metrics.wikilambda_api,
• mediawiki.product_metrics.mint_for_readers

About the limited time for experiments and how that can affect to the stream name, I would say it's not needed to add anything there (you mentioned to add Y25 to the stream name). The event date is included automatically as a core field so you could filter events properly to work only with the ones from a specific period. Anyway, I think that, at this point, we don't have any specific number to specify which is the maximum amount of allowed time for a specific instrument but, at least, we ask for having a plan to have an end date when the instrument should be decommissioned.

And, of course, we are fully open to have any meeting you can need to talk about this and try to help you to start with your new instrument and Metrics Platform. Just invite us and we will be there!

2 drive by comments for ya!

plan to have an end date when the instrument should be decommissioned

I agree with @Sfaci, this seems to be the important part, not so much how long an instrumentation actually lives. You can always extend the end date, but having an end date helps us more easily know if something is unowned or not needed, and can be decomissioned (especially as teams and owners move around).

Re stream name:

This is not a requirement, but because streams contain events of something happening, I find it meaningful to name a stream after an action of some kind. 'interaction', 'change', 'click' 'impression', 'login', etc. are examples. They help you do know what each event represents. 'mint_for_readers' sounds like more of a namespace than an event stream name to me.

BUT! YMMV!

Thanks!

If I take a couple of things together here:

In T370907#10039545, @nettrom_WMF wrote:

[...]

3. @Michael is correct that Metrics Platform instruments are supposed to be short-lived. This is something I've brought up previously with the Metrics Platform team. From what I know, there's currently not a clear plan for what we do with long-term instrumentation (for what I now refer to as "product health metrics").

In T370907#10042035, @Ottomata wrote:

[...]

plan to have an end date when the instrument should be decommissioned

I agree with @Sfaci, this seems to be the important part, not so much how long an instrumentation actually lives. You can always extend the end date, but having an end date helps us more easily know if something is unowned or not needed, and can be decomissioned (especially as teams and owners move around).

Then it seems to me that it would be fine to have these metrics in MP if we have a specific plan to use them in an experiment in the (near) future, and if we are willing to name a concrete decommission date and plan. Though the date can be pushed if we intend to use the metrics from the instrumentation for follow-up experiments.

So, to check my understanding, would it be fine to say that the decommission date for these new metrics is September 30th 2025, i.e., end of Q1 of the next FY. And if we want to keep the instrumentation around a bit longer, then at the end of this FY, we can adjust that decommission date by another year into the future.
Does that sound alright, or is "planned live-time: 1 year" too long or too short?

Also, I note that for things like the Event Platform schema mediawiki.mentor_dashboard.personalized_praise, as I understand it, we used to have an experiment running, but that has concluded a while ago, and there are no plans for follow-up experiments (as far as I'm aware of them). Would it be ok to migrate them to MP, or should we decommission them in EventPlatform, and only create a new version of them in MP if we actually ever have a need of them again in the future? Or should we consider them "product health metrics", as @nettrom_WMF put it above, and generally find a different solution for them altogether?

In T370907#10041476, @Sfaci wrote:

Hi @Michael!

Sorry for the lack of information about how to choose the right stream name! Definitely it's something we should add there. Anyway, you are right about the proposed names, mediawiki.product_metrics should be used as prefix and, after that, you can use your instrument name. For example, the last three instruments that were created recently use the following names:

• mediawiki.product_metrics.wikifunctions_ui,
• mediawiki.product_metrics.wikilambda_api,
• mediawiki.product_metrics.mint_for_readers

About the limited time for experiments and how that can affect to the stream name, I would say it's not needed to add anything there (you mentioned to add Y25 to the stream name).

Thanks! So looking at these, these schema names seem to be more like mediawiki.product_metrics.<product>_<component>. And as mentioned above, they probably also include something like "'interaction', 'change', 'click' 'impression', 'login', etc" somewhere down the line.

But as for structure:

• Is any tool in the MP stack assigning meaning to dots or underscores or dashes in the stream name? Like, would mediawiki.product_metrics.growth.growth_experiments.homepage.community_updates be meaningful different from mediawiki.product_metrics.growth_growth-experiments_homepage_community-updates?
• Is it a good idea to have a team name in the name of the schema?
• Do we have a length restriction for the schema name?

we can adjust that decommission date by another year into the future

Metrics Platform folks should confirm.

But for me: the benefit of the decommission date (at least at Event Platform level) is so we can turn stuff off when owners and users disappear. If owners are around to keep updating the date, I'm fine with it :)

I'll try to answer some open questions that @Michael made. I hope I don't miss any:

Does that alright, or is "planned live-time: 1 year" too long or too short?

As we discussed before there is nothing written down about specific duration but there is about having an end. Based on that, I would say that if you have an end date your duration should be fine. I mean, if you have a solid plan that requires to run your experiment for a year, I would say it should be fine.

Would it be ok to migrate them to MP, or should we decommission them in EventPlatform, and only create a new version of them in MP if we actually ever have a need of them again in the future?

I would say you are righ. I wouldn't make sense to prepare and deploy an instrument with MP if there is no plan to run it. And I guess it should be decommissioned from EventPlatform but I'm not sure about that part. Maybe EventPlatform works different way.

Thanks! So looking at these, these schema names seem to be more like mediawiki.product_metrics.<product>_<component>. And as mentioned above, they probably also include something like "'interaction', 'change', 'click' 'impression', 'login', etc" somewhere down the line.

You are right!. I didn't realize that those names don't include anything regarding the event itself. I fully agree with @Ottomata on including something like the interaction as a part of the name

Is any tool in the MP stack assigning meaning to dots or underscores or dashes in the stream name? Like, would mediawiki.product_metrics.growth.growth_experiments.homepage.community_updates be meaningful different from mediawiki.product_metrics.growth_growth-experiments_homepage_community-updates?

The way we compose the stream name is a pattern we use but I am not able to say why we chose that one. What I can say is that the stream name will be the name of the hive table where the events will be stored, replacing dots and dashses with underscores. For example, if you define a stream name like mediawiki.product_metrics.homepage_community_updates, event data will be stored at a hive table whose name will be event.mediawiki_product_metrics_homepage_community_updates. So, the two samples you mentioned would be the same hive table.

Is it a good idea to have a team name in the name of the schema?

Do you mean schema or stream? I will assume you meant stream because we were talking about stream names:
I'm not sure what to say here. But I would say the team name is not relevant. As @Ottomata suggested before, I think that adding the interaction to the name is a really good idea. I would use something that would describe the experiment you want to use without considering the team name, dates or any other data not directly related to the events you want to get. In fact, starting from the samples you mentioned before and removing the team name and some context, I would say that something like mediawiki.product_metrics.homepage_community_visits would sound good to track the homepage user visits.

Do we have a length restriction for the schema name?

Same here, schema or stream? I will assume stream as well:
I never thought about that. Is there a maximum for a hive table name? I guess that's the real restriction here because, in the end, the stream name will become the hive table name.

good idea to have a team name

IMO, generally, it is never a good idea to put team names in code or data. Team names change every few years.

schema or stream

FWIW, often a stream and schema are named similarly, because a schema is designed for a single usage. But sometimes schemas can (and should) be used for multiple uses (like Metrics Platform schemas are doing :D ).

In either case, the both the schema and stream should be named after the thing it represents, and the stream can be more specifically named or namespaced as needed.

In T370907#10043743, @Sfaci wrote:

Do we have a length restriction for the schema name?

Same here, schema or stream? I will assume stream as well:
I never thought about that. Is there a maximum for a hive table name? I guess that's the real restriction here because, in the end, the stream name will become the hive table name.

Indeed, physical constraints always exist, from a quick stackoverflow search it seems the limit is 256 characters (I haven't checked Hive version, assuming it's correct since the answers are 6 year old).

After reading through all of the insightful feedback here I'm gonna propose two different approaches, one (A) that scopes our experimentation goal to all homepage modules as a "product health metric", hence long lived. A second approach (B) that scopes the experimentation to the "Community updates" feature as a short-lived experiment (eg: 1 year)

Proposal A (Product health metric, long lived)

mediawiki.product_metrics.homepage_module_interaction. The rationale behind the last fragment is:

• Use homepage_module as the <product>_<component>. Consider all homepage modules to play a key role on editor constructive activation and retention, hence a stream to monitor them all
• Avoid including growth as it creates ambiguity between the team (Growth) and the software component (GrowthExperiments), and both are prone to eventually change
• Avoid including FY25 as this stream falls in the category of long lived "product health metrics"
• Include interaction as suggested by @Ottomata, the stream will capture different kinds interactions (impressions, link clicks, hovers, and ad-hoc UI interactions such as navigating using the tasks carousel arrows)

Proposal B (Community updates timeboxed experiment)

mediawiki.product_metrics.homepage_community_updates_interaction. The rationale behind the last fragment is:

• Use homepage_community_updates as the <product>_<component>. Scope the experiment to a specific feature rather than making it a product health metric.
• Avoid including growth for same reasons as in proposal (A)
• Avoid including FY25 as even if this stream/schema would be eventually decomissioned, defining the timebox to FY25 can be tricky to commit after.
• Include interaction as suggested by @Ottomata, the stream will capture different kinds interactions (only impressions and link clicks).

If we find the homepage not specific enough as the product name, or that it could collision with similarly named things around MW we can also prefix with the extension name, eg: growth_experiments.homepage_module_interaction and growth_experiments.homepage_community_updates_interaction.

We need to make a decision on one of the proposals (or suggest other approaches). cc @nettrom_WMF @KStoller-WMF @DMburugu @Michael

The one that specifies the module within the homepage that we're looking at while maintaining consistency with other instruments at the foundation seems better. So mediawiki.product_metrics.homepage_community_updates_interaction

In T370907#10051638, @DMburugu wrote:

The one that specifies the module within the homepage that we're looking at while maintaining consistency with other instruments at the foundation seems better. So mediawiki.product_metrics.homepage_community_updates_interaction

Mind that this is a radical change from how we've instrumented homepage modules until now as it implies an stream per-module.

Thanks for highlighting that.

If we can capture the module name within the payload sending the interaction metrics e.g. {"module": "community_updates"}, then we can adopt this pattern which allows us to easily instrument other modules using the same schema: mediawiki.product_metrics.homepage_module_interaction

I'm in favour of proposal A for two reasons:

1. The Growth team would most likely want to have "product health metrics" (long-term instrumentation) for the Homepage available, and this would be a first step towards that for modules on the Homepage.

2. As mentioned by @Sgs in T370907#10041395, one goal of the Metrics Platform work is to bring consistency to instrumentation data. While the current plan is to only instrument the Community Updates module, I don't think it costs much, if anything, to start planning towards having everything migrated. The added benefit is that it provides the Growth team with more flexibility as other Product Analytics team members are familiar with that kind of instrumentation and is able to work with it.

But, I might be wrong and am happy to learn that.

Change #1062416 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[operations/mediawiki-config@master] EventStreamConfig and stream registration for homepage modules analytics

https://gerrit.wikimedia.org/r/1062416

Change #1062416 merged by jenkins-bot:

[operations/mediawiki-config@master] EventStreamConfig and stream registration for homepage modules analytics

https://gerrit.wikimedia.org/r/1062416

Mentioned in SAL (#wikimedia-operations) [2024-09-11T07:19:48Z] <sgimeno@deploy1003> Started scap sync-world: Backport for [[gerrit:1062416|EventStreamConfig and stream registration for homepage modules analytics (T370907)]]

Mentioned in SAL (#wikimedia-operations) [2024-09-11T07:24:40Z] <sgimeno@deploy1003> sgimeno: Backport for [[gerrit:1062416|EventStreamConfig and stream registration for homepage modules analytics (T370907)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-09-11T07:33:44Z] <sgimeno@deploy1003> Finished scap sync-world: Backport for [[gerrit:1062416|EventStreamConfig and stream registration for homepage modules analytics (T370907)]] (duration: 13m 56s)

Checked in betalabs - mediawiki.product_metrics.homepage_module_interaction stream works and two events - impression and click are recorded as expected.
pinging @nettrom_WMF - from my point of view the scope of the task is done.

Looks good to me, moving to "Done"