@SNowick_WMF Thanks for making this ticket!

Most helpful to know:

1. Which widgets are used most frequently by app users?
2. How frequently are users engaging with widgets?
3. How many users open the app through one of the widgets?

Of these three, the only one we can somewhat accurately measure is 1. How many users open the app through one of the widgets?. We can append some type of extra URL parameter to the URLs the widget uses to open the main app, then log those events in the main client app in order to get an aggregate sense of usage.

From an engineering perspective, we just need to confirm a. we can append some kind of query or URL parameter to the widgetURL modifier URL the widgets use, b. intercept that in UIApplication's openURL and log the event, and c. be sure with that query or URL parameter is removed and the normal NSUserActivity routing works to send the user where they expect.

To better understand how we may possibly measure the others, some clarifying questions:

2. Which widgets are used most frequently by app users?

What are we meaning by "used" here? Most frequently added to their home screens? Most frequently interacted with?

3. How frequently are users engaging with widgets?

What are we considering "engagement"? Viewing the widget? Tapping the widget? If it's viewing the widget, that's just not concrete information we have available. If tapping the widget, that's something that'd be covered by item 1.

Nice to have:
Are any of the current widgets unstable or causing crashes?

This is an interesting one – I think it's dependent on how we're defining "unstable". I'd say (unscientifically) 95% of the widget stability related issues we have control over are API/Wikifeeds related. Perhaps measuring the failed Wikifeeds calls is useful? The widget crashes we get reported to us via Apple/Xcode crashlogs are exclusively iOS level errors or related to hard memory limits which are specific to iOS stuff and are not necessarily applicable to Android's widget implementations.

From an engineering perspective, if we have to measure Wikifeeds stability in the widgets, we could perhaps leverage the shared app container to log failures (date, widget type, etc.), and event log then delete these shared container logs when the user next opens the main client app.

Hi @Dmantena will answer your Qs inline:

1. Which widgets are used most frequently by app users?

If we can measure 3. How many users open the app through one of the widgets? by user, can we also measure app open events by user/date so we can get usage frequency? And using those same metrics we can determine which Widget is used the most. So the first 2 questions hinge on the 3rd if that's the only thing we can reliably measure.

But if we can measure engagement (as tapping the widget) as well, as an event by unique by date that gives us more data to compare widget popularity. Adding the widget to home screen wouold be a less frequent event and give us less info but if we can get that too let's include it.

I will defer to Jazmin on if we want to implement the Wikifeeds stability measurement in the widgets as described here - it's hard to know if this is worth measuring if we don't know if it happens at all, is there any quick way to get an idea without doing the full engineering task just to get an idea?

If we can measure 3. How many users open the app through one of the widgets? by user, can we also measure app open events by user/date so we can get usage frequency? But if we can measure engagement (as tapping the widget) as well, as an event by unique by date that gives us more data to compare widget popularity.

So long as my engineering notes in the comment above prove viable, we should be able to measure this: taps the user makes on a widget, by widget type, with local date of that tap event.

Adding the widget to home screen wouold be a less frequent event and give us less info but if we can get that too let's include it.

I don't feel like we can reliably capture this information (an event that the user has added a particular widget to their home screen).

it's hard to know if this is worth measuring if we don't know if it happens at all, is there any quick way to get an idea without doing the full engineering task just to get an idea?

Other than relying on user reports or our own in-use experiences, I can't immediately think of a way to get in-aggregate device estimates that doesn't rely on fuller instrumentation engineering in the iOS app. But an alternative to instrumenting something in-app/a new event schema though is an approach I've been advocating since the widgets became a feature: create a WMF labs instance/job that hits the same Wikifeeds endpoints the widgets do on a regular basis, in every Wikifeeds language we support, and report potential issues in current and upcoming days back to us. This may as you mention give us a better idea without fuller engineering work on the iOS client. I'm still very open to doing this.

Thanks @Dmantena - w/r/t measuring

an event that the user has added a particular widget to their home screen

maybe we could add a flag event in action_data for the first time user intiates any engagement with widget that would only fire once per widget per user, separate from/concurrent to other engagement events, I can then count 'first use' events to get an idea of how often new widget users are activated. Lmk what you think.

Curious to hear the response to Shay's idea. Also worst case if we look at opens from widget and can distinguish from widget type, we can just go with this.

But an alternative to instrumenting something in-app/a new event schema though is an approach I've been advocating since the widgets became a feature: create a WMF labs instance/job that hits the same Wikifeeds endpoints the widgets do on a regular basis, in every Wikifeeds language we support, and report potential issues in current and upcoming days back to us.

@Dmantena Agreed this could be helpful to confirm upcoming featured articles / on this day / POTD will return the correct data. Our most problematic widget lately is Top Read, which cannot be fetched days in advance. Also, based on your comment here, it seems like the top read data could be there or missing depending on when the job runs, so a daily call may not be accurate for this one. So I think for those particular live issues we should consider asking Content Transform if they are able to add error logging for whenever top read elements are missing in their response.

maybe we could add a flag event in action_data for the first time user intiates any engagement with widget that would only fire once per widget per user, separate from/concurrent to other engagement events, I can then count 'first use' events to get an idea of how often new widget users are activated. Lmk what you think.

@SNowick_WMF @Dmantena I came across this API in WidgetKit, it looks like it can give us information about what widgets the user has currently configured. We could log it occasionally from the app (maybe in the ios_user_history schema?). I haven't experimented with it but if you don't need to get really specific like the exact time a user adds a particular widget to their home screen, this may work.

This needs to be completed before any work on Widgets is prioritized.