CentralNotice should display how many banner counts each banner and campaign has received. The counts should also be available through an API.
The hope is that public numbers will make it easier for CN admins to do A/B testing and other impact and effectiveness analysis. It will also enable discussions between stakeholders about relative shares of impressions.
(Might be a duplicate card?)
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T124132 [Epic] Analytics, graphs and monitoring for banners and CentralNotice | |||
| Open | None | T122589 Create tools to monitor the operation of CentralNotice for targeted groups of users | |||
| Open | None | T115042 Make banner impression counts available somewhere public | |||
| Restricted Task |
One possible visualization: pacman-sized pie charts in the campaign's summary table line, with a pie for the share of each affected wiki's traffic, or a golden freaking pie when it's using a noteworthy share of traffic over all sites.
@Nuria fr-tech, @Jseddon and I would like to discuss this topic. We understand analytics would need to work on part or most of this. An API would be a great start but visualizations similar to pivot would be a good P2 or 3.
Can we discuss how much effort this would take and where it might fit in your roadmap?
@DStrine: banner counts are available on Pivot, see: http://bit.ly/2pgY9vN
pivot is available through NDA so I believe the bulk of FR staff has access to those.
@Nuria there are many CentralNotice admins who are not under NDA and most campaigns in CN are not for fundraising. We are interested in getting some data to them to judge the effectiveness of their campaigns. Obviously this needs to be appropriately aggregated/anonymized too.
We are interested in getting some data to them to judge the effectiveness of their campaigns. Obviously this needs to be appropriately aggregated/anonymized too.
Let me understand:
Please edit the ticket description with this information, it is abit meager to understand what does this project entitle.
It seems it contains all banners as you have a choice to filter based on different banner types.
- What data do you need to measure effectiveness of a campaign?
I can speak to the needs of one community project and that is Wiki Loves Monuments (this applies to the majority of other Wiki Loves contests):
At the moment, it's very hard for the admin who sets up the banner or the people who are responsible for banners to understand how well the banner is doing. In the case of Wiki Loves Monuments, we have ~40 different banners shown to traffic from different countries. It has happened in the past that the contributions from one country has been significantly lower than expected, and as community members with no access to banner view data, it's impossible to know, for example, if the banner is being viewed by the traffic from the country or not. Some hourly information about the number of times a banner has been shown, the number of impressions, per country can be really helpful for diagnosing these problems at scale. This is of course only one example. We are always interested to learn how effective are the banners we use for Wiki Loves Monuments. If we want to start testing these banners, we need to know more about the impressions these banner create. There are more examples, of course.
Basically, because banners are heavily used by the community, it makes sense to empower the community to analyze the data around it to improve community projects. :)
- What data is of private nature and needs to be anonymized?
Let's say you want to release banner impression by country on an hourly basis. Would it be a problem from the privacy point of view if you had only 1 pageview from the country in a specific hour and the data shows that the one pageview has clicked on the banner? (The answer may be yes, especially when it comes to logged in users, but I'm not sure if we show banners to logged in users at all, so this may not be relevant.)
Basically, because banners are heavily used by the community, it makes sense to empower the community to analyze the data around it to improve community projects. :)
@LilyOfTheWest From your description I gather you need banner impressions per country. However how does that measure that a campaign was effective is not clear. I see how it could help you diagnose that a campaign "malfunctions" (it is not shown in the country that it was supposed to) but not much otherwise. I do not know much about how these banners are setup so I might be missing something obvious. Are banner impressions driven by clickthrough?
Let's say you want to release banner impression by country on an hourly basis.
Any type of country-level pageview data has many privacy implications, and the level of anonymization you need to render that data safe might actually do away with findings you can extract from data. So it is unlikely that we will ever be able to release any disglosed (banner or per-article pageviews) data for small geographical "entiites". Please see: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Pageview_hourly/Sanitization
We are due to work on this problem next year..
@Nuria I'm wondering if we can meet and talk about options. Community members need to know a few base things like:
*are we getting any impressions in a campaign?
*impressions per banner in a campaign
*impressions per country
I think there are a couple other basic ones but I'd like to start with something. They currently have no idea if their banners are reaching an audience.
Also I want to acknowledge that you have a bunch of other priorities. I'm happy scope out a super lightweight version and see where it fits in your schedule.
@DStrine we can met at your convenience .Before meeting I would like to understand what we want to measure. Is there any documentation anywhere as to how banner impressions are triggered? (as I said I do not much about this, I am trying to understand the context a bit more) If our objective is to measure effectiveness of a campaign we should define what that is. We try to map out the questions we want the data to answer before hand to make sure that the data you are requesting actually answers those questions. This the way we work with any data request we have for the creation of a new dataset.
Ping @LilyOfTheWest please let me know if you can provide more info. We can certainly compile a dataset of banners/country but please note that measures "eyes on campaign" which is quite different from effectiveness of campaign (you could have tons of impressions but no clicks)
Pitching in from WMDE Fundraising. Our tracking requirements are two-fold:
Pitching in from WMDE Fundraising. Our tracking requirements are two-fold:
@gabriel-wmde :Let's please not mix use cases, fundraising data for banner counts is already available internally and has been so for a while
Troubleshooting data: Comparing page impressions to banner impressions to detect if the banners were blocked or have an error that prevents them from being displayed.
Again, this is already available. Have you used pivot? http://pivot.wikimedia.org
Banner and pageview data are available at different granularities
@gabriel-wmde : now that i think about it my comment above assumes WMDE Fundraising works just like overall fundraising. Do take a look at data in pivot and let us know what you find is missing.
@Nuria I don't think WMDE has access to pivot or any LDAP related systems. That may be very complicated to setup.
@DStrine: that is incorrect, we have several users of pivot from WMDE office. @gabriel-wmde: do check with your peers, this is how you can file a ticket for access: https://wikitech.wikimedia.org/wiki/Analytics/Data_access#Pivot
Hi folks!
Following up on the meeting that we just had (meeting notes), a couple questions:
CentralNotice has features to target users by any combination of the following criteria:
Also, some data stored on the client that's used to determine which banner to show could potentially "leak" information about users' habits into impression records. For example, since large Fundraising banners are only shown once to any user (within a certain time period) frequent users have a lower likelyhood of seeing a large banner on any given page view. And users who have donated within the last year won't see Fundraising banners on the device they used to make the donation.
Finally, since the banners can contain arbitrary Javascript, they can show or hide based on any criteria visible to that code. For example, there have been campaigns that target only users with a minimum number of edits (for community elections). There was also a test Fundraising campaign that targeted a specific set of articles. CentralNotice does expose an API to in-banner Javascript which allows it to control what is sent to the servers via beacon/impression. If the in-banner code is written as it should, decisions to show or hide banners like this will be available in impression data.
- Regarding clicks on banners, wouldn't it be possible to fire a JavaScript event when the banner is clicked, and send click-related information to the same beacon, just before the browser requests the page linked in the banner?
Beyond a few API methods it exposes, CentralNotice does not interact with in-banner code. From our point of view, banners are just blobs of stuff (HTML, JS, CSS) that we inject into the page. There's no standard interaction with any of our code when a banner is clicked. We could set up a mechanism like the one you describe to provide click-through rates... but that'd be a new feature. Maybe it'd be easier to limit the current task to sanitizing and publishing existing data? Also, I don't know that we have any requests for such a feature... I think normally raw banner-click totals are handled directly by the campaign organizers (and FR has its own way for that)... So, by providing them impression totals, we'll give them the means to calculate click-through rates on their own, maybe? I can certainly see how it'd be nice to have unified impression and click-through data, but again, it might be best to leave that as a separate step?
Thanks so much!!!!!!! :D
@Nuria : Have you read the meeting notes mentioned above? To my naive eye, they seem to contain some of the types of questions you asked for earlier. If they're not what you're looking for, then iIt might be helpful to have a meeting with an analyst and/or researcher, to help others understand the kinds of questions that would be particularly meaningful.
Is there enough information now to schedule a meeting? Or should we continue the conversation here for now?
@mforns : Can you say who was at that meeting?
Closing for inactivity, please re-open if publishing this somewhere public is still a priority.
@Milimetric I can say that this would still be helpful from the community point of view. Not sure if that warrants reopening, and no idea what happened at the meeting - so don't have the full picture.
I have reopened this. This is still a fairly important project. I have conversations about how to prioritize this at least once a quarter.
@DStrine: ok, let us know when you have a solid plan for it and how it fits in with other work.
@Milimetric Hashing out the first step of what's needed from the community organizer end is relatively straightforward if we can understand better the limitations from the privacy perspective on your end. On the community organizer end, we can think of a relatively simple model to start from: On a daily basis output something along these lines: date, banner_name, project, sampling rate, number of impressions.
Do you see any limitation for releasing such a data string? If yes, what are those? If not, give us a bit more time and we will finalize the string we're asking.
There are a bunch of privacy issues. This requires a lengthy research project and some level of community involvement. @Jseddon is the community liaison and he and I are pretty much on the same page. We just need to find the time to tackle this.
@DStrine what are the timelines you're looking into for setting aside time for this? Knowing how long you expect to need time before you can pick up this task can help us understand if we have to look into other means to answer the questions we have in the meantime. We don't want to start new endeavors to get answers to our basic campaign related questions unless they're absolutely needed as they will take a lot of volunteer resources. Given that this task has been open since 2015, I'm concerned that if we don't set some timeline on our ends, this will be left open for at least another year or two.
Untagging Analytics as there hasn't been any input in a while. When you know more details, tag us again.