The previous results of analyzing the abandonment rate between the survey and control groups can be found in T206380#5045440. That analysis includes a split between desktop and mobile registrations, and we found that on the mobile side there was no significant difference in abandonment, while on the desktop side the difference was statistically significant. Because of this difference in effect I've also chosen to split further analysis in the same way.

We've decided on a data retention strategy and added the schema to the whitelist in T220033. I don't see a need to keep this task open.

@JAllemandou : Thanks for clarifying that, very much appreciated!

From reading this, it sounds to me like it'll be possible to identify these events for pages because of the difference between the creation and first edit timestamps. What I'm wondering is if this means that we'll have two fields in mediawiki_page_history? In other words, that we'll have page_creation_timestamp, which reflects the creation timestamp per the logging table, and page_first_edit_timestamp, which has the timestamp of the earliest edit for that page?

Nothing that I updated P8309 with the current state of the schema, so that's also reflected here.

Noting that I've updated P8308 with the current state of the schema, so that's reflected here as well.

Agreed, it looks like we'll stay well below the limit. The team discussed this today, and I've added to our list of leading indicators to also measure the volume of hover events, to make sure that we stay below the limits. That will then also provide us with an indication of how this will scale when adding additional wikis.

Both @MMiller_WMF and I did some back-of-the-envelope estimation of active users today, and reach roughly the same conclusions, that we'll be well below thousands of active users at a given time. I based mine on the number of users who register on these wikis and used a generous estimate, and find that we might be looking at 10 visits per minute.

In T219432#5095574, @kostajh wrote:

@nettrom_WMF please confirm that https://meta.wikimedia.org/w/index.php?title=Schema%3AHomepageVisit&type=revision&diff=18999420&oldid=18984461 is OK with you. Thanks!

In T219435#5088554, @kostajh wrote:

Similarly to my comment on T219432, doesn't the email module have three states? You can need to add your email, need to confirm your email, or be complete.

Commented in T219432, but since we treat the UX of email module as "complete" or "incomplete" that's what the representation in the schema shows. We can definitely change this, just let me know what you prefer to have in the data.

In T219432#5086949, @MMiller_WMF wrote:

I just glanced over this, and I think it looks good. The only questions I have are:

• @nettrom_WMF -- I think we probably discussed this and I forgot exactly -- we are recording the states of the modules in the schema so that it is easy to answer the question of "what state was the homepage in when the user visited?" Is that right? To which of our research questions it that related?

@Niharika : I'm not sure who on the Analytics Engineering team is responsible, and I noticed that neither of the tasks are assigned to anyone. My current understanding is that these changes are likely to arrive with the next snapshot, which should be available in a few days, or the one after that (in early May).

In T219432#5070818, @kostajh wrote:

More questions:

1. Do you just care about reading vs editing, or do you want to know if the user was looking at e.g. page info (action=info), or page history (action=history), etc.
2. referer_namespace for now this is set based on the assumption that the user has clicked on 1) homepage tab from one of user page or user talk page or 2) personal tools link from any page. There is another scenario, where someone has [[Special:Homepage]] on some arbitary page; do we want to try to attempt to get the namespace from visits when a user clicks on a link on some random page? I assume we don't need that right now, but if we do please let me know.

In T219432#5069653, @kostajh wrote:

In other words, would a key/value approach work better than separate fields?

Do you mean doing something like `"module_states": "impact=unactivated;start_tutorial=complete;start_userpage=incomplete"? That could work althoughiIt seems like that would be cumbersome to parse, while adding a new property each time a new module is added doesn't seem like too much work. I'll defer to you on this one.

@Niharika : I picked this up again last week. At this point, I'd like to wait until partial block data is in the Data Lake to continue the work, because then I'll get block duration and edit revert detection for free rather than handle those myself. It would also be great to have IP blocks in the Data Lake, because so far a lot of the partial blocks are of IPs.

The measurement specification already listed the HelpPanel schema as an associated schema. I've now also updated the spec to mention how the helppanel_session_id and homepage_pageview_token fields will match for Homepage-related events.

@kostajh : This looks good to me, and I've gone through the measurement specification and updated it to reflect the current state of what we're capturing.

In T219432#5068374, @kostajh wrote:

@MMiller_WMF @nettrom_WMF I updated the schema per our meeting earlier today. I'm not sure how you'd like to have the start module and impact module states logged @nettrom_WMF, so please have a look at those fields in particular and if you'd prefer a different format let me know.

In T219432#5067041, @kostajh wrote:

I'm a bit confused at this point about what we're proposing to do here, so here's a question to try to clear that up: is the suggestion here to use EditorJourney for capturing this, rather than have an event in the proposed HomePage schema with action=impression?

Yes. The proposal is to log to EditorJourney schema rather than create a new one.

I'm a bit confused at this point about what we're proposing to do here, so here's a question to try to clear that up: is the suggestion here to use EditorJourney for capturing this, rather than have an event in the proposed HomePage schema with action=impression?

Just noting the discussion we had in T219387, as mentioned there this change is OK with me! I've also updated the measurement specification so it mentions the HelpPanel schema and how we'll be using it.

In T219387#5066123, @kostajh wrote:

In theory I would favor splitting out the question asking part, as that's how the code has evolved over time – there's now a generic question posting API and a generic question poster JS widget that's then used by three different implementations. That said, it will be more work to revise this (both creating a new schema, and updating the code) so unless it's a big help to your analysis then I'd say we leave it as is.

@MMiller_WMF : this looks good to go to me!

Should we consider renaming the schema since it's becoming a more generic schema for posting questions to pages? Or split out the question-asking part from the search & links part (in other words: de-normalize the schema)?

We've added table with the findings as a new section in the Help Panel experiment plan. There's also a description of the affected indicators and which ones we've acted upon.

I went back and dug into the data around the spike in Echo blacklist usage in October 2017 a bit more. From what I can tell, there doesn't seem to be a reason to suspect the data is invalid. Since the spike does make interpretation of the remainder of the data very difficult, and we're in this case mainly interested in development over time with a higher weight to more recent developments, I changed the start date of the Echo blacklist usage graphs to 2018-01-01 as that removes the spike.

@Trizek-WMF : I just posted T206380#5045440 with the results of the analysis, in case you'd like to refer to that.

I started this analysis on 2019-03-18, at which point we had 3,624 non-autocreated registrations since switching on the survey/control A/B test. Using the week of data prior to deployment I had earlier estimated the overall abandonment rate at 17.2%. A power analysis indicated that if the control group's abandonment rate equalled the estimate, we would be able to determine a significant change if the survey group's abandonment rate was outside the [13%,21%] range.

I used data from this schema in T216185. My experience was similar to what @Tbayer mentions in that some preferences appear to have issues with duplication, and some do not. In this case the echo-notifications-blacklist and email-blacklist preferences were affected. It also seems that the issue changed over time, in other words that some users logged lots of preference changes during certain periods, and not during others.

@Etonkovidova : Searched through the data from EditAttemptStep and HelpPanel in the Data Lake starting from 2019-01-01, and I didn't find any indications of HTML in the page_title field in either of those.

Using all available data from EditorJourney, excluding known test users, and allowing "visit" to be either viewing the page or opening the editor, the results are as follows:

I have completed the snapshot analyses as well as some data cleanup to remove the graph spike in email usage in July 2017. Will do a little bit more work to look into the Echo notification usage spike in October 2017 but I suspect that could be initial adoption and that we might prefer to omit the first few months and start graph on January 2018.

In T215986#5015962, @kostajh wrote:

@nettrom_WMF we discussed this in T215986#4970275. IIRC the reasoning was as you described – it somewhat duplicates existing search, and we didn't have a clear spec for how the search should be implemented in the UI.

What's the current state of search in this module? I went back and read through the comment above about the result of the deep dive, but didn't see much about it since then (and it's not present in the mockup). Curious about that, as it'll inform the measurement specification.

@TBolliger : Thanks again for providing those tasks and dates, much appreciated, and I'll make sure to incorporate those as I continue this work.

In T211950#5009094, @JAllemandou wrote:

However I think that logging table doesn't contain detailed historical information on partial blocks. This will prevent us to rebuild historical (and therefore more interesting) information on partial-blocks.
Can you talk to your team see if more detailed logging could be developped?

We've completed our initial experiment and found no obvious detrimental effect from the survey. We've also run a second experiment against variation C, and found that Var A is preferable. Currently, we are running an experiment on Vietnamese Wikipedia with Var A and a control group, to learn more about the abandonment rate on that wiki (ref T216668 and T216669).

Analysis by @MMiller_WMF and some digging by me indicates that events are not recorded correctly if the Visual Editor is used. Those events are instead recorded as reading events.

We monitored the SWAT deployment of the patch yesterday and confirmed through Kafka that reading events were flowing in. I've now also confirmed that reading events are present in the Data Lake.

During the analysis of usage of the email and Echo notification block lists, an issue with spikes in usage has come up. One key question about analyzing historical data of features like these is when these features were available, and in what capacity. For example, the email block list might've been available as a beta feature for a while. Secondly, there might've been announcements of the feature being available that would affect usage. Having that kind of information available aids analysis as it removes guesswork around why certain patterns emerge, so I'm documenting that here for future reference in other analysis tasks.

I couldn't find any reading-related events in the Data Lake, nor was I able to see any flow in when I monitored the events in Kafka. Instead, I find that my browser throws the following error to the JavaScript console:

In T217391#4993001, @MMiller_WMF wrote:

@nettrom_WMF -- do you think it will be better to manage the change to the "guide" link if the name in the configuration also changes? Or should we leave it as "guide" because its destination hasn't changed? We may want to look at the difference in traffic before and after the change was made.

We're fine with letting this data get purged as it otherwise would, so I'm closing this.

Just adding a comment to document that since we're also planning to soon add User & User talk, I'll regard the timestamp of that going live as the start of experimentation with additional contexts.

It's been about 7 hours since this went live, and using data from the replicated database, I get the following overview:

In T215986#4986717, @MMiller_WMF wrote:

Regarding instrumentation, I'll ping @nettrom_WMF for his thoughts on whether this should go into the HelpPanel schema or not. My thought is that given that we are writing separate code for this help module, and that it will behave somewhat differently than the help panel, it might give us more flexibility for it to store data separately.

In T215666#4982933, @MMiller_WMF wrote:

• @nettrom_WMF can look to see if events are flowing to the data lake.
  
  We would like all three of these things to be complete by Wednesday, so that we can start the 50/50 experiment on Thursday.

I've completed a quick analysis of the abandonment rate using the eight days of of data we have between the deployment of EditorJourney (on 2019-01-16) and the Welcome Survey (on 2019-01-24). In this analysis, I used the same approach that I used for the control group during the initial Welcome Survey experiment: does the user have more than three events[1] logged in the EditorJourney data? Split by whether the account was created on the mobile or desktop site, for Vietnamese the result is:

In T215664#4974525, @kostajh wrote:

@nettrom_WMF just noting that, when the help panel is loaded in the context of creating a new article, page ID will be 0. Of course you can filter these out by looking at page_id=0 AND editor_interface='reading' but just wanted to note it. For reading context, I could set page ID to -2 or some numeric value you'll know means it was redacted, just let me know.

In T215664#4973639, @kostajh wrote:

@nettrom_WMF here's an example event:

{
  "action": "impression",
  "action_data": "",
  "user_id": 1,
  "user_editcount": 127,
  "editor_interface": "reading",
  "is_mobile": false,
  "page_id": 0,
  "page_title": "",
  "page_ns": 12,
  "user_can_edit": true,
  "page_protection": "",
  "session_token": "ac0bc60ed5b559989816",
  "help_panel_session_id": "97487760b5c4b4519e76"
}

Note that page ID has to be an integer, not omitted entirely, so I've set it to 0. Let me know if that should be something else. For page title, I set it to an empty string, but I could make it some other value if you prefer.

The data captured by the HelpPanel schema has been QA'ed in various fashions since deployment. We monitored and verified data during deployment, have been investigating it afterwards, and also have checked it during analysis. Closing this task as resolved.

The instrumentation document has gone through the DACI and received approval. We've put the instrumentation in production and monitored the data periodically, and things seem to be going well. Am therefore closing this task.

In T216669#4970544, @MMiller_WMF wrote:

@nettrom_WMF -- EditorJourney was deployed in Vietnamese on Jan 16 and the welcome survey was deployed on Jan 24. Could we use the period in between to get a sense of the abandonment rate of users who don't get a survey (even though it's not a real A/B test because of the temporal change?)

In T215664#4955259, @kostajh wrote:

Storing reading there seems OK but not great. We could consider a new property on the schema like impression_mode and set that to "edit" or "view". @nettrom_WMF do you have a preference?

@MMiller_WMF : We should document the questions we have around this variation, similarly as we did for search. I think at this point, my main interest is behaviour across namespaces, in other words: how do users who see the help panel in the Help & Wikipedia namespaces differ in behaviour from those who see it when editing? We'll be able to answer that because we track the page namespace. While it might be interesting to know if certain pages are more likely to lead to questions being asked, I suspect we might also learn about that from the questions that are asked? It could also be something we consider looking into if we start seeing questions flow in.

Apart from "recommendations work", I'm not sure which parts of SuggestBot are relevant :) Maybe that at we did attempt to simplify the process of getting suggestions for newcomers through WP:Teahouse/Suggestions. It's still complicated, though, partly because we were operating within the constraints of MediaWiki and SuggestBot's infrastructure.

In T213974#4938729, @kostajh wrote:

I assume you prefer to just have [obfuscated page title string] and not [obfuscated surrounding elements and page title string]. In other words, don't include the surrounding elements in the string that's going to be obfuscated, but purge them entirely. Is that right?