Summarizing some of the things discussed with @dchen during the Editing design meeting. We need to test the visual diffs in two scenarios. It seems that it might be easier to test them on the history page than on the "preview changes" step of visual editor. It might be a good idea to use the history page use-case for the pilot test because...

While editing

The diffs might be easier to interpret since the user has recently made those changes and knows the context within which they were made. They are meant for confirmation rather than investigation.

They are however slightly harder to test because:

1. The test needs to be moderated and the tester needs to be explained what kind of edit to make
2. The prototype will have to be "real" in that it should reflect the changes that were made by the user (obviously) and it might be a significantly larger effort to make something like that

History page

These are used to find out what changed in the page in the past and the viewer has no context. Its for investigation.

They might be easier to test because:

1. No moderation required about the type of edit to be made (since there is no edit to be made)
2. No interaction with real/beta website
3. Easier to mock-up (since it is not dependent on user input)

I had two general queries about the methodology.

Comparison

Is a direct comparison between the two diffs required?

Do we want to compare their effectiveness or speed of understanding, or user preference (which might differ for experienced editors and novices)? On which variables do we want to compare them?

What testers test

I also wanted to understand what we'll be showing to the testers.

• Will we show the same tester two different versions of the same diff?
  • Would that make it easier to understand the diff the second time?
• Will we show either the wikitext or the visual diff of a particular diff to the tester, and the other one to another tester?
  • If so, would it be fair to compare their effectiveness as perceived by different people?
• If we re only going to be running test on new/non-editors, are we planning to show them the wikitext diff, or just options on the visual diff?

@dchen has been making notes about this. We should probably either close this task, point to where her notes are, and discuss there, or move her notes to this task and discuss here.

In T171456#3541293, @Deskana wrote:

@dchen has been making notes about this. We should probably either close this task, point to where her notes are, and discuss there, or move her notes to this task and discuss here.

If you're referring to the Google Docs, my comments there refer to this task.

@dchen @Deskana would it be ok to move the notes to phab (whether in this task or another)?

In T171456#3548150, @Prtksxna wrote:

In T171456#3541293, @Deskana wrote:

@dchen @Deskana would it be ok to move the notes to phab (whether in this task or another)?

Or a wiki page, I'm not sure which is best. It would be good to get them public, in some form. :-)

@dchen informs me that this research should hopefully be finished by the end of the week.

closing ticket - initial round of testing completed.

write-up of findings can be found here.

further testing on visual diffs will be discussed and, if needed, new phab tickets added.