I totally agree. I think that Visual Diffs allows us to easily recreate some individual examples of realistic diffs with mockups and place them in front of users to get feedback. I'd definitely recommend doing such exercise before investing time in the code to support possible directions. @dchen can provide more thoughts and ideas for such process.
Summarizing some of the things discussed with @dchen during the Editing design meeting. We need to test the visual diffs in two scenarios. It seems that it might be easier to test them on the history page than on the "preview changes" step of visual editor. It might be a good idea to use the history page use-case for the pilot test because...
The diffs might be easier to interpret since the user has recently made those changes and knows the context within which they were made. They are meant for confirmation rather than investigation.
They are however slightly harder to test because:
- The test needs to be moderated and the tester needs to be explained what kind of edit to make
- The prototype will have to be "real" in that it should reflect the changes that were made by the user (obviously) and it might be a significantly larger effort to make something like that
These are used to find out what changed in the page in the past and the viewer has no context. Its for investigation.
They might be easier to test because:
- No moderation required about the type of edit to be made (since there is no edit to be made)
- No interaction with real/beta website
- Easier to mock-up (since it is not dependent on user input)
I had two general queries about the methodology.
Is a direct comparison between the two diffs required?
Do we want to compare their effectiveness or speed of understanding, or user preference (which might differ for experienced editors and novices)? On which variables do we want to compare them?
What testers test
I also wanted to understand what we'll be showing to the testers.
- Will we show the same tester two different versions of the same diff?
- Would that make it easier to understand the diff the second time?
- Will we show either the wikitext or the visual diff of a particular diff to the tester, and the other one to another tester?
- If so, would it be fair to compare their effectiveness as perceived by different people?
- If we re only going to be running test on new/non-editors, are we planning to show them the wikitext diff, or just options on the visual diff?