Page MenuHomePhabricator

Summarize learnings from user testing
Closed, ResolvedPublic

Description

Background
The week of August 11, the apps team participated in a hack week to test out WikiGames as a concept. In Q3 the Android team will return to this work.

Task

  • Work with Design Research and other designs that worked on WikiTrivia to develop a comprehensive document with the analysis of the prototype and demo, and include recommendations and open questions

Event Timeline

Some thoughts following ~2 weeks of discussing, sketching, and playing around with different ideas for a Wikipedia Trivia Game. Leaving out topics that are generic to any game (e.g. leaderboards, streaks, etc.).

Initial ideation phase

Before I joined the conversation the rough idea for the game was:

  • programmatically get a list of historical events from an API (e.g. Selected anniversaries)
  • present players with a historical event (e.g. "The Treaty of Selymbria was concluded between the Republic of Venice and the Ottoman prince Musa Çelebi."), and have them guess which year the event occurred on

Some of the questions/alternatives that had been considered were:

  • Should people be presented with an event and guess the year, or be presented with a year and guess which event (from a short list) occurred on that year?
  • If guessing the year of a given event, should it be multiple choice or fill in the blank? If it's fill in the blank, could scoring work such that getting the correct century or decade is worth something?
  • Should the list of events/questions be randomly chosen from the output of the API, or should they be written/curated by a person?
Further considerations

As we began to sketch out the basic flow of the game additional questions came up:

  • How many questions should be in a round of the game? How much time would people want to spend playing the game each day?
  • A lot of the events returned from the API are tragic (plane crashes, mass shootings, genocides, hate crimes, etc.) — how would people feel about those?
  • What is the right level of difficulty for the game (determined both by the questions and the answer choices)?
    • For the multiple choice approach, how would we generate the incorrect answers to achieve the right level of difficulty?
  • How can we enable/encourage people to further engage with the relevant knowledge/content after each game?
  • How branded should the game be?
    • Is there a more general WikiGames brand, and then a specific brand for this game? Is there a family of games? What is the relationship between the general brand and the specific game brand?
    • What's the relationship between the game brand(s) and the rest of the app — should it stand out? Be more playful? Follow the Wikipedia Brand Guidelines more than the Android/Material style?
  • What are the goals for the game? Education/knowledge/learning? Fun? Competition?
Personal opinions & more nuanced stuff

After playing around with the 4 game prototypes I created, some more nuanced/subjective questions came up. The highest level question was: what makes a game a game? I think matching historical events with dates is an activity that you can build a game around, however simply getting a feed of (random) events and having to guess the year (prototype 1) ended up feeling a bit flat to me. Comparing this to the Wikitrivia game where the challenge is to place events on a timeline in the correct order, helps add some perspective/shape to this somewhat amorphous question:

  1. Placing event cards on the timeline makes it so that each "question" has a relationship to every other question. It doesn't feel like you're just encountering random/discrete questions/events. Related to this, it feels like you are building/constructing something. Both in a material sense — you are making this sort of map/timeline. And also in the sense of building knowledge via this sequence of events. Even though the events are random, there are these kind of "ah-ha" moments when you learn that something unexpectedly happened before/after something else, and it feels like you're building up some larger understanding of the shape of history.
  1. The simple interactivity of dragging an event card onto the timeline gives me some feeling of engagement that selecting a multiple choice answer doesn't

The 4th prototype, which was inspired by the Wikitrivia game, to me felt the most like a "game". In addition to the points above, Carolyn also used the concept of a puzzle to explain why this game was more interesting than the other ones, which I think is a concept worth investigating further.

A few threads that follow from this:

  1. When answering random/discrete questions you are technically still learning/building knowledge. However there is nothing particularly interesting about what you're learning. There is no narrative, no specific topical area. There is no larger insight/learning beyond the individual nodes. I think this is part of why it lacks excitement. Some questions that follow:
    • What would happen if there was some logical progression/connection between the questions? How can the questions relate to each other such that it feels like you're building an understanding?
    • What would happen if there was some kind of lattice that the questions/events were being mapped onto?
    • What would happen if there was some kind of narrative/theme to the questions? Somehow conveying insights or otherwise unexpected aspects of history

I think another way to approach the challenge of making it feel like a game is by adding competition mechanics. If you were competing in a live round of trivia, or a trivia tournament, or going to head-to-head with a friend, etc. it would probably make the game feel more alive. Even things like a leaderboard or knowing how you performed relative to other people help to some extent. However I think this would be a kind of secondary solve, i.e. packaging the game in an interesting way rather than doing the more difficult work of making the game interesting in and of itself.

In conclusion: my instinct is that without some kind of curation (as we were calling it), it will be difficult to make an interesting game. A limited form of curation might be constraining the set of events/questions for a given day or week to a certain decade or century, at least adding some coherence. A step beyond that might be using a list of keywords to create topically coherent sets of events/questions. Ultimately I think if the questions were chosen manually the game would be significantly more interesting. And I think if we're planning on doing a limited experiment it's worth testing this. Run two versions of the game for a month — one with random questions, one with manually chosen questions — and compare the play/retention rates.