I partially disagree about this routing. In future, we will have two user interfaces: 1- One for labeling 2- One for admin actions. I think stats should be pushed into 2 and 1 should be for one interface for a whole wiki not different pages for different campaigns in one wiki.

Why not have the admin functions show up in the regular user interface?

I was imagining that we'd list out all campaigns for a wiki on one page with some details (e.g. allowing admin actions like "enable"/"disable") and then include more information about a specific campaign on its page (e.g. "request workset" or "upload items")

@Ladsgroup and I talked at the Wikimedia Hackathon (2017) and we decided to try to bring in a designer to talk with us about a general strategy for directing users around the Wikilabels interface. @Pginer-WMF (cc @jmatazzoni), would you be able to spare a small amount of time to discuss how we're looking to arrange the pages (workflow) of Wikilabels users with us? I'm thinking we could make enough progress thinking about designs in a meeting in order to work on our own. If this all seems OK, I can either put a task on the Collaboration-Team-Triage board or just schedule a meeting for when we have some timezone overlap?

Also, I have some sketches that I'll share soon.

Just for adding some context: What is the "admin" in this context and how does that affect the interface (does there need to be an additional page? Elements that may be needed to be integrating in the labeling UI?)

In T165046#3298618, @Halfak wrote:

Also, I have some sketches that I'll share soon.

That would be awesome for understanding it better (You probably know it, but one can Ctrl+V images in phabricator, which is very useful)

It's pretty rough. I was hoping to find time to re-sketch in a vector editor.

Gist is:

1. Home page /. Lists out campaigns by recent events (creation, completion, crossing critical thresholds, etc.)
2. Wiki page /campaigns/<wiki>. Lists out campaigns in a wiki by date of creation.
3. Campaign page /campaigns/<wiki>/<id>. Lists out details and progress information.
4. Labeling page /campaigns/<wiki>/<id>/<userid>. Lists out worksets for a current user. Allows a user to request new worksets.
5. Labeler overlay (n/a). Allows a user to label tasks from a single workset.

Just for adding some context: What is the "admin" in this context and how does that affect the interface (does there need to be an additional page? Elements that may be needed to be integrating in the labeling UI?)

An admin is someone with advanced rights in the labeling system. They create and manage campaigns, whitelist and block users, and do other types of mopping up.

@Jan_Dittrich: Some more context on admin T102335: [Spec] Implement admin actions for wikilabels

In T165046#3298616, @Halfak wrote:

@Pginer-WMF (cc @jmatazzoni), would you be able to spare a small amount of time to discuss how we're looking to arrange the pages (workflow) of Wikilabels users with us? I'm thinking we could make enough progress thinking about designs in a meeting in order to work on our own. If this all seems OK, I can either put a task on the Collaboration-Team-Triage board or just schedule a meeting for when we have some timezone overlap?

Sorry for the delay in replying. I'd be happy to help. I cannot say much about specific time commitments until my work plan for next quarter is more clear, but finding time for a meeting or an upcoming hackathon (Wikimania-Hackathon-2017) collaboration seems doable.

Some initial thoughts (based on my limited knowledge of the tool): the hierarchy of concepts you propose seems good but we need to determine how prominently expose it to the users. It would be good to think about what "campaigns" and "worksets" are, how are they meaningful for users and what motivates users to participate/complete them.

Right on!

So, a Campaign is a chunk of work for a group if people to do over a period of time. A Workset is a chunk of work from a Campaign for me to do in one sitting. I might do multiple worksets in one sitting, but a workset should never be so large that it will take more than one sitting to complete it.

Motivations vary here. The people who do labeling work to train models in ORES are motivated by what a prediction model will allow them to do (e.g. catch vandalism and stuff). There are other types of labeling work though. E.g. I worked with some folks to label a random sample of pages from the New Page Patrol backlog so that we could get a sense for what kind of pages were in that backlog. I'd say that "model training" and "content analysis" are two dominant motivations for Wiki labels use. Individuals might want the model or the results of the content analysis completed.

As a user, I want to keep track of my worksets so see the work that I have done and possibly to review and update a label that I'd applied in the past. I might want to search through all labels I applied by providing an identifier. Or I might just want to look up a label/task by the timestamp at which I labeled it. This will be less common. Mostly, I'm just interested in labeling stuff to get to the goal.

Thanks for the clarifications @Halfak

Regarding the "workset" concept I wonder if that unit of work is defined because it is convenient for the machine (i.e., needed for technical reasons) or for humans (i.e., to better organise their work). If I am labelling edits to distinguish vandalism, why do I prefer to get a group of edits rather than a more continuous list with all items in the campaign? are those worksets sharing some properties that make them distinctive (e.g., possible vandalism on scientist biographies)?

The quick answer is both! There's a UX theoretical reason why people like doing work in chunks that take less than 5 minutes (see some of my own research, but there's a lot more). There's also an important technical reason to do this. It helps make sure that we minimize duplicate work.

chunks that take less than 5 minutes

And aside of the timeframe: It probably gives a sense of achievement to have finished a workset (or however we shall call it in the future), a continuous list is never done…

In T165046#3383143, @Halfak wrote:

The quick answer is both! There's a UX theoretical reason why people like doing work in chunks that take less than 5 minutes (see some of my own research, but there's a lot more). There's also an important technical reason to do this. It helps make sure that we minimize duplicate work.

In T165046#3385250, @Jan_Dittrich wrote:

chunks that take less than 5 minutes

And aside of the timeframe: It probably gives a sense of achievement to have finished a workset (or however we shall call it in the future), a continuous list is never done…

Providing content to users in small chunks and avoiding duplications makes sense. Let's imagine that a user gets to a page of a campaign where there are 5 items to complete, and the user has an option to "get 5 more". The user would be completing those 5-item "worksets" without being exposed to a workset concept.

Is there a need for worksets to be explicitly/prominently presented as such? In that case, what constitutes their identity to distinguish them from each other?

One benefit to presenting worksets is that it would make it easier for users to review old work. Other than the tasks in a workset, they can be identified by their temporal sequence (e.g. "I remember seeing something weird in (the last workset I finished|the workset I worked on a couple days ago)"). It would be nice if we could report summary statistics on each workset (e.g. "50 revisions, 45 goodfaith, 6 damaging, 2 unsure") to help users remember them.

FWIW, we know users do this because we implemented deterministic ordering of tasks within a workset so that finding old tasks would be easier. (e.g. "I remember that weird revision was towards the middle of a workset I did yesterday.")

The user would be completing those 5-item "worksets" without being exposed to a workset concept.

@Pginer-WMF: So the idea would be to give the explicit concept "Workset" less emphasis while still providing tasks in chunks? If yes, it the idea, that "workset" is irritating and/or could be implicitly conveyed? Or something else?

In T165046#3414738, @Jan_Dittrich wrote:

The user would be completing those 5-item "worksets" without being exposed to a workset concept.

@Pginer-WMF: So the idea would be to give the explicit concept "Workset" less emphasis while still providing tasks in chunks? If yes, it the idea, that "workset" is irritating and/or could be implicitly conveyed? Or something else?

The idea is to avoid a hierarchy of concepts that is unnecessarily complex. While campaigns seemed an intuitive concept (an area of contribution around a topic that is active for me to participate), the concept of "workset" seemed more artificial to me. So I wanted to learn which are the goals that worksets try to support in order to see which options exist to support them and how much prominent to present them to the user.

I don't know enough about this area to recommend a solution, I'm describing example solutions that follow a different direction to surface such knowledge. For now, considering the different purposes described (providing tasks in small chunks, avoiding duplication of work, and being able to find specific units of work or statistics about them) seem to be solvable without the need of exposing another concept to users. That is, you don't need to have a "my worksets" section or label anything as a "workset". You can just provide access to work items in small chunks and show the user progress grouped by time (days and/or hours) to support the underlying needs without introducing more concepts.

It's interesting that you raise this issue. I've not heard from any users that it was a confusing concept. Is having "worksets" without calling them "worksets" less confusing?

I guess I'm not sure I understand why we would not fully embrace a concept that users will be interacting with often.

In T165046#3415745, @Halfak wrote:

It's interesting that you raise this issue. I've not heard from any users that it was a confusing concept. Is having "worksets" without calling them "worksets" less confusing?

I think we should expose concepts to users when they bring something useful, not just when their presence does not create confusion. I was describing the hypothesis of having a flatter structure to the eyes of the users in order to surface what would be missing with a simpler model, and understand what having explicit "worksets" provide to users.

Let me put it in a different way (with a totally made up example), if someone was proposing to group further the work items inside a workset into 2-item groups named "workpairs", I think that it would be reasonable to ask what is helpful to both (a) group items in pairs and (b) exposing users with a new concept. Regarding (a) it would be useful to know that users prefer completing some tasks having items side-by-side. Regarding (b) it would be useful to know that the items in the group are opposites and surfacing they are a pair helps for the user to identify that they form the opposite of a spectrum. However, knowing that nobody complained about "workpairs" does not provide many
answers to either (a) or (b).

I guess I'm not sure I understand why we would not fully embrace a concept that users will be interacting with often.

Maybe hearing more about how users "interacting with [worksets] often" would help to surface how exposing the workset concept to users is helpful.