Page MenuHomePhabricator

Evaluation of Zulip as a mentoring tool for Wikimedia outreach programs
Closed, ResolvedPublic

Description

Zulip team members were kind enough to let us have access to the Zulip beta earlier this year (link to our Zulip instance).

As suggested by outreach program mentors from previous rounds of GSOC/Outreachy in T150732, we decided to use Zulip for the new rounds.

They also shared with us some guides that helped us in geting started & following best community building practices. Links are below:

In about a month or so, we are nearing to the completion of the GSOC/Outreachy rounds and we would like to evaluate Zulip as a mentoring tool. We are planning on asking students and mentors, some specific questions via a program feedback survey that will help us evaluate the usefulness of the tool in the context of our outreach programs.

After consolidating the results, we will decide what our next steps are on this!

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 11 2017, 10:21 PM
jayvdb added a subscriber: jayvdb.Jul 12 2017, 12:51 AM
srishakatux raised the priority of this task from Low to High.Oct 4 2017, 6:39 AM
srishakatux moved this task from Jul-Sep 2017 to Oct-Dec 2017 on the Developer-Advocacy board.
srishakatux added a comment.EditedOct 7 2017, 1:37 AM

What we learned about the use of Zulip as a mentoring tool

  • Most mentors didn't use the tool to communicate with students and org admins. Here are comments from some of them:
    • "I did not use it actively, but I got emails from it occasionally, which was useful"
    • "I greatly prefer phabricator and I won't use it again. But maybe worth keeping it as an option"
    • "I and our student never used or logged in to Zulip, so I can't really comment on this"
    • "I only checked it when I got email notifications. That worked fine, except I think my co-mentor once didn't get an email notification"
  • One student wrote: "While it is open source and ethuses the community it is not a reliable source of communication"
  • Analysis and next steps:
    • Most of the mentors didn't use the tool to communicate with students and mentors. It is not surprising as they were already on Phabricator group chat before the projects were accepted. Some of the mentors were using different synchronous communication channels to meet with their students. So, this is a fair assumption that switching to Zulip, an entirely new platform would have been extra work for them. Still, some of them who were familiar with the platform emphasized that email notifications they received were useful and that we could worth keeping Zulip as an option. As an organization administrator, the tool turned out to be extremely useful in keeping a close communication with students all throughout the program, specially in matters of urgent sync up. Zulip platform has a nice feature that allows threaded discussions. It was helpful that way to have organized and seperate discussions about each phase of the program. It might be worth to continue using the tool in the next round to be able to draw more concrete conclusions. One additional thing to try would be to introduce Zulip to mentors and students when the application period opens.

The summary above is my personal opinion, a second opinion would be helpful. Whatever we end up deciding as a next step, I will follow up with Zulip team accordingly.

Sidenote: Overall lessons learned related to GSoC/ Outreachy including the summary about Zulip added above is here https://www.mediawiki.org/w/index.php?title=Google_Summer_of_Code/2017/Lessons_Learned, also linked from the administrators pages of both GSoC and Outreachy https://www.mediawiki.org/wiki/Google_Summer_of_Code/Administrators

Tgr added a subscriber: Tgr.Oct 8 2017, 9:37 PM

Is this still a thing that exists? I found this task by accident; I doubt many of the mentors for the Outreachy winter round are aware that there is a Zulip trial going on (if it is still ongoing). I've been using Conpherence which is very broken (especially due to T765, although that seems to be on the verge of being fixed now); an alternative would have been interesting. Although in the long term I would really, really like to see something with good IRC integration; any other approach will just fragment our mentor/helper community IMO.

This comment was removed by srishakatux.
Qgil added a subscriber: Qgil.Oct 12 2017, 4:00 PM

Some of the mentors were using different synchronous communication channels to meet with their students.

Do we know which ones?

Some of the mentors were using different synchronous communication channels to meet with their students.

Do we know which ones?

Google Hangouts, IRC, Slack, etc.

Qgil added a comment.Oct 13 2017, 3:39 PM

I'm intrigued at why Slack is considered synchronous and Zulip not. I guess because Slack has a mobile app?

srishakatux closed this task as Resolved.Oct 19 2017, 9:35 PM

@Qgil Zulip has a mobile app too.

We will further evaluate Zulip, after the end of GSoC 2018 round. I'm closing this task for now. See more details here > T150732#3698157

Qgil awarded a token.Oct 20 2017, 10:24 AM

Just a quick anecdote. I used zulip as part of GSOC 2017 mentoring for all the (weekly) meetings which we held, and we also used gitter to communicate occasionally for some of the more technical questions that needed an immediate response to avoid delays in the project. This is because gitter notifications worked better than zulip or IRC at ensuring a rapid response on specific chat rooms, because it also sends an email with any unread messages. OTOH, Zulip was better for scheduled meetings, and discussion threads which spanned large periods of time, and for keeping in contact with the other GSOC people (admins, mentors, students). Both clients were light weight, and low permissions, so it wasnt a huge drama for us to have multiple clients installed. We also both used IRC occasionally also, but not as a mechanism for having a discussion between mentor and mentee.

At the Mentor Summit, many GSOC orgs indicated they have migrated to, or are migrating to, either it or RocketChat for their GSOC programs. They are usually also keeping IRC, and often bridging everything together with Matrix behind the scenes. Zulip org is happily providing free hosting for any of these orgs so the other orgs can focus on their mission, and Zulip focus on its mission. I am happy to hear that Wikimedia will continue to keep a Zulip instance available for mentoring.

Zulip is rapidly improving with lots of GSOC and GCI fueled development in addition to the core team. As far as I can see, it is the fastest moving open source chat system. They have recently released an errbot connector, which enables many existing bots designed for gitter or Telegram to operate on Zulip also.

Worth noting that since the last evaluation of options, gitter has been 'purchased' by GitLab (who also 'owns' MatterMost) and was open sourced, so that is also another option on the table for self-hosting, if we wanted to go down that path. However development of gitter is moving slowly compared to Zulip.