Page MenuHomePhabricator

Creating a Facebook Messenger Bot for Wikipedia
Open, Needs TriagePublic

Description

Hello Everyone,

I am Balaji from India. I would like to propose a Facebook Messenger bot project which Wikimedia could integrate into the Wikipedia's official Facebook page. Major features of the bot will be similar to the Wikipedia Android and iOS apps.

  1. Article of the day - Users can subscribe to get an article of the day from Wikipedia through their Facebook Messenger.
  2. Picture of the day - Users can subscribe to get a picture of the day from Wikipedia through their Facebook Messenger.
  3. Top read - Users can request the bot at any time to show the top read Wikipedia articles of the hour/day/week.
  4. On this day - Users can request the bot to get major event happened on the current date.
  5. Random article - Users can request the bot the recommend any random article from Wikipedia
  6. Places around the user - Users can share their location and get the places around them
  7. Saved articles - Using this feature, the bot can let the user save any article for their future reads
  8. Read History - Using this feature, the bot will show the articles read by the user previously.

I would like to get feedback from the Wikimedia community on this proposal. Upon approval of the community, I would like to work on this bot in my summer vacation.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 7 2018, 11:22 AM

Hello Balaji,

I work on the Wikimedia Foundation's social media team and could see this being a really useful tool that would allow people to pull content rather than having us push it. I am looking forward to seeing feedback from the community — this would be particularly useful if the bot could work in several languages, which would help us reach a more global audience on social media (and could possibly be adapted for chat platforms such as WhatsApp and others...)

Mel

Hello Balaji,
I work on the Wikimedia Foundation's social media team and could see this being a really useful tool that would allow people to pull content rather than having us push it. I am looking forward to seeing feedback from the community — this would be particularly useful if the bot could work in several languages, which would help us reach a more global audience on social media (and could possibly be adapted for chat platforms such as WhatsApp and others...)
Mel

Thanks for your kind words. Yes, It could be possible to roll out the bot in many languages if we train our NLP API like Wit.ai properly in the corresponding languages.

RHo added a subscriber: RHo.May 7 2018, 5:17 PM
Balaji030698 added a comment.EditedMay 9 2018, 8:42 AM

Will there be any Wikimedia policy related issue in using Facebook's Wit.ai as NLP API for this project?

@Balaji030698 I will ask our Policy and Legal team today and let you know what they say.

@Balaji030698 I will ask our Policy and Legal team today and let you know what they say.

Thanks. Looking forward to hear their response.

@MelodyKramer Any update from the Policy and Legal team?

Hi @Balaji030698 - @Slaporte (from the legal team) will respond shortly! I know he was juggling quite a lot last week, but it's on his to-do list.

Hi Everyone,
As a part of creating Facebook chatbot for Wikipedia Facebook page, I have created a Ruby gem to get Wikipedia contents like the random article, Today's featured article, the image of the day, and daily news. You can find the source code of my Ruby gem Project in my GitHub repository.

@Dereckson @ArielGlenn If possible, Please look into my project source code and share your feedback. Also, What is the other preliminary things I can do while waiting for the policy team response.

I like the app so far, though I haven't tested it out yet. One thing that might be nice is to allow users to configure it for wikipedias in other languages; I suppose for now you would want to stick the the top ten or so, if they have similar featured feeds.

Another thing you'll want to bear in mind, if any of these routines are to be called repeatedly in a shot period of time, is the maxlag parameter: https://www.mediawiki.org/wiki/Manual:Maxlag_parameter
and generally everything here: https://www.mediawiki.org/wiki/API:Etiquette

I don't see why you can't do a proof of concept using whatever ml/ai backend you want for now, unless you think switching out backends later will be a huge amount of work.

Welp. You're using the restbase stuff so the above isn't valid for it, my bad.

But do set a good user agent someplace, as it says in https://www.mediawiki.org/api/rest_v1/

Welp. You're using the restbase stuff so the above isn't valid for it, my bad.
But do set a good user agent someplace, as it says in https://www.mediawiki.org/api/rest_v1/

@ArielGlenn Sure! I will add an user-agent option in the gem soon. Regarding proof of concept, I will start working on it and I will update it here as soon as I complete it.

Cirdan added a subscriber: Cirdan.May 24 2018, 9:05 AM

According https://www.mediawiki.org/wiki/REST_API, the REST API is a Wikimedia-specific but not Wikipedia-specific API.

If you repurpose the library to handle other projects like de.wikipedia etc. perhaps it would be a good idea to support other sites like wiktionary, wikinews, etc. too

In such case, Wikimedia REST Client (wikimedia_rest_client gem name, require 'wikimedia_rest_client' or better wikimedia-rest_client, require wikimedia/rest_client) would be a better name.

A MediaWiki page has a lot of properties, as such your class Page has too many methods. I'm not sure if it's valuable to provide a specific class for content urls or images (with source, width, height) to reduce a little the amount.

Will there be any Wikimedia policy related issue in using Facebook's Wit.ai as NLP API for this project?

Hey @Balaji030698, I spoke with Mel about this question. Thanks for your patience :)

One question, do you plan to use Wikimedia Cloud Services or Wikimedia ToolForge?

On licensing, it looks like the Wit.ai API promotes open apps, and has a few open clients on github, but is not itself an open source-licensed service. Per our guiding principle on freedom and open source, we avoid relying on proprietary services for Wikimedia's core services. This isn't a strict requirement for a volunteer project like yours, though. I recommend looking at some other open source NLP frameworks/services to see if any of them will meet your needs. If Wit is the best tool for the job, in your opinion, then you can use it.

It will be your responsibility to make sure the app follows Wit's and Facebook's terms of service. On a related note, I saw that Facebook recently updated their review process for bots, which requires developers to follow a few rules in their bot policies.

Thank you for working on this project. I cannot provide legal advice, but I'm happy to see if I can help if you have any questions about Wikimedia policies.

According https://www.mediawiki.org/wiki/REST_API, the REST API is a Wikimedia-specific but not Wikipedia-specific API.
If you repurpose the library to handle other projects like de.wikipedia etc. perhaps it would be a good idea to support other sites like wiktionary, wikinews, etc. too
In such case, Wikimedia REST Client (wikimedia_rest_client gem name, require 'wikimedia_rest_client' or better wikimedia-rest_client, require wikimedia/rest_client) would be a better name.

Hi Dereckson,
Thanks for your feedback. I'll add multi-language support in the next version.

Since the requesting Base URL is "https://en.wikipedia.org/api/rest_v1" I thought of naming the Ruby gem as "wikipedia_rest_client". Since I have already released the gem, I couldn't find a way to rename the current gem. I could work on the Wikimedia gem after finishing the Facebook bot. Is that fine?

Balaji030698 added a comment.EditedMay 26 2018, 4:48 AM

A MediaWiki page has a lot of properties, as such your class Page has too many methods. I'm not sure if it's valuable to provide a specific class for content urls or images (with source, width, height) to reduce a little the amount.

I have provided all the parameters in the JSON response since I could be helpful for some other projects for their needs.

Balaji030698 added a comment.EditedMay 26 2018, 5:29 AM

Will there be any Wikimedia policy related issue in using Facebook's Wit.ai as NLP API for this project?

Hey @Balaji030698, I spoke with Mel about this question. Thanks for your patience :)
One question, do you plan to use Wikimedia Cloud Services or Wikimedia ToolForge?
On licensing, it looks like the Wit.ai API promotes open apps, and has a few open clients on github, but is not itself an open source-licensed service. Per our guiding principle on freedom and open source, we avoid relying on proprietary services for Wikimedia's core services. This isn't a strict requirement for a volunteer project like yours, though. I recommend looking at some other open source NLP frameworks/services to see if any of them will meet your needs. If Wit is the best tool for the job, in your opinion, then you can use it.
It will be your responsibility to make sure the app follows Wit's and Facebook's terms of service. On a related note, I saw that Facebook recently updated their review process for bots, which requires developers to follow a few rules in their bot policies.
Thank you for working on this project. I cannot provide legal advice, but I'm happy to see if I can help if you have any questions about Wikimedia policies.

Hi @Slaporte,
Since I'm new to Wikipedia contributions, I don't know capabilities of Wikimedia cloud services and Tool Forge. I will discuss it with senior community members on the tech side and get back on this.

I have come across an open-source NLP framework Rasa as an alternative to Wit. One of the main disadvantages on Rasa is that I couldn't find a feedback system. Whereas Wit's has a feedback system to view all the user inputs and update our NLP models in real-time. I would prefer to use it in the initial phases to understand the user behaviors. If required, we can switch to Rasa once our NLP models are matured.

Regarding Facebook's ToS, I have prior experience in developing Facebook Messenger bot. I will make sure, we comply with all their guidelines and policies.

Balaji030698 added a comment.EditedJun 1 2018, 5:11 PM

Hi everyone,
I have built the PoC for the Facebook Messenger bot. You can find the demo video here.
Currently, I have built it for English and Tamil. It can be extended to other languages if volunteers contribute to training the NLP models. You can find the source code at my GitHub repository.

Balaji030698 added a comment.EditedJun 1 2018, 5:16 PM

I designed it in such a way that, the bot's preferred language will be based on the language set in the mobile and Facebook account language settings. If you need access to the bot, ping me in IRC at Balaji030698

@Balaji030698 I would love to test out the bot - do I need to friend you on FB?

Great demo, I'm going to check out the code soon!

Balaji030698 added a comment.EditedJun 5 2018, 3:42 AM

@Balaji030698 I would love to test out the bot - do I need to friend you on FB?

You need not be my friend on FB. Just send me your profile URL link. I'll add you to the testers list for that bot.
Since I'm using Heroku free tier, the server goes down after 30 minutes of inactivity and takes 1 minute to respond to the first message alone.

Balaji030698 added a comment.EditedJun 5 2018, 2:20 PM

Done. I've added you to the testers list for the bot. You can find the bot at m.me/wikipediatestbot . Try out the bot and give me your feedback to improve its performance :)

@MelodyKramer Have you tried the bot? Any feedback?

Vvjjkkii renamed this task from Creating a Facebook Messenger Bot for Wikipedia to ohdaaaaaaa.Jul 1 2018, 1:11 AM
Vvjjkkii removed Balaji030698 as the assignee of this task.
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
Balaji030698 renamed this task from ohdaaaaaaa to Creating a Facebook Messenger Bot for Wikipedia.Jul 1 2018, 9:00 AM
Balaji030698 claimed this task.
Balaji030698 updated the task description. (Show Details)
Balaji030698 added a subscriber: Aklapper.
CommunityTechBot raised the priority of this task from High to Needs Triage.Jul 5 2018, 6:37 PM

Hi everyone,
I have deployed the initial version of this project in Toolforge and Its working fine now. I would like to hear feedback from you.
Since it hasn't been published yet on Facebook, please send me your Facebook profile URL link to my IRC (Balaji030698). I will add you to the testers list so that you can try the chatbot.

@Balaji030698 Sharing the link of my FB profile: https://www.facebook.com/SrishAkaTux. I can help test it!

@SrishtiSethi Thank you for your interest in this project. I've added you as a tester. Try out the chatbot at https://www.messenger.com/t/wikipediatestbot. and provide your feedback about it.

Very neat implementation :-) I loved your bot, it works flawlessly!

I could only find one minor bug, and then I have a few suggestions for improving content navigation:

  • In the 'News' section, when I click on the 'Get Summary' button, it does not generate more content, maybe 2-3 more lines. Might be nice if full text is shown for the news headlines and the Get summary (text replaced with 'Read More') could be redirected to the full news article. I am guessing that you are pulling the headlines from https://en.wikipedia.org/wiki/Main_Page. If, it is not possible to link the headline to an article, then, in that case, you can show the full text without the buttons.
  • The above point also applies to content for "On this day" section.
  • Again, small thing but maybe Random Article + Featured could be combined on the menu. On other note, all options could be on one single list to avoid multiple clicks. And, then 'Help' and 'My Subscriptions' menu items could be separated from others by a line break.
  • For the 'Most Read' section, there is a small typo that I found. For one of the articles, Page title is shown along with Italic tags: https://en.wikipedia.org/wiki/The_Meg (see screenshot below).

@SrishtiSethi Thanks for your valuable feedback :)

  • The Wikimedia REST API that I'm using to fetch contents only gives the headlines from https://en.wikipedia.org/wiki/Main_Page. In Facebook Messenger, there is a constraint that the card view can have a maximum of up to 80 letters in the text field. If the headline message exceeded 80 letters, then the text will be ended with dots. So that I've added a button 'Get summary' to get full text in the form of text message. Instead of this, I'll look into the possibilities for redirecting it to the article.
  • In Facebook Messenger chatbot, it can have a maximum of up to 3 buttons on the main menu, and 5 buttons in the secondary menu. So, I can not give all the buttons on the single list. So I provided them in a nested way.
  • I have also noticed the issue that some of the content titles having italic tags with it. I'll check this issue and fix it as soon as possible.
1997kB added a subscriber: 1997kB.Dec 20 2018, 12:47 PM
Mahveotm added a subscriber: Mahveotm.
1997kB removed a subscriber: 1997kB.Jan 25 2019, 3:11 PM