Page MenuHomePhabricator

Project Weekly Reports for Automated Testing and Integration of IFTTT support to Wikidata
Closed, ResolvedPublic

Description

Week 1 (May 23rd - May 30th)

  • Integration of Travis-CI testing bot to my personal repository at the moment for development. This bot will be used for continuous testing and integration of the app. Once we (I and the mentors) decide on the general repository to use for development, I will just transfer the Travis settings to the repo.
  • Travis-CI is configured to do the following functions;
    • Post messages (build related messages) to the Wikidata feed channel on Freenode (#wikidata-feed) so as to notify others on how the development is going (just for monitoring purpose)
    • When the build fails or passes, an email is sent to me also for notification.
  • I and the mentors have decided a repository that I will be pushing my codes to and it's the official Wikimedia IFTTT repository that can be found here: Wikimedia IFTTT Github app.
  • Created a GSoC development branch that will be used to push my codes to on the official Wikimedia IFTTT repo. This will enable mentors to review, comment and merge my codes when its ready to the master branch.

Week 2 & Week 3 (30th May - June 12th)

  • Wrote unit test for Wikipedia Article of the Day trigger and ran a client test against the trigger and it was working correctly returning the correct result set.
  • Wrote unit test for Wikipedia New Article trigger and ran a client test against the trigger and it was working correctly returning the correct result set.
  • Wrote unit test for Wikipedia Word of the Day trigger and ran a client test against the trigger and it was working correctly returning the correct result set.
  • Wrote unit test for Wikipedia Picture of the Day trigger and ran a client test against the trigger and it was working correctly returning the correct result set.
  • At this juncture, 4 tests are working correctly (passing) out of the 6 non-database driven tests I have to write. But 2 tests are still failing. Working on these two test and also fixing some few bugs and doing code clean up.
  • Wrote a Wikidata trigger with name "Item Revision" for which when a particular item in Wikidata is revised, for example(Q12345), the trigger fires and performs the respective actions.
  • Developed an RSS View for Item Revisions trigger and an XML template for it for rendering of the data using the browser. The commit can be seen here on my Github account: The commit can be seen here on my Github account: RSS View Commit.
  • Fixed bug that was causing the non-DB to fail and now all tests, including the test for my new Item Revision trigger is passing. Also, updated the read me file to contain news triggers that are being developed.
  • Synchronized my online and local development Github branches with the official GSoC development branch in the Wikimedia IFTTT repo and re-committed the whole work I have done, this is because of conflicts that originated when I made a PR to the official Wikimedia IFTTT code base which was different (setup) from that of Stephen that I was using before. The working PR without any conflicts can be seen here: Pull Request

Week 4 (13th June - June 19th)

  • As instructed by the mentor, I started deploying me work on Wikimedia Tool Labs for integration testing with IFTTT. The mentor added me to a Wikipedia staging channel for me to do integration testing with IFTTT.
  • The Wikimedia Tool Labs instance I am using is "ifttt-testing" and can be accessed by using this link: IFTTT Test Instance (RSS Feeds Example) and it has been configured to run my application for testing within the server and also in the Wikipedia staging channel.
    • NOTE: Once in a while, we shall have a 404 - Not Found error while trying to access this link since its still unstable as I am working on the server on a daily basis, its a test instance.
  • Deployed the application on Tool Labs and Integrated to IFTTT Wikipedia Staging to test the functionalities and also the trigger I have developed.
    • Trigger was working properly when configured.
    • All RSS Views where functioning properly
    • All tests where passing on Tool Labs
  • Fixed Links in the feeds.html file to point to the correct lab instance and no longer "localhost" as before and fixed all the codes to adapt to the configurations of my Lab Instance.

Week 5 & 6 (19th June - July 8th)

  • Submission of Midterm evaluations by both mentors and students. Students evaluate mentors and organisations activities and mentors evaluate their students and both submit evaluations to Google.
  • Midterm results announced by Google
  • My midterm evaluations results: Passed
  • Started my semester exams
  • Wrote my semester exams for 2 weeks as on the time line.

Week 7 (9th July - July 15th)

  • Built the RSS view for Database Driven trigger (New Hashtags), Since I have access to the DB now on Tool labs, I can retrieve data concerning this trigger directly from the Database. Here is an example of the RSS: New Hashtag for test hashtag.
  • Unit test for New Hashtag passes on Tool labs including other tests because of the presence of the DB but on localhost, this test fails and other database driven test because of the absence of the Wikidata database on my localhost.
  • Fixed the Issue of RSS views working with GET request without checking channel key. The issue here was "if RSS views are working, the integration tests fails on IFTTT or if Integration tests are passing, the RSS views are not working due to the @app.before_request me that checks the channel key before any request". So I added a check to ignore all these on get request using if flask.request.method == 'GET': # do something. At this point, both RSS views and Integration Endpoint tests on IFTTT are working (but some regression occurs, needs to be looked at).
  • Integration of newly developed trigger "New edit to Wikidata Item" into IFTTT and integration testing. So far, the trigger is working fine and fires when a particular item is edited.
  • Added Style to the RSS views to improve on UI/UX of the RSS views of the App. For the bootstrap integration into the app, I used the flask.ext.bootstrap and specifically the Bootstrap Module.

Week 8 (16th July - July 22nd)

  • Debugged IFTTT integration tests together with Stephen LaPorte and all was passing. Integration of Wikidata Edit Item trigger to IFTTT.
  • Wrote code for GET request to handle parameters for triggers just as the POST request in the app. The commit can be found here: Fixing Regression Commit.
  • Re-wrote and Improved on the Unit tests for all the triggers found in the ifttt-testing.py file. This improved on the efficiency and getting default trigger values from the trigger.py file and feeds directly into the tests fields. Here are the commits Working Commit - Commit 1 and Efficient and Working commit - Commit 2
  • After debugging IFTTT integration / endpoint tests with Stephen, this caused a bug in the unit tests which where all not returning data because app.debug was set to True. I debugged this and constrained app.debug to be True only in the context of the FlaskClient tests instance (app.test_client()) so that in the global context, it will be False and the test in the IFTTT integration will all pass. The result of this made both the unit tests and endpoint tests on IFTTT to pass. Here is the commit: Commit ( Shocked :-) )
  • Added RSS/XML for two last database driven triggers. New category member and Category member revisions, these triggers where challenging to build the RSS since we had to search on Wikipedia categories that are recently edited. It took me some time but finally got it with the help of my mentor (Slaporte). The category we used is 'All articles lacking sources'.
  • This two triggers (as mentioned above) was failing the unit tests and had no RSS, but after these commits: Commit I and Commit II, the RSS was available and unit tests where passing (all unit tests in the app).
  • A sample of the output of the RSS of the triggers can be seen here: New category members, Category member revisions
  • Item revisions trigger integration tests on IFTTT was failing because of a bug (configuration) that doesn't match the code in the IFTTT app, but together with Stephen on our weekly meeting (21st July, 2016), we debugged the integration tests and the Wikipedia Staging channel passed. Now all tests on IFTTT passes and also all Unit Tests on Tool Labs also pass. Hurrayyy!!! :)

Week 9 (23rd July - July 29th)

  • Learning the Wikidata Query Services (WDQS) and SPARQL. Experimentation is done here: Wikidata Query Service (WDQS).
  • Created a Google doc as instructed by my mentors to draft a blog post for the project. The link to the Doc can be found here: Blog post draft.
  • Thinking and sampling trigger Ideas with the mentors to find the right one to implement for Wikidata using the WDQS and the flask app. So far, we have a list of these trigger ideas:
    • Notify me whose birth days is today.
    • Notify me when today is a public holiday (we can decide for particular countries) e.g. Notify me when today is a public holiday in the US
    • Notify me when my favorite author releases a new book.
  • Working on the blog post draft on Google docs and developed on the following sections;
    • What is Google Summer of Code?
    • Why did I apply for GSoC?
    • Why the Wikimedia Foundation?
    • How has the development of my project looked like?
    • Request for help from community members on my project
    • Appreciation and Thanks to my mentors and the Wikimedia Foundation
  • More work on the blog post and addition of content to the various sections
    • My recommendations for anyone to participate in GSoC/Wikimedia and Why?
    • What I did not anticipate that will happen during GSoC?

Week 10 (30th July - August 5th)

  • Studying of Wikidata Items (objects) and Wikidata properties to come up with trigger ideas.
  • More trigger Ideas sent to mentors for evaluation, below are the trigger ideas
    • Notify me about the number of new Wikidata items added today.
    • Notify me about the number of new Wikidata item properties added today
    • Notify me about how many people have birthdays today (could be a very huge number). But this one will be limited to the amount of data in Wikidata (definitely not the whole world).
  • Worked on the "Experience gained through this project during GSoC" section of my GSoC blog post document and submitted for review to @Lydia_Pintscher.
  • Developed the "How my experience has affected my Day-to-Day life" of my blog post and submitted to @Lydia_Pintscher for review.
  • Developed the "About the project and what I have done so far" section of my blog post and submitted for review.
  • Finished developing the blog post, completed all the remaining parts of the blog and submitted for review.
  • Blog post finally reviewed and finishing of the blog post. Blog post divided into 3 parts and prepared for publishing. Done with blog :)

Week 11 (06th August - 12th August)

  • Started building the Birthday trigger view, for the base case, at this level, I limited all the subjects just to 10 and the trigger returned birthdays of entities on Wikidata.
  • Birthday trigger build using the Wikidata Query Service and SPARQL queries integrated into the flask app.
  • Created RSS for the birthday trigger which can be found in the popular_persons_birthday.xml template and also added a Unit tests for birthday trigger which is passing on tool labs.
  • Debugging of IFTTT endpoint tests for birthday trigger.
  • Birthday trigger endpoint test now passing on IFTTT and also tested it by creating a sample personal recipe and got an action for birthdays of people in Wikidata.
  • Improvement on the Birthday trigger so that it can use the lang trigger field to get the birthdays. This trigger field will be set on IFTTT recipe configuration by the user creating the trigger.
  • Fixed bug which prevented RSS content for the triggers to be translatable, now the RSS views for all the triggers are translatable in various languages provided the localization is available.
  • Modified the birthday trigger to get information about recent birthdays and ordering it in descending order of date. An example instance of the query can be found here: Recent birthdays for computer scientists and the query is below:
SELECT ?entity ?itemLabel ?propertyLabel ?date (year(?date) as ?year) 
WHERE 
{   
 	?entityS wdt:P569 ?date .   
  	?entityS wdt:P106 wd:Q82594 . 
    values ?property { wd:P106 } . 
    values ?item { wd:Q82594 } . 
              SERVICE wikibase:label {
              	bd:serviceParam wikibase:language "en" . 
                        	?entityS rdfs:label ?entity . 
                			?item rdfs:label ?itemLabel . 
                			?property rdfs:label ?propertyLabel
              } 
            FILTER (datatype(?date) = xsd:dateTime && month(?date) = month(now()) && day(?date) <= day(now()))
} ORDER BY DESC(day(?date)) DESC(?year)
  • Added more trigger ingredients in the birthday trigger. Ingredients are now 3 instead of 1 which are; <lang>, <property> and <item>. These trigger ingredients optimizes the query to produce letter results and also gives room for the IFTTT users to customize the query to suite their needs.
  • Added options for property and item ingredients on the IFTTT front-end for users to select various options in creating recipes for birthday triggers.
  • Cleaned up the RSS view for birthday trigger to accommodate the new results that are returned like; age of the person, date of birth, properties and items etc....

Week 12 (13th August - 19th August)

  • Created an account on the Medium Website in which I will use to post my GSoC blog as recommended by the mentors.
  • Pencils down, Code clean up and documentation of various section of my codes. The various commits for the documentation are here: Docstring Fixes in ifttt-tests.py, Code Cleanup and Documentation and Code Cleanup.
  • Filling up the various occupations that IFTTT users will select as option in the IFTTT front-end in the birthday trigger configuration. Property used is occupation, so I am finding the most interesting occupations on Wikidata that will return results from Wikidata to fill up the IFTTT trigger occupation options.
  • Merged pull request to Wikimedia IFTTT GSoC branch here: Pull Request 15.
  • Combination of all my work in GSoC from the first day can be seen here (a list of all the commits): All commits
  • Migration of the IFTTT application on Wikimedia Tool Labs from Grid Engine to Kebernetes. This was the suggestion from Stephen (primary mentor) and it sounded good and more over why not use recent tool labs technology. So the application now runs on Kebernetes.

Event Timeline

xSavitar renamed this task from IFTTT GSoC Project Weekly Reports to Project Weekly Reports for Integration of IFTTT support to Wikidata.May 9 2016, 7:01 AM
xSavitar renamed this task from Project Weekly Reports for Integration of IFTTT support to Wikidata to Project Weekly Reports for Automated Testing and Integration of IFTTT support to Wikidata.May 15 2016, 10:49 AM
xSavitar updated the task description. (Show Details)
xSavitar updated the task description. (Show Details)

@D3r1ck01 Please update your status update to reflect recent updates.

xSavitar updated the task description. (Show Details)
xSavitar updated the task description. (Show Details)
xSavitar updated the task description. (Show Details)
xSavitar updated the task description. (Show Details)

Thank you for the weekly reports. Feel free to close this down, as the program just got over.