Page MenuHomePhabricator

Weekly reports of GSoC 2017 project "glam2commons"
Closed, ResolvedPublic

Description

Weekly reports of the project will be available here:

5 May - 12 May:

  • Friday: Talking with all coaches, we will be using the #wikimedia-sibu channel on IRC for communication. Every sunday at 13:00 UTC we will have a meeting with everybody, and in between there will also be communication.
  • Sunday: Discussion about the name. Maybe we should go for glam2commons.
  • Monday-Friday:
    • Created a new tool account in Tool Labs
    • Set up the environment for a new Flask OAuth tool.
    • Made a test app called 'sibutest'.

13 May - 19 May:

  • Written GSoC weekly blog post.
  • Added OAuth authentication using wikimedia credentials to the test app sibutest.

20 May - 26 May:

  • Completed the metadata mapping for a single image to be upload to commons from the Nationaal Archief glam. Please find here.
  • The sibutest app is ready except that the Mediawiki OAuth is failing for some reason. Please find the issue in the comment.

27 May - 02 June:

  • Read some documentation of the Europeana API from here.
  • Successfully configured the pywikibot for uploading images in the test app sibutest. The upload is not taking place for some APIError as written in the comment.

03 June - 09 June:

  • Designed and implemented a basic UI for the app. Please find here.

10 June - 16 June:

  • Modified code so that the category field given by the user is correctly included in the commons metadata now.
  • Added a button for adding multiple categories for an image.
  • The "Home" button behavior has been properly modified. Now it shows the login button if not logged in else it shows the glam form to be filled for uploading an image.
  • Replaced the textbox for entering the glam name with a dropdown.

17 June - 23 June:

  • Written generic library functions for the following tasks: (Please find here)
    • Function to load the json from given url and get the parsed json.
    • The function that builds the wiki metadata from the glam metadata of the image.
    • Function to genrate a standard title for the image to be uploaded.
    • Function to upload the image file.
  • Created the help page for unique ID information of an image.
  • Written code to display the Commons URL once an image is successfully uploaded to Wikimedia Commons.

24 June - 30 June:

  • First evaluation has been completed.
  • Discussed with my mentors regarding the possible approaches to make the app scalable so that support for new GLAMs can be easily added to the app.

01 July - 07 July:

  • Written code for easily adding new GLAMs to the test app by following an OOP approach. Please find the code here.

08 July - 14 July:

  • Discussed extensively with my co-mentor Zhuyifei regarding the possible implications of our OOP approach.
  • Studied the code of the Flinfo app for uploading images.
  • Changed our OOP design suitably so that each image upload is an instance of the base GLAM class.
  • Added infobox templates Photograph and Art Photo.
  • Click here for detailed report.

15 July - 21 July:

  • Integrated the new code into Toollabs.
  • Properly organized the github code into two branches with the help of Zhuyifei.
  • Changed the implementation of the uploading module from UploadRobot to site.upload. But the uploads aren't taking place. I am investigating the cause.
  • Discussed at large about providing support for multiple uploads with my mentors. Judged the pros and cons of the possible approaches.

22 July - 28 July:

  • Enabled the flow for Nationaal Archief GLAM for uploading images via search string.
  • Made the UI for the same.
  • Created a page in Wiki Commons to describe how a new GLAM can be added to the tool.

29 July - 04 Aug:

  • Locally integrated Amsterdam Museum into the tool. Some more work is remaining such as showing the images as same size thumbnails in a gallery in a non-overlapping manner.
  • Rewritten some code to accommodate the changes corresponding to the addition of Amsterdam Museum GLAM.

05 Aug - 11 Aug:

  • Thumbnails are made equal sized in the image gallery
  • Rewritten Nationaal Archief code in the lines of Amsterdam Museum code
  • Made the overall code generic with scalability in mind
  • Changed the texts in the interface as suggested

12 Aug - 18 Aug:

  • Incorporated the final review comments into the tool. These include
    • Importing GLAM classes dynamically.
    • Loading the dropdown of GLAMs dynamically.
    • Showing proper error messages on the UI interface when exceptions occur.
    • Handling of duplicate uploads properly.
    • Putting pydocs and external documentation in place.
    • Modifying UI text and color etc.
    • Designing and incorporating a new logo for the tool.

Event Timeline

Infobliss renamed this task from Weekly reports of GSoC 2017 project "Single Image Batch Upload" to Weekly reports of GSoC 2017 project "glam2commons".May 23 2017, 10:14 PM

Traceback (most recent call last):
File "/data/project/sibutest/www/python/src/app.py", line 75, in oauth_callback
identity = mwoauth.identify(app.config['OAUTH_MWURI'], consumer_token, access_token)
File "/data/project/sibutest/www/python/venv/lib/python3.4/site-packages/mwoauth/functions.py", line 255, in identify
"content: {0}".format(e))
mwoauth.errors.OAuthException: An error occurred while trying to read json content: Expecting value: line 1 column 1 (char 0)

from https://github.com/mediawiki-utilities/python-mwoauth/blob/master/mwoauth/functions.py#L255
and https://github.com/infobliss/sibutest/blob/master/app.py#L75
Is the json response broken?

Is the json response broken?

Possibly. Maybe you can check what r.content actually is?

About the https://github.com/infobliss/sibutest/blob/master/NationaalArchief2.py file: nice work, some comments:

Line 197, this should be indented one less, now if the title is shorter than 85 characters the file does not get uploaded?

line 155-165, what happens if it is none of these licenses? the upload should abort/return an error as it has no license viable for commons.

Do the files end up in https://commons.wikimedia.org/wiki/Category:Images_from_Nationaal_Archief?

I know the code is based on the code I wrote, however that code is not really clean and rather hacky. The long function load_from_url has several parts which could be split. I believe it is cleaner to first obtain all variables and then use templating (.format() or maybe something better) to fill a standard body of text.

What happens when there is no 10000x10000 image available? Would it also be an option to take the largest file?

@Basvb thanks for reviewing the code.

Line 197, this should be indented one less, now if the title is shorter than 85 characters the file does not get uploaded?

My bad. I will rectify it now.

line 155-165, what happens if it is none of these licenses? the upload should abort/return an error as it has no license viable for commons.

Will add an else block to handle this case.

Do the files end up in https://commons.wikimedia.org/wiki/Category:Images_from_Nationaal_Archief?

(Don't know now. Will test and report here. )
Update : Yes they do. Please check here.

I know the code is based on the code I wrote, however that code is not really clean and rather hacky. The long function  load_from_url has several parts which could be split. I believe it is cleaner to first obtain all variables and then use templating (.format() or maybe something better) to fill a standard body of text.

This can be done. I will refactor the code as suggested.

What happens when there is no 10000x10000 image available? Would it also be an option to take the largest file?

Ok then I will add a check for the availability of 10000x10000 thumb, if not found then the code will choose the largest available thumb.

After I got a new consumer registered with 'upload files' grant and used those consumer details in the config file, I got the following error from pywikibot.

pywikibot.data.api.APIError: permissiondenied: The action you have requested is limited to users in one of the groups: *, [[Commons:Users|Users]]. [help:See https://commons.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes.]

Hi Infobliss, the UI looks like a good start. Am I correct that there is nothing behind the pages yet? Or do I have to login using OAuth first?

The app still is not approved. Where can I find the request for approval?

Hi @Basvb, only the "Home" page has been created as of now and the glam form that takes the image details once you login.
The new consumer for the app is yet to be approved. I am not sure where the request for approval can be found. As suggested by zhuyifei, I will leave a message at Steward requests/Miscellaneous so that the request is reviewed.

Infobliss updated the task description. (Show Details)

Hi! If there is not anything remaining in this task could you please resolve it?