Page MenuHomePhabricator

GSOC 2024 Proposal for Lingua Libre SignIt extension (Migration towards Manifest V3)
Closed, ResolvedPublic

Description

Profile Information

  • Name: Kabir Singh
  • Github : kabir-afk
  • Phabricator:GonFreeaks
  • Location: Delhi, India
  • GSoC full time : Part time (avg 20h over 8 weeks)
    • Workload: 175h
  • Timezone: UTC+05:30

Synopsis

The "SignIt extension" project allows you to translate a word into French sign language on any web page. When you read a text and come across a word you don't know, highlight that word, right click and click on the Sign it icon: the sign in LSF and the definition of the word in French will appear on a window. The aim of this project includes migrating from manifest V2 to V3 and then later exploring cross browser format.

How it Benefits Wikimedia Projects:

SignIt is Wikimedia project to increase visibility and usability of our signed languages lexical resources (video). As Manifest v.2 is being phased out by Chrome Webstore, the project is under critical threat.

Manifest V3 extensions recoding will secure and expand Wikimedia's SignIt presence on Chrome and Firefox's web stores. It will also, de facto, create a « click-to-translate » web extension codebase, reusable by other Wikimedian or non-wikimedian open source enthusiasts.

Mentor
Have you contacted your mentors already?

Yes, I have initiated communication with my mentor,@Yug and I have successfully completed and fixed some microtasks in collaboration with the mentor.

Deliverables / Timeline

Main
  • July 31st : Working, fully recoded manifest v3.0 web extension modular, easy to maintain code for
    • Firefox
    • Chrome
  • August 31st : Web extension available extensions stores :
    • Firefox extensions store
    • Chrome extensions store
June 23, 2024

Coding Phase begins !!!

Week 1 June 23 -Jun 28, 2024

Migrating to service worker as per me is the biggest challenge of this project which needs to be overcome. Hence the long duration. Since there are many content scripts to deal with , moving their DOM and window calls to offscreenAPI is going to be a big task. Popup.js might not show many such calls , but other files like signit.js and wpintegration.js which seem to be the integrating files have many calls to deal with, including the ones that are responsible for triggering other components .Not only direct window and DOM calls but even the ones which manipulate DOM indirectly , like using jQuery post requests could also be moved to such offscreenAPI without having to change the request much.

The project has already used browser.storage.local api to much extent and utilizes its full potential , but as per V3 , we can no longer use the state variables like we used to since service workers terminate and re run multiple times during a single session, which might lead to not loading of these variables since the previous context was torn down. To get around this we’ll have to persist these state variables as well inside the storage API. Benefit of using it is that it’ll be available to all the content scripts and hence unnecessary message passing won't be needed. Those set states can then be easily accessed using chrome.storage.local.onChanged method, which is convenient.

Keeping the service worker alive is another thing to be taken care of during this period. Other things are there as well when migrating to service worker but they cam be taken care in a day or two - like converting timers to alarms.

Week 2 July 1 -July 5, 2024

Updating API calls doesn't require much time and one week should be more than enough.The only thing that has to be taken care of is replacing callbacks with promises. In the contributions I have mentioned down below , I have already updated tabs.executeScript() and tabs.insertCSS() with scripting.executeScript() and scripting.insertCSS(). Besides that, unsupported APIs have to be replaced.

Week 3 July 8 - July 13, 2024

Replace blocking web request listeners
In order to handle specific network requests , earlier we used to , programmatically tinker with them , but now is not the case, we can lay a rule sheet and make good use of it by giving declarativeNetRequest permission in the manifest.I haven’t explored it much but based on what I have read , it should enhance the security of the extension as well make for a long lasting future proof extension.

Week 4 July 15 - July 20, 2024

Improving extension security involves:-

  • Removing execution of arbitrary strings.
  • Removing remotely hosted code
  • Updating content security policy.
  • Removing unsupported content security policy values

Out of the 4 above , 2 have been partially addressed in the contribution mentioned below , issue id 55. The CSP has been updated as per manifest V3 and unsupported wikimedia-commons CSP has been removed. That only leaves us removing execution of arbitrary strings and removing remotely hosted code.

Mid-Term Evaluation

The timeline is made keeping in mind the Chrome MV2 deprecation stable rollout which is not clear as of now but as per resources it’ll take place in June 2024 + 1-X months , so safe to assume , at the very least , till mid-july. By midterm the extension will be having:-

  • Full functionality with Manifest V3 , along with some isolated functionalities that rely on Manifest V2.
  • Simplifying Localization such that i18n functions are exposed and accessible to all the content scripts.
  • Retention of language settings throughout the user experience
  • Improved extension security.

In the midterm evaluation phase, I will be addressing the reviews of the mentor and will be working to improve on them. By this time I’ll make sure that Beta version of the extension is readied and works to some degree both on chrome as well as firefox.

Week 5 July 25 - July 27
  • In this period I wish to focus on cross browser compatibility by changing API namespaces and the way asynchronous events are handled have to be dealt with.
  • With the introduction of Manifest V3, all the main browsers adopted the standard of returning Promises from asynchronous methods. Firefox and Safari have full support for Promises on all asynchronous APIs. Starting from Chrome 121, all asynchronous extension APIs support promises unless documented otherwise.

To take advantage of promises , WebExtension browser API Polyfill has to be reinstated and worked around . . . says mdn web docs_

  • Resolving background scripts vs service worker by implementing browser_speciifc_settings.
Week 6 Final Submissions and Results (July 30 - August 3)
  • Submit all final code contributions, documentation updates, and testing reports.
  • Await the official Google Summer of Code results and acknowledge them promptly.
Week 7 Extended Coding Period (August 6 - August 11)

Continue contributing to SignIt if unable to meet the tasks in the scheduled timeline.

Final Evaluation

By the end of the GSOC – 2024 program (final evaluation), apart from fully functioning recoded manifest V3 extension , it shall also be having cross browser compatibility ready to be published on both chrome and firefox web store.

Final Submissions and Results (August 31)

Finalize any remaining contributions .

Participation

Communication Plan:

I am active on Phabricator ,email and discord. I plan on regularly updating my progress to the mentors, and getting feedback for the same.

Seeking Help:

Whenever stuck, I will notify on community channels or discord groups in order to seek help. Additionally, I will actively participate in the weekly meetings with mentors to discuss progress and address any issues as well as try to provide daily updates on code.
This plan outlines goals, benefits, mentorship, and a detailed timeline, showcasing a strategic and organized approach for Wikimedia Lingua Libre's SignIt extension during Google Summer of Code.

About Me

Education

I am currently a 2nd year student pursuing Bachelors in Computer Applications from Amity University,Noida.

How did you hear about this program?

I started writing code during my initial years of college. I have built various mini projects, and have collaborated on others as well.I participated in Hacktoberfest 2023 which further taught me how to contribute to open source . I came to know about GSOC from my friends and some online research .

Will you have any other time commitments, such as school work, another job, planned vacation, etc, during the duration of the program?

No, I do not have any other time commitments, such as school work, another job, planned vacation, etc., during the duration of the program and shall be contributing more to the organization.

We advise all candidates eligible for Google Summer of Code and Outreachy to apply for both programs. Are you planning to apply to both programs and, if so, with what organization(s)?

No, I plan on participating only in Google Summer of Code 2024, with Wikimedia Foundation.

What does making this project happen mean to you?

Open sourcing has always fascinated and excited me sherely because of the potential it has . The learning opportunities that come with this project are even bigger and better. I get hands-on experience of what it's like to make an extension work as well as migrate it to a higher version.

Past Experience

I have been contributing since January, and have submitted patches to Lingua Libre SignIt. This helped me to get familiar with the codebase. Here are some of my contributions to the extension.

IdTitleStatusNotes
#49Onboard @kabir-afkMergedChanged Manifest file from V2 to V3
#55Manual installation error on Chrome and EdgeMergedRemoved insecure CSP and replaced it with V3 comaptible CSP along with sandbox key
#63Migration to service worker initiatedIn-ReviewRemoved polyfills
Introduced type checking for cross browser compatibility
Successfully imported banana module inside service_worker as well as restored its functionality in popup script.
Updated browser APIs.
Partially restored popup and modal functionality wrt V3.
Replaced jQuery post requests with fetch API.
declared Icontext variable in popup.js
Describe any relevant projects that you've worked on previously and what knowledge you gained from working on them.

Ip-Address-Tracker: This project honed my skills in using React’s useEffect hook for side effects management and integrating Leaflet for interactive IP geolocation visualization. It sparked exploration of best practices for user experience with maps.

Coding skills
  • I’m skilled in React.js and well acquanited with other full-stack technologies like node and express.js.
  • In databases I am well versed with both MySQL as well as MongoDB.
  • I utilize Postman as a core tool within my API testing workflow.
  • I’ve had some experience with C/C++ as well while I was learning DSA.
  • I also have little experience with Blender and 3js for a personal project I was working on.

Any Other Info

Hacktoberfest 2023: I was among the first 50,000 participants to have their first 4 PR merged and had the privilege to plant a tree in my name. Additionally, I received a coveted Swag kit as recognition for my contributions and dedication to open-source projects.Here are my badges and Tree-Certificate
Leetcode 2023: I had 500 submissions in the past 1 year and last year I was also awarded with 50 days Badge 2023.
CSSBattles : I have global rank 605 in cssbattles with over a 100 targets played and a total score of 70193.53, click here to see.
Frontend Mentor : I am frequently active on the said platform with a mentor score of 1000, see here.

Event Timeline

Received. I'm reviewing your proposal.

Yug updated the task description. (Show Details)
Yug updated the task description. (Show Details)
Yug updated the task description. (Show Details)
Yug updated the task description. (Show Details)

Weekly report

1. Overview of Tasks Completed:

  • Task 1: Passing message to execute functions, changing icons and removing unnecessary headings
  • Task 2: Refactor SignItCoreContent for browser compatibility & modal init fix
  • Task 3: Introduce internationalization inside SignItVideosGallery.js
  • Task 4: Updating _locales messages as per browser.i18n API

2. Key Accomplishments:

  • Accomplishment 1: I guess executing functions by passing messages can count as accomplishment , as this was pretty challenging. Earlier even when the

codebase worked in firefox it used _backgroundPage API as namespace and fetched the contents of background page like object key-value pairs. But since we cant do so in chrome , I figured passing messages from popup/content scripts to excute functions in service worker would just work fine. I have expanded on this under te challenges faced.

  • Accomplishment 2: Internationalizing SignItVideosGallery.js isn't that big of accomplishment, but counts as development. Importing banana module inside other files head been a challenge for quite a while. While I was not able import it either but I made a function which mimicks the behaviour of what banana.i18n did, thereby restoring the functionality inside the above mentioned file. Now when the user changes UI language , the language inside videos gallery changes as well.

3. Challenges Faced:

  • Challenge : Extension to the above mentioned accomplishment., well the reason I count this as challenging as there were other methods as well. Like:-
    • Using a shared common-functions file - Technically it is the ideal approach , but it wasn't working as expected. It threw errors regarding the functions not being found.
    • Storing functions inside local storage: While we cant do that , it is technically possible by making it a string (like an actual string and not using JSON.stringify())and storing its arguments inside an array , then rebuilding it by parsing it and passing the items inside the functions. Seems a bit bizarre and makes for unnecessary code but if it were to happen , functions would be available wherever needed as they are in local storage.
  • Solution: I ultimately ended up with using messages to execute functions seperately inside service_worker/backgrounf-script.

4. Learnings and Skills Gained:

  • Learning : Got to understand more about how service workers work behind the scenes, how we can use local storage to make things more accessible betwen files when working with extension development and a little about internationalization.

5. Feedback and Support Needed:

Feedback/Support : I appreciate the feedback I've been getting on my approach! To ensure a smooth merge, I was hoping we could have a closer look at the code changes themselves. A more in-depth code review would be appreciated.

6. Goals for Next Week:

Goal 1: Restore settings tab fully functional.
Goal 2: Work on hindi i18n messages
Goal 3: Work on how Popup UI's language changes differently as compared to modal's language and video gallery's language change. It has something to do with local storage , I'll look into it.
Goal 4: Refactor toggleColoredWords function such that it works in Chrome as well.
Goal 5: Convert timers like setTimeout to alarms as part of Migrating to service worker checklist.

debt renamed this task from GSOC 2024 Proposal for Lingua Libre SignIt extension to GSOC 2024 Proposal for Lingua Libre SignIt extension (Migration towards Manifest V3).Jun 4 2024, 5:22 PM

Weekly Report (3 June - 9 June)

1. Overview of Tasks Completed:

While these were technically not the goals I mentioned earlier , they still count as development.

2. Key Accomplishments:

  • Accomplishment 1: Making the video files and records' POST request work in extension environment can be considered as an achivement. Since many other components relied on it to work.
  • Accomplishment 2: Recognizing the incompatibilty of .ogv file in Chrome environment. Turns out that due to security issues this format was going to be removed and many people had already transitioned to .webm format. Issue to migrate to a different format has been raised. See here

3. Challenges Faced:

  • Challenge : Overcoming the API endpoint issue can be considered a challenge. Earlier we used jQuery to send post requests but since we can no longer do that in chrome extension environment , using native fetch API remained the only way. The problem is it responded with HTTP Status code 405 : Method Not Allowed when sent with fetch API. This had been a persistent issue for quite a while now.
  • Solution: Instead of sending payload data as part of req.body, I tried sending it (necessary queries and data format) in the form of query parameters and it worked! I changed the URL accordingly. I'm aware that the code still looks like a get request but I will work on refactoring that in the next week.

4. Learnings and Skills Gained:

  • Learning : Learnt that endpoints need to be thoroughly tested in every way possible.

5. Goals for Next Week:

I would continue on my previous week's goals as I delved into some other issues this week as mentioned earier which were equally produtive if not better.

Weekly Report (10 June - 16 June)

1. Overview of Tasks Completed:

  • Task : fix : i18n. Used message passing to perform i18n tasks directly inside sw/background-script. While this made the function async in nature it also fixed a lot of issues like instant refresh of both Popup and Modal UI upon changing to desired language and also removed the use of unnecessary multiple i18n systems , be it self-made(in Chromium) or the use of old banana.i18n library(in Firefox). Now we solely rely on the old libraray for all the browsers ,so we can say that it is a centralized system. As compared to what we did earlier it is less verbose as well. Also helped close the issue #84

2. Key Accomplishments:

  • Accomplishment : Harmonized i18n system such that it renders as expected in popup, modal->core content and modal ->video galllery when UI language is changed. Earlier this wasn't the case as the constructor function reponsible for creating the SignItCoreContent (modal->core content) object wasn't being initialized properly. This led to delay in updation of both popup as well as modal when changing UI language.

3. Challenges Faced:

  • Challenge : As mentioned above that, passing message for i18n made it an async function which returned promise.So everytime banana.i18n was called , we had to await the promise in order to get its fulfilled result. Now this was easy to deal with in prototype functions as they could be made asynchromous. Main issue was updating the strings inside SignItCoreContent constructor function , as this was can't be made async.
  • Solution: In order to load the new language's i18n messages inside the constructor, I made a prototype function, declared it async and then awaited the necessary messsages inside it. After the meessages were resolved , appending the results inside the constructor by accesing the individual elements.

4. Learnings and Skills Gained:

  • Learning : got a better understanding of OOPS and Promise constructor function.

5. Goals for Next Week:

I would continue on my previous week's goals as I delved into some other issues this week as mentioned earier which were equally produtive if not better.

Weekly Report (17 June - 23 June)

1. Overview of Tasks Completed:

  • Task 1: feat : adding alarms API. The purpose of this PR was to replace timers with alarms API, timers here meaning functions like setTimeout and setInterval. Timers are for more persistent background pages which were used in MV2 but in MV3 , since service_worker is not persistent and it's lifecycle ends and restarts over and over again , alarms API was introduced. This task was the last among the checklist of migrating to service_worker , which has now been achieved after a long due.
  • Task 2: Refactor/callbacks to promises. This was another task under API updation. The extension API methods that used callbacks inside our background script had to be replaced with promises. Fortunately subconsciously we were relying on promises throughout our revamp so not much had to be done.

2. Key Accomplishments:

  • Accomplishment : Figuring out alarms API and its workaorund. Despite being a single function , figuring it out as per extension's requirement helped a lot.

3. Challenges Faced:

  • Challenge : While alarms are cross-browser compatible, achieveing the same in FF turned out to be a tedious task.
  • Solution: I kept the old approach as well in order to alteast keep things working. While it makes for an insufficient code , can still be refactored later.

4. Learnings and Skills Gained:

  • Learning : Alarms API

5. Goals for Next Week:

  • Workinng on local storage persistence issues
  • Replacing blocking web request listeners with declarativeNetRequest API

Weekly Report(24 June - 30 June)

1. Overview of Tasks Completed:

  • Task : No task was completed this week. Although it is WIP.

2. Challenges Faced:

  • Challenge : As mentioned earlier , I chose to replace blocking web request listeners with declarativeNetRequest API. While thought to be working properly , it bugged the youtube site into not playing anything, which ofcourse shouldn't be the case when moving to production.
  • Solution: Although I haven't come up with any solution yet, I made some interesting discoveries on why we were modifying the CSP headers and how does declarativeNetRequest API work which I have mentioned down below.

3. Learnings and Skills Gained:

  • Learning : The reason we were modifying CSP headers, was , so that if some site allowed video requests from a particular domain like youtube.com,which can further be due to a strict CSP , could also allow videos from wikimedia.commoms in order for our sign video to play. Thanks to Antoinne for shedding the light on the issue who was one of the founding coders of the project.

The declarativeNetRequest API provides a syntactic advantage over what were doing earlier by specifying the rules in a seperate json file and by making some changes in the manifest. It is also more performant and provides extra privacy than webRequestBlocking permission. So good for end users. Although it is currently not possible for us to append headers like we could earlier under declarativeNetRequest API, due to it not being much flexible. Thanks to this conversation right here.

4. Goals for Next Week:

Find and explore workarounds that help us modify CSP headers to allow sites to not restrct videos from our(wikimedia site) domain.

Weekly report (1 July - 7 July)

1. Overview of Tasks Completed:

  • Task : web request listener issue resolved

2. Key Accomplishments:

  • Accomplishment : resolving the issue without having to deal with headers itself.

3. Challenges Faced:

  • Challenge : The reason we were modifying headers was so that our extension's sign language videos could work on sites that have strict CSP headers, like Github or X. As mentioned in previous report I chose to replace old API methods with the newer ones using declarativeNetRequest API. But since it is not yet fully functional , doesn't allow us to append headers , doesn't support regex , I dropped it off and chose to explore other alternatives.
  • Solution: I came across iframe tag and its ability to not violate sites' CSP headers since videos were being embedded inside it. While it solved our current problem , it came with a new set of problems that I wish to resolve in the coming week.

4. Learnings and Skills Gained:

  • Learning : iframe tag and how we can embed whole documents / html pages inside our DOM.

6. Goals for Next Week:

Goal : Using iframe is certainly fruitious but doing so we cant manipulate the video like we were doing so earlier. The twospeed feature when enabled played our video first at normal speed then a little bit slower (0.75x). But since we can't access the video like we used to , it quite a problem.

Final Report

1. Overview of Tasks Completed:

2. Key Accomplishments:

  • Accomplishment : I won't say that this is something I accomplished on my own. Infact it was m mentor who came up with the idea of making twospeed work in iframe. Although it indeed is an accomplishment that we get to keep another feature of our extension while making it work on all the sites.

3. Challenges Faced:

  • Challenge : Accessing the content document of iframe was the biggest challenge , since the embedded document didn't belong to the parent document iframe was embedded in, it was causing difficulty in accessing the inner elements of the embedded document.
  • Solution: That's the twist. My mentor deployed a github page which itself had an iframe to play the video and based on the query parameters present in the URL of the page , we figured what video to play and whether to enable twospeed or not. To enable twospeed we did some pre-processing on the deployed page itself. All that was left to be done was change the <video src> in the extension video component.

4. Learnings and Skills Gained:

  • Learning : Workaround that my mentor came up with was quite a learning

All that is left is to publish the extension which is not really in my hands but with this my project has officially ended It was a pleasure to work for LinguaLibre and Wikimedia 🎉🎉