Page MenuHomePhabricator

GSoC19 Proposal : VideoCutTool to cut videos in Commons
Closed, ResolvedPublic

Tokens
"Orange Medal" token, awarded by Aklapper."Love" token, awarded by MichaelSchoenitzer_WMDE."Like" token, awarded by maskaravivek."Love" token, awarded by Prtksxna."Love" token, awarded by Zoranzoki21.
Assigned To
Authored By
Gopavasanth, Apr 5 2019

Description

Profile Information:

Name: Gopa Vasanth
Email: gopavasanth1999@gmail.com
IRC Nick: Gopa
MediaWiki Username: Gopavasanth
Gerrit: gopavasanth
Github: gopavasanth
Portfolio: Gopavasanth
Blog: gopavasanth.wordpress.com
Time zone: UTC +5:30 (IST - India)
Location: Kerala, India
Phabricator: T217503
Mentors: @Hassan.m.amin, @Rogueassasin123, @Doc_James, @maskaravivek.
Working Hours (in IST): 42+hrs per week

Synopsis:

VideoCutTool will help to trim videos on-the-fly in Wikimedia Commons. Currently, a video in commons cannot be edited online. They have to be downloaded, changes made and later re-uploaded. As this process tends to take a good amount of time, the VideoCutTool makes authorized users work hopefully up-to 10x faster. This tool will be deployed on Wikimedia Toolforge, a hosting environment that provides services for the Wikimedia movement.

Main Deliverables:

  • Simple user interface for the VideoCutTool.
  • Video trimming feature using ffmpeg.
  • Integrating the Express backend and React front-end with API’s.
  • Deployment of VideoCutTool on Wikimedia Toolforge.
  • MediaWiki OAuth to validate the users with Wikimedia Commons account.
  • Writing E2E and/or UI tests.
  • Documenting the VideoCutTool code base.

Additional Deliverables:

  • Video Cropping feature.
  • Integration of VideoCutTool with traslatewiki.net.

Detailed Approach:


Fig: 0.1: Flow chart of the VideoCutTool

A simple user interface for VideoCutTool:

  • The front-end will be made using the React Ant design and will be integrated with node.js as the backend by serving APIs.
  • This UI mock-up is made with an inspiration of online -video-cutter and CropTool.
import React, { Component } from 'react';
import { Input, Typography, Layout, InputNumber, Col, Row, Button, Anchor } from 'antd';
import { Player } from 'video-react';
class App extends Component {
  render() {
    return (
        <Layout className="layout">
          <Header>
            <Typography.Title > VideoCutTool</Typography.Title>
          </Header>
        </Layout>
    );
  }
}
export default App;

Sample Code Snippet: 0.1 Expected (Header) from React ant design Code for VideoCutTool front end.

<Content className='Content' >
            <Row gutter>
              <Col span>
                <div className="upolad">
                  <Player playsInline poster="#"
                  src="<Wikimedia Commons Video URL>"
                />
                </div>
              </Col>
              <div className='trim'>
              <Col span>
                <h2>Video Trim Settings</h2>
                <Row gutter>
                  <Col span>
                    <Typography.Text strong >From</Typography.Text>
                    <Input placeholder="00:00:00" />
                  </Col>
                  <Col span>
                    <Typography.Text strong >To</Typography.Text>
                    <Input placeholder="00:00:00" />
                  </Col>
                </Row>
                <Button type="primary">Trim</Button>
              </Col>
              </div>
            </Row>
</Content>

Sample Code Snippet: 0.2 Expected (Content) React ant design Code for VideoCutTool front-end.

<Footer >
            © 2018 <a href="#><span>Gopa Vasanth </span></a> |
           <a href="#"><span> Github </span></a>
</Footer>

Sample Code Snippet: 0.3 Expected (Footer) React ant design Code for VideoCutTool front end.


Fig: 0.1 VideoCutTool takes the URL of the Wikimedia Commons Video.


Fig: 0.2 VideoCutTool after fetching the Video.

Video Trimming using ffmpeg:

Videos are trimmed using ffmpeg by taking fromTime and toTime as user inputs. Here -ss represents start time and -t represent the end time of the video.

fromTime ← Input()
toTime ← Input()

ffmpeg -i in.webm -ss $fromTime -t $toTime -async 1 out.webm

Sample Code Snippet: 0.4 Video Trimming using ffmpeg

Video Cropping using ffmpeg:

Videos are cropped using ffmpeg by taking the out_width, out_height, x and y as the user inputs. or by enabling the user to drag the layer on video to crop and auto-generating these fields similar to CropTool for Images.

out_width is the required width of the video.
out_height is the required height of the video.
x and y are to specify the top left corner of the video.

ffmpeg -i in.webm -filter:v "crop=out_width:out_height:x:y" out.webm

Sample Code Snippet: 0.5 Video Cropping using ffmpeg

OAuth login with Wikimedia account:

  • Register/Login using OAuth with Wikimedia account.
  • Once logged in, check if the user is already registered in the database by querying the database using their wikimedia id which is returned from the OAuth login.
  • If the user is already existing user update the token or save the users information into a new record with username, wikimedia-id, token secret.
$scope.openFile = function(updateHistory) {
  if ( updateHistory === false) {
	$scope.currentURLParms =  {
		username: getParameterByName(username),
		userID: getParameterByName('userId'),
		tokenSecret:getParameterByName('tokenSecret'),
     };
  }
}

Sample Code Snippet: 0.6 Sample code if the users updateHistory is false.

  • Either storing the user data in the session or generate a JWT token and return to the user in JSON format as
{
             “Username”: “Gopavasanth”,
	     “Token”: “JwtToken” 
}
User.update({  mediawikiGlobalId: profile.id }, {
$set: { username: profile.username, token, tokenSecret } },
     { upsert: true }, function(err, user) {
    return done(err, user);
})

Sample Code Snippet: 0.7 The strategy requires a verify callback, which accepts these credentials and calls done providing a user, as well as options specifying a user name, tokenSecret.

app.get('/auth/mediawiki,
  passport.authenticate(mediawiki, { scope: MEDIAWIKI_AUTH_SCOPE }));
 
app.get('/auth/mediawiki/callback', 
  passport.authenticate(mediawiki, { failureRedirect: '/login' }),

function(req, res) {
    // Successful authentication, redirect home.
    res.redirect('/');
  });

Sample Code Snippet: 0.8 Using passport.authenticate(), specifying the 'mediawiki' strategy, to authenticate requests. For example, as route middleware in an Express application.

API Routes:

  • GET ‘/login’
    • Login via MediaWiki OAuth.
  • POST ‘/cut’

This enables the VideoCutTool to trim the videos using ffmpeg and the new video file title should be uploaded with the proper file description and the title is to be verified to avoid the duplication in Commons.

$scope.$watch('titleInput', function() {

    if (!$scope.titleInput) {
        return;
    }

    var params = parseVideoUrlOrTitle({title: $scope.titleInput}),
        key = params.site + ':' + params.title;

    if (params.title && $scope.exists[key] === undefined) {
        if ($scope.title !== params.title) {
            $scope.error = '';
            fileExists( params.site, params.title );
        }
    }
    $scope.currentUrlParams = params;
});

Sample Code Snippet: 0.9 Sample code to verify the file existence in Commons as per the user input.

Create a new record in the database with the video information and author information.
A new thread is generated to track the progress of the video from ffmpeg stdout and (piped through) WebSockets.

  • Inside the new thread
    • Downloading the video from commons at the server for video processing.
    • Start the video cutting operation using ffmpeg.
    • Once the video is trimmed and the new video is to be re-uploaded on Commons with the few parameters (Tittle, Description, Categories) and the logged in users token.
$scope.upload = function(isRetrying) {

var params= {
	Title: $scope.currentURLParams.title,
	Site: $scope.currentURLParams.site,
	Page: $scope.overwrite,
	Overwrite: $scope.overwrite,
	Comment: $scope.uploadComment,
	Filename: $scope.newTitle,
	Elems: $scope.cutresults.page.elems,
	Store: true
};

$http.post('./api/file/publish', params).then(function(res) {
	var response = res.data;

	if (response.result === 'Success') {
		$scope.uploadresults = response;
	}
}

Sample Code Snippet: 0.10 Sample code to upload the new video back to Commons as per the user input.

  • GET ‘/videos/:id’
    • This returns the video recorded by id to track the progress on the visual front-end using WebSockets.

Testing and Documentation:

  • Writing the Unit tests and integration tests (if possible E2E tests) to test the flow of application performance as per the designed way.
    • Documenting the VideoCutTool codebase and prepare materials for further development for the tool.

VideoCutTool hosting on Toolforge:

  • Wikimedia Toolforge supports node.js, MySQL in deploying tools. VideoCutTool is to be compatible with the Wikimedia Toolforge.

About Me:

I am Gopa Vasanth, a sophomore in Computer Science and Engineering from Amrita University, India. I am an active member of an Open Source student club at our University, FOSS@Amrita. I have been contributing to various projects in the Wikimedia sphere since 2017, starting with “Season of RevisionSlider” with Wikimedia DE. As part of the program, I have fixed a couple of bugs and implemented new features [1]. Also, I was a Google Code-in 2018 mentor with Wikimedia Foundation. I organized a Wikimedia Contribution drive at our club [2] in 2018 with around 30 participants. And delivered a talk on WikiMedia Foundation at our university [3].

I will be hopefully (visa application under process) attending the Wikimedia Hackathon 2019 with a full-scholarship at Prague as well.

I plan to publish blog posts during or after my GSoC. I also plan to regularly discuss with my mentors and update my work using weekly status updates. Post GSoC, I plan to be an active maintainer of this tool (as with any Open Source projects).

Micro Task: VideoCutTool Basic functionality

Previous Contributions to Wikimedia:

Gerrit: https://gerrit.wikimedia.org/r/#/q/gopavasanth
Github: https://github.com/gopavasanth
Micro-Task: https://github.com/gopavasanth/VideoCutTool/tree/Express_backend
[1] Highlight revisions from the same user: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/RevisionSlider/+/454526/
[2] Contribution drive: https://www.mediawiki.org/wiki/Wikimedia_Contribution_Drive_amFOSS
[3] Talk: https://twitter.com/akhilam512/status/1106836610321018880?s=19

Project Timeline:

WeeksTimeLineMilestones
Community Bonding
1May 6th to May 27thInstalling and configuring required software in my system.
2May 16th - May 25thWikimedia Hackathon 2019 at Prague and discuss in depth of the project details with mentors.
3May 20th - May 27thFamiliarising with the CropTool codebase.
Coding Period Starts
4May 28th -June 8tha) Designing and Implement the front end UI of VideoCutTool. b) Testing and building the UI of VideoCutTool with React Ant Design.
5June 8th - June 15tha) Validating the video from Commons. b) Checking for supported video formats (webm), c) Implementing error handling techniques. d) Fetching the video to display in the front end.
6June 15th - June 27thTrimming the video using ffmpeg from the user input fields.
Phase-1 Evaluation
7June 28th - July 10thTracking the video trimming progress from ffmpeg by using the information from stdout.
8July 10th - July 17tha) Verification to avoid the repeated tittle of the new video isn’t duplicated on Commons. b) Updating the database with the Title, Description, Categories, and Author.
9July 18th - July 25thTracking the derivatives for the cropped videos with the original video file.
10July 25th - July 26thTesting and Bug fixing.
Phase-2 Evaluation
11July 27th - Aug 10tha) Re-Uploading the video back to the Commons. b) Testing and bug fixing.
12Aug 10th - Aug 22ndVideo cropping feature for additional benefits.
13Aug 22nd - Aug 26tha) Writing Unit tests/integrated tests and Bug fixing. b) Documenting the VideoCutTool codebase.
Phase-3/Final Evaluation

References:

Event Timeline

Restricted Application added subscribers: Danmichaelo, Aklapper. · View Herald TranscriptApr 5 2019, 6:51 PM
Gopavasanth updated the task description. (Show Details)Apr 8 2019, 6:31 PM

@Gopavasanth Looks great! the only thing that might need change is week 9, i don't believe that the milestone assigned to this week would require that time to accomplish, so that milestone can be moved to the preceding week. we need to be able to track derivatives for the cropped videos with the original video file. so it would be great to put that in week 9 instead :)

Gopavasanth updated the task description. (Show Details)Apr 9 2019, 10:58 AM

@Hassan.m.amin Thanks for your feedback, I have modified accordingly.

Thanks, @Hassan.m.amin, @Rogueassasin123, @Doc_James, @maskaravivek, and @srishakatux for selecting my project into GSoC'19, I wish to complete this project successfully in time.

TheDJ added a subscriber: TheDJ.Jul 4 2019, 3:21 PM

Just wondering if there has been progress here since the hackathon.. ?
We got kinda bogged down back then in tool forge

@TheDJ : yes, I am sure we will have a Beta roled out by Aug 7th. The Beta will allow the users to:

(1) Rotate the video
(2) Crop the Video
(3) Trim the Video

Hello @TheDJ

I'm so happy as you reached us back, Yes there is a lot's of progress since the Hackathon, Updates of this work can be found here: https://www.mediawiki.org/wiki/Google_Summer_of_Code/2019/Bi-weekly_Reports.

As we discussed this in the hackathon, I'm facing the issue to host the VideoCutTool front-end (React app) in the toolforge. So the front-end is currently hosted in https://video-cut-tool.netlify.com/video-cut-tool/ (only front-end).

I found https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#node.js_web_services to host the back-end of this app and hosted here https://tools.wmflabs.org/video-cut-tool-back-end/, I couldn't find any documentation to host the react-front-end of this tool in toolforge, I have tried several times but :(

@Gopavasanth You can try posting about the issue on discourse if not already done. Someone might be able to help. :)

https://discourse-mediawiki.wmflabs.org

@Gopavasanth: regarding OAuth, here's some debugging by someone with I think a similar issue, maybe it helps: https://github.com/milimetric/passport-mediawiki-oauth/issues/2#issuecomment-513711239

@Gopavasanth: regarding OAuth, here's some debugging by someone with I think a similar issue, maybe it helps: https://github.com/milimetric/passport-mediawiki-oauth/issues/2#issuecomment-513711239

Hello @Milimetric, Thanks for reaching back to me, I figured out the problem and it got resolved :)

The problem was I gave the wrong callback, I reconfigured it and now it works fine :)

Congratulations on completing the project! If there isn't anything remaining in your proposal to address, feel free to close this task. Before you do so, make sure your project is listed here https://www.mediawiki.org/wiki/Google_Summer_of_Code/Past_projects#2019 and has the following information: Student name, Mentors, Relevant links and Outcomes (in not more than two lines).

Gopavasanth closed this task as Resolved.Sep 4 2019, 7:41 AM

Thank you @srishakatux

Tool is not live on Commons yet... Should we wait until that is the case before closing this?

Gopavasanth reopened this task as Open.Sep 4 2019, 7:55 AM

You can close this task as GSoC is officially over and create a separate task on making the tool available on Commons.

Gopavasanth closed this task as Resolved.Sep 16 2019, 4:11 PM

VideoCutTool is live here: https://tools.wmflabs.org/video-cut-tool-front-end/ and the project workboard can be found here