Page MenuHomePhabricator

Develop Dashboard with Flask For Database backup inventory improvements
Closed, DeclinedPublic

Description

Profile

Name: Dharmraj Rathod
IRC nickname on Freenode: DharmrajRathod98
Web Profile: https://dmrathod.github.io/
GitHub: https://github.com/DMRathod
Location (country or state): Gujarat, India
working hours: 5:30PM to 10:00PM (Time Zone: IST(UTC +5:30))

Synopsis

Short summary describing your project and how it will benefit Wikimedia projects

Database backup inventory improvements:
Rightnow, structured log of backup output is provide by database backups. Main objective of this project is to build a dashboard for list of metadata like name, host, type, size, status(ongoing, finished, failed, deleted),some filters with different attribute like list of all failed backups & its erros and status for ongoing backups objects like how much it will take to finish(Expected ETA), which backups will run after finishing the current backup(Next Scheduled run) so that we can track the status of ongoing backups and show them on dashboard.
Another task is to fill information like backup_id, size, databases linked in backup_objects(table). which is not filled at this moment so, we can make sure that no objects of backups are left uncopied.
This will require reading from existng tables backups, backup_files through APIs show them on dashboard and fill in the backup_objects(table).
This project will be developed from the base as no existing code available for dashboard.

Things to be kept in mind (Setup Overview):

Targeted System : Debian 9,10
Database : MariaDB 10.4
Server : Apache
Language : Python 3.7/3.5 (We should not use f string)
Front-end : HTML/(Bootstrap/Skeleton)/Javascript/Jquery
Back-end : Flask
Testing : tox
Documentation : may be sphinx
Compitibility with WMF environment

Benefits

Main benefit is DBA can easily track and report the status of the backups, no need to find success/failure/errors manually.
filled backup_objects so, DBA can make sure no backup objects are left can get overview for all backups and its size.

Possible Mentor(s)
Jaime Crespo and Manuel Arostegui

Have you contacted your mentors already?
Yes i have been in contact with mentor and discussed about the project idea.

Deliverables

Describe the timeline of your work with deadlines and milestones, broken down week by week. Make sure to include time you are planning to allocate for investigation, coding, deploying, testing and documentation

 > Working Dashboard for metadata like id, name, status, source, list of failed backups and its error, number of times failure, some more filters with different attributes will be decided as per requirements.
 > Working FLASK app for backend components(CRUD) for metadata backups objects.
 > Working Dashboard for ongoing backups objects that shows status for onging objects, some more filters will be decided as per detailed requirements.

:: First Evalution ::

> Working FLASK app for backend components for ongoing backups objects.
> Maintanable filled backup_objects table.
> testing and Documentation for the whole project.

:: Final Evalution ::

Proposed Timeline:

18 May-6 June
( community bonding period )

  • Studying how backups are working at WMF and getting more familiar with wikimedia by solving some more microtasks (if possible).
  • Understanding the existing code for reading and writing to the backups database.
  • Analysis of backups database.
  • Detailed Requirement Analysis for the project.
  • Setting Up the Basic Environment for developement (server).

Week-1 & 2 (7 - 20 June)
( Project Setup )

  • Setup Flask App including Database for backups (Adding dummy Information).
  • Planning the Design of the front-end and back-end.
  • Add basic flask App repository in codebase for dashboard and backend.

Week-3 (21 - 27 June)
(Front-end (backup-objects - metadata) )

  • Studying which fields from the backups tabel are to be shown on web dashboard.
  • Understanding the field status for backup objects and their errors.
  • Designing the front-end(UI templates) for backup objects.
  • Mentor review for front-end UI and Improvements for the same.

Week-4 (28 June - 4 July)
( Back-end (backup-objects - metadata) )

  • Add models for communicating with database.
  • Retrive metadata on dashboard.
  • Add Filters for status.
  • Add Unit test for flask app.
  • Mentor review for back-end and Improvements for the same.

Week-5 (5 - 11 July)
( Front-end (ongoing backup-objects) )

  • Studying which fields from the database tabels are to be shown on web dashboard for status of ongoing backups.
  • Understanding the field status for ongoing backup objects and their errors.
  • Designing the front-end(UI templates) for ongoing backup objects.
  • Mentor review for front-end UI and Improvements for the same.

Week-6 (12 - 18 July) (12-16 July First evalution)
First Evalution

  • Evalution and Improvements according to feedback.

( Back-end (ongoing backup-objects) )

  • Add models for communicating with database and retrive them on dashboard for ongoing backups objects.
  • Add Filters accordinly for ongoing status.

Week-7 (19 - 25 July)
( Back-end (ongoing backup-objects) )

  • Add Unit test for improved code.
  • Mentor review for back-end(ongoing backups objects) and Improvements for the same.
  • Improvements for whole dashboard if needed.

Week-8 (26 July - 1 August)
( filling in the backup_objects table )

  • Add models for communicating with database.
  • Add Information in backup_objects(table).
  • Add Unit test for improved code.
  • Mentor review for back-end and Improvements for the same.

Week-9 (2 - 8 August)
( Testing )

  • Testing the Flask App rigorously and change accordingly if needed.
  • Improvements if any

Week-10 (9 - 15 August)
( Documentation )

  • Documentation for whole Flask App.

Week 11 (16-23 August) (Final week: Students submit their final work product and their final mentor evaluation)

  • Final Submission of Flask App.

NOTE :
It may possible that I may have failed in describing points in long sentenses but I beleive the point is very clear and all the timelines and points explained above are well understood by me.

Participation

Describe how you plan to communicate progress and ask for help, where you plan to publish your source code, etc
I will be publishing source code on git-gerrit with dev branch. so after reviewing and testing we can merge code to master branch.
I will use Phabricator for managing tasks.
I will be available in working hours (4:00PM to 10:00PM(UTC +5:30)) during GSOC program. I will be availabe on email non-working hours.

We can setup the meeting once or sometimes twice a week in any form as per mentor's convenience for disccusing plans for next week and work completed.

About Me

Your education :
I am pursuing Master of Technology(Computer Science and Engineering) in Indian Institute of Information and Technology-Vadodara. I completed my Bachelor of Engineering(Computer Engineering) in 2020.

How did you hear about this program?
I heared about GSOC from one of my seniors. This is my first time participation in GSOC and i am very excited to work on this project.

Will you have any other time commitments, such as school work, another job, planned vacation, etc, during the duration of the program?
I don't have any specific time commitments during the GSOC program as my Mtech(second semester) exams will be on 12th, May 2021 as per my academic schedule. Next semester will be starting from 19th,July 2021, so after 19th July also we don't have any classes as in my final year i will have to do research thesis so i can manage my time for both. I am only planning for GSOC for this summer.
I intend to spend at least 20-22 hours per week [(3.5 × 6(day) ) * 9 (week) ]for the GSoC Project and on sunday mostly i will be reviewing and self analysing my work. I have added 12-15 hours as a buffer time.

We advise all candidates eligible for Google Summer of Code and Outreachy to apply for both programs. Are you planning to apply to both programs and, if so, with what organization(s)?
I was intrested to apply for Outreachy as it is very good initiative but unfortunately after reading eligibility criteria, i am not eligible to apply.

What does making this project happen mean to you?
Wikimedia is opensource, non-profit, charitable and global recognized organization that would be great to work with wikimedia. I am quite new to open source and having read the project description & discussed with the mentor, I believe this project is very well suited for me. I have developed some dashboard in flask and different Technologies, which has helped me understand the nuances of designing such dashboard. I beleive with my experience i can complete this project with wikimedia and i will get great experience of open source contribution.

Past Experience

Describe any relevant projects that you've worked on previously and what knowledge you gained from working on them.
I have very strong fundamental knowledge of computer science in C language, Algorithm, Data Structure, Operating System, DBMS, Theory of Computation as i have cleared GATE 2020.
I have experienced in working with python,java,C,R ,HTML,CSS,php. I mostly work on mysql,postgress.
I worked on two personal flask projects. One is simple (CRUD)dashboard gave me better insight of working with database and another is flaskblog application where you can write your own small blog with login module,CRUD.
I have almost 1.5 year of experience in web developement.

  1. https://github.com/DMRathod/RD_web
  2. https://github.com/DMRathod/web_flask

In SSIP gujarat heckathon we have devloped one dashboard(PHP) for Employee Attendance Record keeping System that gave me great experince of team collabration, time management and effective time deliverables.

  1. https://github.com/DMRathod/SSIP-Heckathon-project

I have some good experience with android(Java) developement. We have made simple introductory website for our application.

  1. https://gwadp-2c6dd.web.app/

Apart from this i have done some assignmenst at IIIT-V which gave me good insight of the ML,AI and Data Analytics.

  1. https://github.com/DMRathod?tab=repositories

Describe any open source projects you have contributed to as a user and contributor.
I have very small contribution to PecanProject. The aim was to make regex more robust so that connection string can accept url with or without trailing slash.

  1. https://github.com/PecanProject/rpecanapi/pulls

Good Comminication is the key for opensource, In the same organization they need some community bulding efforts by asking the requirements for Extended-R-API (Adding more functionalities to API) to the community or Person who is using API have better idea what kind of improvements required so it easy to extract the better insights from the databse so, i did this by making google-form and gathering those requirements from the community so we can increase the usefulness of the API.

You must have written a feature or bugfix for a Wikimedia project during the application phase (see the section about microtasks in the application process steps), please link to it here. We give strong preference to candidates who have done so.

I am currently working on this microtask and it is almost finished.

  1. https://phabricator.wikimedia.org/T277754

I am Reading how backups are working at WMF and trying to understand the https://phabricator.wikimedia.org/diffusion/OSWB/ repository where some changes required for my project idea.
I am enjoying contributing to open source and would enjoy to continue doing so during my summer and thereafter.

Event Timeline

Hello , @jcrespo @Marostegui
Can you please review my draft proposal?

Thank you

@jcrespo @Marostegui
Hello, what will happen as my and h.krishna's proposal are for same project idea. Is there any other way to resolve conflict ? should i propose for some other project idea ? I am worried about selection.

I can devote 40 hours/week for GSoC project

I think you are expected to dedicate 25h per week, or 40 hours, but take some days off in between. Please adjust your project outcome based on a total of 178h of work, I think is the guidelines (but please double confirm on the Google Summer of Code website).

I am worried about selection

Both Manuel and I haven't talked about the quality of any submission so far, and more likely only will do after the deadline. I haven't checked or compared your submission to that of anyone else. The submission is entirely judged on what you submit, and not compared to anyone else. There is (probably) only going to be 1 student selected among all submissions, independently of the idea. You should chose the idea that you see more interesting or fitting for your skills, not the one that is more likely to be chosen, because there is not such an idea. We will chose a student, not an idea.

Having said that, if you don't feel comfortable with your idea (for any reason), you can change it- but my advice is to not do it because it is repeated (all ideas are likely to be repeated by the end of the process)- that will make no difference. Focus on improving your proposal to demonstrate you are the best candidate. Last year, the top candidates chose different ideas purely by chance.

These are just answers to your questions, I will try to do a second pass later.

One thing that probably is improvable is the deliverables section. While you have the challenge of probably not knowing the scope of the work before a deeper analysis, the deliverables are a bit vague. For example,

Develop Dashboard for metadata backups objects.

Looks very similar to the project's title, and while I can understand the general idea, it would be nice to reword that in a more clear way, to answer "what you will be able to show at the end of this period". For example (you don't need to follow this structure, it is a random example): "at the end of milestone X, a website will be browsable showing 1) the list of backups 2) its failures and 3) the time (in roman numbers) of failure". This is I think a bit more descriptive than "metadata backups", even if it is the same, but tells me you understood what you will deliver.

jcrespo triaged this task as Medium priority.Apr 9 2021, 12:53 PM

Hey @DharmrajRathod98

Thanks for showing your interest to participate in Google Summer of Code with Wikimedia Foundation! Please make sure to upload a copy of your proposal on Google's program site as well in whatever format it's expected of you, include in it this public proposal of Phabricator before the deadline i.e April 13th. Good luck :)

I can devote 40 hours/week for GSoC project

I think you are expected to dedicate 25h per week, or 40 hours, but take some days off in between. Please adjust your project outcome based on a total of 178h of work, I think is the guidelines (but please double confirm on the Google Summer of Code website).

I am worried about selection

Both Manuel and I haven't talked about the quality of any submission so far, and more likely only will do after the deadline. I haven't checked or compared your submission to that of anyone else. The submission is entirely judged on what you submit, and not compared to anyone else. There is (probably) only going to be 1 student selected among all submissions, independently of the idea. You should chose the idea that you see more interesting or fitting for your skills, not the one that is more likely to be chosen, because there is not such an idea. We will chose a student, not an idea.

Having said that, if you don't feel comfortable with your idea (for any reason), you can change it- but my advice is to not do it because it is repeated (all ideas are likely to be repeated by the end of the process)- that will make no difference. Focus on improving your proposal to demonstrate you are the best candidate. Last year, the top candidates chose different ideas purely by chance.

These are just answers to your questions, I will try to do a second pass later.

@jcrespo Thanks a lot for reviewing my proposal i have made changes.

GSoC application deadline has passed. If you have submitted a proposal on the GSoC program website, please visit https://phabricator.wikimedia.org/project/view/5104/ and then drag your own proposal from the "Backlog" to the "Proposals Submitted" column on the Phabricator workboard. You can continue making changes to this ticket on Phabricator and have discussions with mentors and community members about the project. But, remember that the decision will not be based on the work you did after but during and before the application period. Note: If you have not contacted your mentor(s) before the deadline and have not contributed a code patch before the application deadline, you are unfortunately not eligible. Thank you!

@DharmrajRathod98 We are sorry to say that we could not allocate a slot for you this time. Please do not consider the rejection to be an assessment of your proposal. We received over 100 quality applications, and we could only accept 10 students. We were not able to give all applicants a slot that would have deserved one, and these were some very tough decisions to make. Please know that you are still a valued member of our community and we by no means want to exclude you. Many students who we did not accept in 2020 have become Wikimedia maintainers, contractors and even GSoC students and mentors this year!

Your ideas and contributions to our projects are still welcome! As a next step, you could consider finishing up any pending pull requests or inform us that someone has to take them over. Here is the recommended place for you to get started as a newcomer: https://www.mediawiki.org/wiki/New_Developers.

If you would still be eligible for GSoC next year, we look forward to your participation!