I strongly disagree with this evaluation.

My patches were up as early as June the 17th, and I was awaiting reviews, and responding to them as soon as possible. The reviews did come, but none of them addressed all the issues at once. Each review seemed to bring up issues that weren't brought up in the earlier patchset, and the -1s kept on stacking up. If the reviews were all provided at once, or in a couple of patchsets, I would have resolved them much earlier, naturally. The delay, I believe, is as much of the mentor's responsibility as it is mine. I have regularly updating patches according to the necessities, but if the need was to have the MVP up earlier, I believe the reviews should have been up earlier, too.

Also, it is worth noting that jenkins was down till just one week before the deadline, and that denied me the opportunity to test my patches against the server. It should have been resolved earlier.

The MVP said that the patches would be considered if they had been +2ed by jenkins. Despite jenkins being down for so long, I managed to meet that, and feel that I have been very unfairly left out by the 'consideration'.

I beg @jayvdb and @Legoktm to reconsider this decision, as I believe I have shown, in terms of code ( given the constraints ) and the dedication, that I am capable of finishing this project.

I have both patches up with +2, met the other requirements, and since this is now a value judgement, I urge you to reconsider, and guarantee the same level of performance I have shown in the last three weeks, especially.

In the last meeting, you had asked me if I would rather a good review and fail now, than a poor review that ends up having me fail the project later on. I went for the latter, and stated confidence in the belief that since I was up to scratch withthe code now, I would not fail the final deliverables, either. You said the code was looking close to the end product, and that since I was on top of the code, I could improve it. You had also said that you had passed worse students than me, and that you believed that this could be achieved. I urge you to stick to that rationale, and give me the chance to continue contributing to this project.

@jayvdb @Legoktm @QuimGil @polybuildr - please respond. Pass or fail, I'd like to clear up the fact that I had done what was required of me, and was yet failed when it came down to personal judgement.

FWIW, the write = True bug, which is being quoted as an example here, was indeed raised on https://gerrit.wikimedia.org/r/#/c/294901/2/tests/thanks_tests.py on the 22nd, and I responded with my doubt on the 23rd. @jayvdb missed my reply comment, and responded to it today at 11am IST. The tests were passing without it, and thanks_tests wasn't writing to the wiki, so my reply was very reasonable. I find it extremely unfair that it has been used to disqualify me. It isn't a valid reason at all.

@darthbhyrava The decision of the mentors is final, and I'm not sure if there would be any effect in trying to change that now.

Its stated clearly that the MVP should be in merged state before the mid-term deadline, and being there at the both end of GSoC, I need to agree with @jayvdb that it is the duty of the student to find alternatives and get their code mergeable somehow.

The mentors are expected to spend only 5 hours per student per week, while the student a minimum 40, clearly showing the need for extra work. I am sorry that the decision is tough, but that's how the program works. Its the duty of the student to understand what the community and mentors expect from them within the GSoC evaluation timeframe, and failure to find that out in time will end up in situations like this.

I can assure you that your mentors have thought a lot on the decision (over conpherence) and have justified the decisions with solid reasons, which is taken care of. Thank you for your contributions to the project, and I hope the feedbacks help you to the next level!

Hi, I was a GSoC student last summer, and while I haven't mentored a student for GSoC, I did do a little mentoring for the previous Google Code In. Having had a great experience working with the Wikimedia community, I introduced @darthbhyrava to it.

In sharp contrast to my wonderful experience, the state of this evaluation and the responses of the community to it cause me great pain and since I care greatly for the way we as a community treat contributors, I felt that I must leave this rather long response. Please forgive me if I inadvertently offend someone, I assure you that is not my intention.

It is true that we demand students' complete their MVP by the midterm evaluation, but I'd like you to take a moment and ask yourself why. IMHO, it is because we want to ensure that students are not negligent in their duties - whether it be working on code and responding to reviews or whether it is staying in touch with the mentors. I have seen the state of some students past Wikimedia mentors and org admins have failed - the students very clearly demonstrated a serious lack of dedication to both of these duties. Try as I might, I simply don't see a a parallel here. The student in question has submitted patches for review, modified them based on review comments and kept in touch with the mentors. I see no sign of negligence.

I would like to further point out that not everything is under a GSoC student's control. How do their patches reach mergeable state? Because community members and mentors review them and suggest changes - that's why the GSoC program has mentors in the first place, doesn't it? As @01tonythomas points out, a much larger effort is required on the part of the student, and I whole heartedly agree. However, can we as a community possibly expect improvement in the patches that a student submits if there are no reviews pending for them to work on? It is simply impossible for a student to get their patches to a meargeable state if they don't have advice on how to improve them. Another argument that was made was of certain things being so obvious, there should be no reason for them to be review comments. Sure, there exist such things. But seriously, we're human beings, we make mistakes, we need advice - sometimes repeated for it to stick. Derogatory reactions to these kind of mistakes is really not what I expect from the community - and certainly not a failed mid term.

All of my above arguments were attempted counters to reasons pointed out earlier. However, and this is probably the most shocking part for me to witness - the reasons pointed above for failing don't convince me in the first place. I haven't looked into the matter thoroughly, but it seems to me that the student has actually completed the MVP, albeit not being up to the mark. The proposal says (with the mentors' approval) that a V+2 by Jenkins can also be considered sufficient, provided the patches are also integrated on a github branch with the respective tests passing. It seems like they are. Also, one must also investigate why the patches aren't submitted. Is it because the student has not written any code to be submitted? No, it's because the patches are under review. Is the student not responsive to review comments? It doesn't seem like that either. The student also mentions that pywikibot master itself had issues with tests failing - irrespective of his patches. This is certainly possible and certainly seem like extenuating circumstances. After all, we don't expect gsoc students to fix all the mistakes other people make, do we?

I'm completely at a loss to understand why this student was failed. My confusion is not helped by the fact that the mentor has quite clearly stated that he is pleased with the quality of work and the patches are in good shape. In addition, he also says that the student would probably finish the project. Why in the world are we failing this student then?

It makes me extremely sad to see that we as a community would fail a student because we sometimes have issues with reviewing things promptly (which is perfectly reasonable in itself) and because we are simply ignoring extenuating circumstances that are not in the student's control and failing them based on a rule we set up in the past while completely ignoring the spirit behind it.

Is this the state that our community has been reduced to? The same community that encouraged me to learn new things, try out things I thought were beyond my understanding, and even encouraged me to ask stupid questions publicly on IRC? Because in the end, we're all learning? I am indeed sorely disappointed.

If this is what we have been reduced to, then it seems like we have failed as a community to encourage open source contributors - and that really, really hurts.

Just remembered something from my experience last year, and I thought it would be relevant to share. From the IRC office hours before the start of GSoC and Outreachy last year:

@Niharika: Bonus points for asking silly questions, so don't be afraid.
@Aklapper (andre__): Ask! Don't be shy! And no question is "stupid" or such, no worries please!

The was the attitude of the community that welcomed me into the Wikimedia movement and that's the one I'd want to be part of. :) I'm sure you all would agree.

@polybuildr , I wont be debating the decision in public, as that is messy for all concerned, can't alter the outcome at this stage, and is contrary to the GSOC rules.

I suggest that you read http://bhyrava.me/code/gsoc/wikimedia/2016/06/23/GSoC7/ . On June 23rd, after the mid-term evaluations had begun, Darth knew he was failing, and publicly stated it in his blog post, along with a decent 'meta' assessment of why he was failing.

You may find it useful to step through the project as it progressed to see why, and especially carefully look at the review comments in gerrit to understand why the effort of the last week wasnt enough to get past the MVP line by the last day of the mid-term evaluation period. If you still have concerns after that, please discuss them privately with Darth, the GSOC admins and/or myself as appropriate.

In T138304#2410111, @01tonythomas wrote:

Its stated clearly that the MVP should be in merged state before the mid-term deadline, and being there at the both end of GSoC, I need to agree with @jayvdb that it is the duty of the student to find alternatives and get their code mergeable somehow.

@01tonythomas, it seems that you have made your decisions without looking at my MVP. My MVP does not require code to be merged, it needs a +2 from jenkins, which I have achieved. I post my MVP here:

• MVP for the mid-term evaluation:
  • The MVP would be subtasks 1 and 2 implemented, reviewed and wrapped up.
  • The code for subtasks 1 and 2 must all be merged to pass the mid-term evaluation.
  • If the code is not completely merged, then the code may still be considered if all of it has been submitted in Gerrit with a +2 from jenkins-bot.
  • There must be a branch on my GitHub fork which has all the aforementioned patches integrated and all the tests passing.
  • Apart from the technical deliverables, I would also be having weekly reports for progress along with a weekly blog post, and regularly update the documentation as required.

Please read point 3. It can be considered, and I find the consideration very unfair and illogical, as it not being backed up, and my valid counter-arguments are being dismissed without being responded to.

In T138304#2410111, @01tonythomas wrote:

@darthbhyrava The decision of the mentors is final, and I'm not sure if there would be any effect in trying to change that now.

No, it is NOT final. Please read this. I quote:

In the unlikely event that a mentor and organization administrator do not agree on a student’s grade for any evaluation, the decision of the organization administrator is the final one.

In the also unlikely event that a student does not agree with a mentoring organization’s evaluation decision at either the midterm or the final, the student may choose to submit his/her entire project plan, timeline and code sample to Google’s program administrators. Google will choose a Google engineer who is not working with any Google Summer of Code mentoring organization or the Melange project to review the code and arbitrate the decision. The decision of Google’s independent engineer is final.

I urge @01tonythomas and @Sumit to look at this new piece of information and reconsider their decisions. YOUR decision is final, if you believe that I should be passed.

In case you choose not to, and do not have justifiable decisions to back them up, I would be left with no choice but to approach the independent Google engineer, and other such options.

In T138304#2410965, @jayvdb wrote:

@polybuildr , I wont be debating the decision in public, as that is messy for all concerned, can't alter the outcome at this stage, and is contrary to the GSOC rules.

I disagree with this. I think the decision should be continued in public, as you are not providing me any justifiable decisions in private, and I feel wronged and robbed of this opportunity. I cannot afford to pay my next semester's college fees, and had this summer to earn. I turned down other options to go for GSoC, and since I feel I am being unfairly denied the chance to continue, I would like other people like @Qgil, Stephanie Taylor at Google, and others to objectively look at the MVP, my work, the claims due to which I have been failed, and then provide their decision.

I suggest that you read http://bhyrava.me/code/gsoc/wikimedia/2016/06/23/GSoC7/ . On June 23rd, after the mid-term evaluations had begun, Darth knew he was failing, and publicly stated it in his blog post, along with a decent 'meta' assessment of why he was failing.

This is selective quoting. In my post, I'd said:

As of today, as I type out the words into the midnight of the 23rd, I am failing. Close, but failing.

Focus on as of today. That was because my patches did not yet have a +2 by then.

I had also mentioned, later in the post, that

But back to the moment. I have a patch to submit, a patchset to fix, and three and a half days left in order to meet MVP objectives. Lots of work. Here’s to a great journey so far, and here’s to the final charge.

And very soon, I sent both @jayvdb and @Legoktm a mail at Jun 25th 12:05 am (IST) stating that my patches had been +2ed by jenkins, and that I met the other requirements, and if there was anything left to do to pass mid-term.

I never received a reply.

Back to the main concern: here's the evaluation @jayvdb submitted to Google:

I am generally quite pleased with the improvements over the last three weeks, and your last blog post (http://bhyrava.me/code/gsoc/wikimedia/2016/06/23/GSoC7/), especially the part about reading code, indicates that you're on the right path to understanding Pywikibot, open source and most importantly yourself. However the deadline is fast approaching, and the MVP hasnt arrived. The patches have quite a bit more work required to be polished enough to be merged. But the main problem is that the tests are not extensive, and do not provide test coverage of all of your code, including lots of edge cases. Given more time, I believe you could finish the MVP. I strongly recommend that you participate in Google Summer of Code again, and put in 100%+ from day 1, and focus on the design and the tests - if you get those right, the code writes itself.

@01tonythomas, @Sumit, @Legoktm, @Niharika, @Qgil - please look at the MVP posted above, my patches at 294901 and 295132, the timing and nature of reviews, and then make an unbiased call.

@darthbhyrava, your two mentors are experts in their area, but most importantly they have extensive experience mentoring new developers both informally and as GSoC mentors. They have mentored dozens of developers in the past years. They have a good grasp of when a project is moving forward fluently or is struggling, and they can compare a project like this one with the work of other developers they are mentoring now or mentored in the past. I trust their criteria, and Wikimedia in general trusts the criteria of their GSoC mentors.

In past editions, when I was Wikimedia org admin for GSoC and Outreachy, I have advocated for being more strict assessing mid-term and final evaluations. Wikimedia has participated many years in these programs, and we have seen a pattern of projects struggling in the first half, still passing to avoid the tough call, and then struggling even more in the second half. No student thinks that this will happen to them, but we have years of records to sustain that this pattern has a base. Meanwhile, other students invest a lot of time, energy, and passion pushing their projects with strong motivation and regular reports, staying in touch with their mentors regularly. They get a Pass, and we think that not diluting the value of such Pass important. This is why we included the need to be more disciplined in mid-term reviews in our lessons learned.

It is easier for mentors to let students pass. Resolving a different decision is not an easy step for them. This is why they think very carefully before failing an evaluation. The decision is firm. Please accept it. A lesson here is to work regularly since the beginning and stay in touch with your mentors as in a continuous conversation. Final sprints are always risky.

In T138304#2410965, @jayvdb wrote:

@polybuildr , I wont be debating the decision in public, as that is messy for all concerned, can't alter the outcome at this stage, and is contrary to the GSOC rules.
...
If you still have concerns after that, please discuss them privately...

I'm surprised at this response. Wasn't another one of the tenets of the Wikimedia movement open and transparent communication - thus ensuring fairness?

However, I can see that everyone seems to have made up their mind in this case, and that further comments will just be noise. I'd just like to point out one last time that:

1. According to an agreement between the student and the mentors, the student appears to have completed the MVP.

2. Patches improve with code review. Code review is not always prompt - but to blame a student for this seems extremely unwise.

3. Humans make mistakes and humans have biases. Giving this case a closer objective review with multiple eyes cannot hurt and might help portray this situation in a different light.

After this, all I can do is rely on the Wikimedia community and have faith that they'll make the right call.

In T138304#2412552, @polybuildr wrote:

In T138304#2410965, @jayvdb wrote:

@polybuildr , I wont be debating the decision in public, as that is messy for all concerned, can't alter the outcome at this stage, and is contrary to the GSOC rules.
...
If you still have concerns after that, please discuss them privately...

I'm surprised at this response. Wasn't another one of the tenets of the Wikimedia movement open and transparent communication - thus ensuring fairness?

Students are entitled to contest the mid-term decision, and it seems that Darth is inclined to, but they need to follow Google's relevant rules and processes for the program, which do include an outside review.

There are plenty of public records for this project that provide enough transparency that an outside review can be done, should it be necessary.

That said, it is worth noting that Wikimedia also doesn't practise total transparency all the time either, as there are times that doing so would break the law, or would unnecessarily harm another party without their prior and informed consent.

If Darth doesnt formally contest of the decision, but still wants a more detailed description of how the MVP was not reached, I could privately provide it to Darth for him to share publicly if he wished to.

@Qgil, I understand your decision to trust the veteran mentors for my project, and I agree with the fact that they are experts in their areas, and that they are experienced. I have personally seen evidence of their expertise over the course of the project, and respect them for that. However, I urge you to consider this case in isolation, and to look at the details and counter-arguments that I have provided.

@jayvdb, I do disagree with this evaluation, and hence I am still appealing against the decision, but I would appreciate it if you could privately provide me the details of how the MVP was not reached, if possible.

I would also like to take the opportunity to publicly state that even if the evaluation stays this way after I am done appealing to the Org Admins here, and the Google independent engineers later on, I would still like to complete the deliverables I had promised in my proposal. As stated in my last blog post here, I have learnt a lot over the course of this project, and despite the personal experience being very unpleasant, I enjoyed the professional learning curve.

As @jayvdb put it, it would augment my formal education, and give me confidence to see my patches merged, apart from teaching me the professional side of work that a technical developer needs to be proficient at in order to progress. I do like what Pywikibot is about, and I would like to continue to be a part of it, albeit on a volunteer basis.

However, I would not be sticking to the same timeline, and may take two-three months longer to achieve my goals, since my priorities would change. Also, I would not also expect the same levels of commitment from the mentors anymore, and look to the community as a whole in times of doubt or guidance.

GSoC mid-term evaluation is done and currently out of our hands. We have heard @darthbhyrava's reasons, we have heard @jayvdb's reasons, and with the information available I recommend you to ask the GSoC programs administrators for their evaluation of the situation. This is your best guarantee of receiving a fair and neutral evaluation. We will follow GSoC program's resolution without discussion.

@darthbhyrava, needless to say you are welcome to continue participating in this or any other Wikimedia project on a volunteer capacity, at your own path and free from the requirements that we set in GSoC projects.

I hope we can find the best solution to the current situation. GSoC is supposed to be fun for students and mentors and I am very sorry to see that you both are going through a lot more trouble and stress than needed.

In T138304#2413864, @Qgil wrote:

GSoC mid-term evaluation is done and currently out of our hands. We have heard @darthbhyrava's reasons, we have heard @jayvdb's reasons, and with the information available I recommend you to ask the GSoC programs administrators for their evaluation of the situation. This is your best guarantee of receiving a fair and neutral evaluation. We will follow GSoC program's resolution without discussion.

@darthbhyrava, needless to say you are welcome to continue participating in this or any other Wikimedia project on a volunteer capacity, at your own path and free from the requirements that we set in GSoC projects.

I hope we can find the best solution to the current situation. GSoC is supposed to be fun for students and mentors and I am very sorry to see that you both are going through a lot more trouble and stress than needed.

Thank you for resolving the state of affairs, @Qgil, and pointing me to a definite course of action I can take now.

I apologise for all the mess that my disagreement with the evaluation has caused, and I am sorry that I have dragged so many professionals through this sad and stressful experience, too. This was never my intention at all, and this doesn't diminish, in any way, the gratitude I have for the things I have learned here so far from mentors. I acknowledge the the positive comments regarding my work in the review - thank you very much for that, jayvdb.

There will not be any more noise on this task from my end, unless absolutely required.

I just wanted to post an update of proceedings here. Google has reviewed my case, and ruled against overturning the decision provided. Hence, the decision stays the way it is right now. Despite not agreeing with the decision, I am now left with no choice but to accept it.

I apologize once again for all the inconvenience caused, and thank you for the time that has been put into this project. I shall put in more efforts the next time to ensure such a scenario doesn't occur.

As stated earlier, I shall continue contributing to the project and hope to finish it, albeit on a volunteer basis.

@darthbhyrava thank you for your understanding and your kind words. I am looking forward to seeing you around, involved in your project on a volunteer capacity.