| Aklapper | |
| Nov 20 2016, 7:14 PM |
| F7511380: T151161.png | |
| Apr 12 2017, 7:32 PM |
| F4796075: x.png | |
| Nov 20 2016, 7:14 PM |
Followup on T146631
On https://wikimedia.biterg.io/app/kibana#/dashboard/Git-Demographics in "Last Attracted Developers", "First Commit Date" is not the date when the initial patch set revision was committed, but when the last patch set revision was committed.
This makes it unnecessarily hard to really get a list of the last new contributors and actively make sure that their patches get quick feedback.
Comparison:
Upstream: https://gitlab.com/Bitergia/c/Wikimedia/support/issues/16
I'm not sure, so please @Dicortazar correct me if I'm wrong, but the Git demographics panel is correct, the thing is you would need demography information applied to Gerrit.
@Dicortazar how difficult is to have that information for our upstream product with current ES indexes?
Ahem, Luis made me realize that Git-Demographics is about... Git (merged stuff). Not Gerrit (potentially non-merged stuff).
As we use the data on http://korma.wmflabs.org/browser/code_contrib_new_gone.html to nag developers to review proposed patches by new contributors, we'll also need a Gerrit one in Kibana which we don't have yet.
Hence I'm changing the scope of this task.
(Unrelated sidenote: Due to its UI this can only work after changing the default (2y) timespan in the upper right corner to something like "Relative > 20y ago". Jesus pointed this out.)
That git_demographics_newcomers visualization shown in the "Attracted developers" widget is based on the author_min_date field in the git_enrich index.
There is no similar field author_min_date in the gerrit_enrich index available. :(
@Aklapper we have to work in two ways with this analysis:
Thing is: We want to be able to support new contributors already before they finally manage to get their changeset merged. For this, we need to know which Gerrit accounts are new and have provided a first changeset (regardless of the changeset status).
This is a priority, as we need this data by the end of May (due to T160430).
I guess implementation would have to happen in [[ https://github.com/grimoirelab/GrimoireELK/blob/master/grimoire_elk/elk/gerrit.py | elk/gerrit.py ]] (for reference: this is the commit which introduced author_min_date in elk/git.py)?
Indeed, we would need demography support for Gerrit for doing this.
This is a priority, as we need this data by the end of May (due to T160430).
I guess implementation would have to happen in [[ https://github.com/grimoirelab/GrimoireELK/blob/master/grimoire_elk/elk/gerrit.py | elk/gerrit.py ]] (for reference: this is the commit which introduced author_min_date in elk/git.py)?
We have this in our roadmap, but we don't have a deadline for implementation. We can try to push this forward, but at this point we're in the middle of a couple of transitions, and it's going to be difficult.
Let us discuss about this, and I come back with something more conclusive.
Trying to find short term workarounds fir this problem,
Very welcome surprise by Bitergia (thanks folks!): This should be possible soon. Admins can access a sneak preview which lists New Authors in Gerrit by first contribution date (among many other useful things):
Config: For Wikimedia, we can kill the "New Authors per First Project"/C_Gerrit_Demo_Projects_Pie and the "New Authors by Top First Projects"/C_Gerrit_Demo_Project_TS widgets, as we are one single boring project. :)
This is something maybe you want to change at some point. In case it would make sense to have "subprojects" with some meaning for you, we could configure that in the dashboard.
So.... when could this go live? :) Any obstacles?
Do you need any input from me? (I could imagine to integrate these widgets either into Gerrit-Backlog or Gerrit, to not have a 3rd Gerrit related dashboard.)
FYI I've updated gerrit-reports to use the new gerrit API parameters and we have new updates: https://www.mediawiki.org/w/index.php?title=Gerrit%2FReports%2FOpen_changesets_by_newbie_owner&type=revision&diff=2486934&oldid=2201127
I plan to use this information as usual if I have some time in the next few weeks, i.e. assist new users to get their patches improved and merged.
@Aklapper we've updated
Done!
Sorry for the delay. Here are the new versions for Git and Gerrit:
https://wikimedia.biterg.io/app/kibana#/dashboard/C_Git_Demo
https://wikimedia.biterg.io/app/kibana#/dashboard/C_Gerrit_Demo
Do you need any input from me? (I could imagine to integrate these widgets either into Gerrit-Backlog or Gerrit, to not have a 3rd Gerrit related dashboard.)
What we are planning in a nearly future is to have a demographics panel with several datasources.
Yay, thank you a lot! happy to see that deployed "by default" soon!
Slightly offtopic and rather a note to myself:
Whoah, initially I got utterly confused. I'm not sure when and why my Chromium browser inserts ? in URLs sometimes, but
https://wikimedia.biterg.io/app/kibana#/dashboard/C_Gerrit_Demo (correct)
https://wikimedia.biterg.io/app/kibana?#/dashboard/C_Gerrit_Demo (sometimes wrong)
are different things and sometimes display different data which is extremely confusing. Which initially made the C_Gerrit_Demo dashboard show Git instead of Gerrit data. I don't get it, but checking that the repos listed are mediawiki/core format (=Gerrit) and not https://gerrit.wikimedia.org format (=Git) helps.
Any news when to see https://wikimedia.biterg.io/app/kibana#/dashboard/C_Gerrit_Demo deployed 'by default'?
And way more important (as this blocks T167085), is it possible to update the data? Currently the latest "new authors" entry is from July 13th and we need recent names...
Any news when to see https://wikimedia.biterg.io/app/kibana#/dashboard/C_Gerrit_Demo deployed 'by default'?
And way more important (as this blocks T167085), is it possible to update the data? Currently the latest "new authors" entry is from July 13th and we need recent names...
Not by default, but I did a run of the scripts, so you have fresh indexes.
Something weird I've noticed is that in the whole history of Git, Facebook has a large share of authors. Drilling down, that seems to be due to some repos that probably you forked from upstream repos with a lot of Facebook activity, but still... Let us know if numbers don't seem to match.
Thank you!
Something weird I've noticed is that in the whole history of Git, Facebook has a large share of authors. Drilling down, that seems to be due to some repos that probably you forked from upstream repos with a lot of Facebook activity, but still... Let us know if numbers don't seem to match.
We're aware; I don't consider this a bug currently as we can manually exclude those repositories, but thanks for the heads-up! Now I only need a fresh DB JSON dump. :D
For the records, we need this data 8 times a year (beginning of Jan,Mar,Apr,Jun,Jul,Sep,Oct,Dec) to either contact new devs for a survey or quarterly stats.
Manual updating is cumbersome and efforts can sometimes collide with our deadlines.
Quoting Bitergia:
We have been working on a new, unified Demography panel for both Git and Gerrit data. Now, instead of visiting different panels for each of these sources, you can use the widget "Data Source" to select the data you want to see (see attached screenshot below). [...] Please note that the former Git Demography panel will remain as a legacy version at https://wikimedia.biterg.io/app/kibana#/dashboard/C_Gerrit_Demo , so we encourage you to use the new version from now on.
The new panel does not list the "repos by new authors" on which the "last attracted developers" worked (a small regression, but we will survive that), but as the legacy panel does not receive updated data anyway there's IMHO no sense in keeping it around.
Furthermore a data inconsistency bug got also fixed by Bitergia in https://github.com/chaoss/grimoirelab-elk/pull/451 but I could still reproduce that problem in one case. Hence leaving this ticket open though it's nearly fixed.