Congratulations @Kdhingra2210 on getting selected for this year's Google Summer of Code. It was a pleasure interacting with you, @EBernhardson, @TJones, and @srishakatux. I know I fell short in delivering expectable goals during the pre-GSoC period. Any advice from mentors, admin or @Kdhingra2210 is welcome so that I can apply with greater zeal next year.
My warm wishes to all of you for this year's Summer of Code. :)

Week 1(25/03-31/03) of Pre GSoC Period:-(since the University classes are going on, the progress is slower)

1. Posted on the project thread regarding the choice of NN library (Keras, Tensorflow, PyTorch). Ebernhardson expressed his reservation with Keras/Tensorflow but still, he's open to the choice of working with other libraries. So I decided to give both of them a try and compare their performances to get a better idea so as to make a better choice before GSoC period starts(so that there's no confusion at that time) I will be extensively testing out the primary differences over the due course of time.
2. Then there's part of getting a preliminary code ready so it's easy to do further work during GSoC period. So here's a sample_ndcg computation. (just trying out things to get a better understanding of information retrieval in practice)
3. There's a great resource that I found regarding the research paper. So going through the slides presented in WSDM 2018 and ECIR 2018. Borisov was part of the panel at WSDM.
4. Evaluations:- Tried out a few combinations on current Keras model but will keep this optimization on hold till the time a model (Tensorflow/PyTorch) is finalized.

The patch associated implements the model using Keras. We can surely continue using the same library for sake of simplicity and figure out the solutions around it or we can choose a different library altogether.
Let’s analyze each one of them:-
Tensorflow:- The widely popular library by Google itself has a lot of potential and is certainly good choice wherever you need more under the hood operations and want to have full control over your model. To be honest, it’s unfair to compare them. Keras is a higher level abstraction of deep learning whereas Tensorflow is lower. Keras was used initially in the patch for the same reason(easier implementation and how powerful it’s some of its models can be)
PyTorch:- Right now Tensorflow has limited support for dynamic inputs in comparison to PyTorch. I personally feel the intuitiveness of PyTorch makes it really stand out. PyTorch is too a low-level abstraction but with more feel of using it as a framework too.
What should we use?
Tensorflow is a library with its advantage in visualization through tensorboard, large-scale distributed training(PyTorch has started supporting that now though) and used to deploy model for production.
Considering this GSoC project has implications as far as testing the potential of newer NCM model and not focused on actually deploying model on full scale(Tensorflow has an edge here). Simply saying PyTorch is made for doing research or even for production(if functional requirements are less). It provides a better debugging and developing experience with no. of debugging tools like pdb, ipdb etc to help. And last but not least, its Python at its core.
This blog really puts my points in a better perspective.

In T186742#4072583, @Kdhingra2210 wrote:

Also, for application over the different size of datasets, without retraining can be done using dynamic lstm cells (preferably gru), have worked over dynamic time series analysis and it provide quite better outputs but need relatively more training

In T186742#4064987, @TJones wrote:

In T186742#4058507, @ansh103 wrote:

1. Crowdsourced labeling platform would be tedious indeed. And we can't guarantee the authenticity of volunteers. I gave one of my batchmates the same form. He started assigning random labels to each of result page of a particular query. The point being anyone will get annoyed in this tedious procedure.

For better or worse a significant portion of the ratings were done by people on our team, which provided an additional source of verification. There is also a reasonable body of literature out there for detecting bad-faith raters (and particularly hard tasks); we used multiple raters per task and Krippendorff's alpha. People getting paid to do tasks on Mechanical Turk cheat or click randomly, so there are many techniques for dealing with that. We had hoped that volunteers would have less incentive to cheat, but with enough data we thought we could sort it out. The unsolvable problem was the tedium; it's just so boring and annoying to do that we couldn't get enough data.

Thanks for clearing that out. Tedium is the only reason then.

2. In comparison to previous one, surely this is a better situation given the fact that person has to answer yes or no question. Here's a doubt I have. If we are going to use this survey data to train NNs then having 4 options(irrelevant,maybe, probably and relevant) in survey(like https://discernatron.wmflabs.org)would be better than just yes or no. In this doubt, I may have misinterpreted you and actual survey had 4 options.

We considered many options in terms of the wording of the question and the number of options to offer. We settled on three answers—yes, no, and something that counts as opting out, like "I don't know" (see T176428). We decided to keep it simple because the higher response volume means we can reasonably assign different levels of relevance based on the proportion of yes and no votes (and other factors in the NN). Anyone interested can see T171740: [Epic] Search Relevance: graded by humans for a lot more detail.

Does make sense given the amount of volume. Will update once I go through the link.

Here's my finding running it with 150 epochs:-
https://drive.google.com/open?id=1qGr4s-ywBuyrl8naqBLjW1cVpRhs4XU7
With multi_processing:-
https://drive.google.com/open?id=19jbLjm1wvwRgYU0nqPfAvd20rVzbIEJj

Here's the detailed analysis of the project. Proposal would be based on this.

In T186742#4058149, @EBernhardson wrote:

In T186742#4055625, @ansh103 wrote:

In T186742#4053235, @EBernhardson wrote:

In T186742#4041251, @ansh103 wrote:

Hi @EBernhardson
I am Ansh, a 3rd-year undergraduate student from India. Over the past week, I found my interests in Neural Networks matching with the project mentioned here. I have done “Load Forecasting Project using RNNs for my Institute’s hourly consumption of units” as a part of a project undertaken.
While going through the research article, I found that I need to understand the concepts of Information Retrieval on a better level. So I briefly went through this survey[1] which sums up all click models in a little more descriptive manner but at same time comprehensible enough.

Good initiative. The author of the book is also the author of the DBN implementation we used, so quite likely to be relevant. I looked over the introduction and it's worth noting that the book refers to click prediction / user simulation as the primary goal of a click models. That is likely true in general, but in this particular task we are more concerned with the ability of the model to predict relevance. The relevance predictions are then used as the target document orderings for an ML ranking function.

After having read the research article thoroughly, I did see how relevance prediction(which is what we are concerned about here in our project) was second part of their conclusions apart from click prediction tasks.

So it took quite a time to review all these material but I came to following conclusions:-

1. Wikimedia is interested in this research article, because “neural click model” proposed here is faster on both click prediction task and relevance prediction tasks. At the same time, it adds an additional feature i.e it can learn concepts which can’t be designed manually.

We aren't as worried about speed of relevance prediction, the main goal is for the accuracy of that prediction. The hypothesis is basically that neural click models can generate more accurate predictions of relevance from click data than the DBN. The paper shows, for their dataset, an improvement on ndcg@1 from 0.717 to 0.775 and ndcg@3 from 0.725 to 0.773. If even half of that improvement is seen on our own data that could lead directly to improvements in search results. One difficulty for this task will be that the researchers had manual relevance judgments for 5k queries to calculate those ndcg@k scores, but we don't have a similar quantity of "ground truth" data. We will need to determine some method of comparing the relevance predictions,

My bad here. 5.2 of Research Article clearly shows how NCM performs better than other PGM based click models on ndcg@1 and ndcg@3. They had 41,275 query-document pairs to back their relevance prediction results. But we dont have it as suggested by you.
There are two ways to solve it(correct me if I am wrong):-

1. Like in the research paper, the human generated binary relevance labels were used from Yandex's Relevance Prediction Set. This is an extravagant option. Russia's popular search engine had such options but here at WikiMedia we can't do that(assumption)
2. Estimated relevance data[2]. The paper clearly shows how close calculations are for estimated relevance data and editorial relevance data. NDCG scores are evaluated over estimated, partial editorial data and combined dataset as well.

Indeed 1 is too expensive. 2 is interesting, i hadn't seen this particular paper before. After reading I'm not sure how it solves the problem. My reading is that the primary contribution is a method to apply lambdamart to the probabilities of a PGM (GCM specifically, but others could be adapted) directly. It seems more like another potential avenue to investigate to improve click modeling, to be compared against DBN and NCM.

I've been thinking about this for some time as well and have a few ideas, but not sure how many will pan out:

• We built a crowd sourced labeling platform and set it up at https://discernatron.wmflabs.org. Query + document pairs are labeled both by people on our team and volunteers we could convince. Unfortunately the work is tedious and not directly rewarding. We labeled around 150 queries with an average of 40 documents each this way. A significant number of these queries are from the long tail though, and are unlikely to show up with any frequency in new clickstream datasets. That makes this data too small to put much faith in to evaluate labels generated from click streams.

• We've built out a survey that runs on article pages and asks the user a yes/no question about relevance of a query to that article. We ran this survey for all q+d pairs in the discernatron data and then trained a NN to predict the labels assigned by human judges from the survey data. This looks to work reasonably well, but time constraints have prevented me from moving forward from there. The next step will be to run a survey with queries we don't have labels for and then use the NN to predict the relevance labels. I'm optimistic we can use the output of this NN to evaluate DBN vs NCM but it's hard to say at this point.

• Previously we have evaluated changes to the DBN by running the full training pipeline to build models, and run AB tests with users with the different models. This is a fairly definitive method but it is also time consuming. We typically run AB tests for 14 days, analysis of the results may take another day or two on top of that. 15+ days to get feedback is much too slow to iterate with.

1. Crowdsourced labeling platform would be tedious indeed. And we can't guarantee the authenticity of volunteers. I gave one of my batchmates the same form. He started assigning random labels to each of result page of a particular query. The point being anyone will get annoyed in this tedious procedure.
2. In comparison to previous one, surely this is a better situation given the fact that person has to answer yes or no question. Here's a doubt I have. If we are going to use this survey data to train NNs then having 4 options(irrelevant,maybe, probably and relevant) in survey(like https://discernatron.wmflabs.org)would be better than just yes or no. In this doubt, I may have misinterpreted you and actual survey had 4 options. Basically the NNs have been trained but they are yet to be tested. This can be done after we have completed setup of NCM in GSoC timeline. This can be a task before phase-1 evaluation so that we have enough time left to try out option 3(in case we fail) you mentioned(running the full training pipeline to build models, and run AB tests with users with the different models which takes 15 days roughly)

2. With that being said, we need to see whether that works really or not.

Exactly.

3. In the starting point of the project discussion, “review the aggregation performed in the patch associated with this ticket, to verify that the input vectors are generated correctly according to the paper.” was set as starting point of the project.

Correct. There is a reasonable chance the aggregation i wrote generates correct sparse matrices, but code could always use review.

I am up for some of the todo tasks here. So I'll write codes according to this timeline and tasks with no timeline would be dealt with later:-

Many of the TODO tasks on the data creation side of things are probably not too important for now. TODO's related to code clarity, optimizations, insufficiently abstracted code, etc. can be deferred until we prove with some level of certainty that NCM is comparable to or better than the DBN. If you want to provide patches for 1 or 2 of those items (such as you have already done for item 1) as part of the GSC application process i think that could only be beneficial to your application as a demonstration of your familiarity with concepts involved.

I'll try to solve some of them as a part of application with links to Github repo.
Since all this code cleanup has lower priority, I'll put other todo tasks on a later stage of GSoC timeline then?

1. support n-dimensions 17th March
2. Use numpy arrays and resize? Would cut memory usage vs python list by 1/2. It might also allow faster copy operations from the vectors 18th March
3. Output a single file with all using np.savez But then we have to dig (only a little) into the sparse matrix implementation
4. Verify conclusions from research paper match our data. Although we don't care that much about click prediction. I suppose though we could try and compare results from a real AB test with that of a test that uses predicted clicks. See if they match up?
5. Instead of these generators and excessive error handling, what if we copy the whole directory at the outset and use a contextmanager to clean it up?
6. Evaluate with larger datasets line
7. mask out the first element of time series, it is only the query and has nothing to predict line
8. On my laptop (with only 2cpu) this is much slower, worth trying on a high core count machine though. workers=1, use_multiprocessing=True
9. Mini batches/Memory Sized Batches More training data? Activation Functions Units reduce_input_dim

4. That was not done, so it was assumed that datasets were generated correctly. And click data was made public with the attached Gerrit links.

Also correct. It might also be worth noting this is only a sample of the full dataset.

Before dealing with the click data generated, I would take up the task of reviewing the patch provided. After that, I'll provide an update here.

[1] https://clickmodels.weebly.com/uploads/5/2/2/5/52257029/mc2015-clickmodels.pdf

I had some doubts though:-

1. Regarding Implementation of n-dimensional matrix:- I am not sure of use of pyspark in existing code. Maybe because I dont know about it much. Here's what I created[3] using numpy only

The n-dimensional matrix isn't super important. Right now the code is of course messy and has two separate implementations (SparseVectorBuilder and Csr3dBuilder) for building a sparse 3d array ontop of the 2d array provided by scipy. There isn't actually a use case (yet) for more than 3 dimensions, it just seemed the code might be more general, and more likely to be correct, if it was written for N dimensions.

I can certainly understand that the pyspark portion of the code is hard to follow. It's not clear from this code at all what the shape of the data read from the input_dir is, or what it represents. Without that information it's likely very difficult to evaluate if the aggregation is correct. I will try to find a way to at least document what the input data is to help make it clearer. In general spark is used here because the data is relatively large and spark allows applying significant amounts of parallelism to the task.

I have started to grasp what is going on here and would be soon well acquainted with PySpark, essentially made for big data processing. Online resources[4] are helping a lot. Got to know how Python would be better way to go than Scala in our case(given the popularity with NNs as said by you in an initial comment). Understood why findspark was imported and have started reading about RDDs and dataframes. Alongwith with this guide and the information you'll provide later on, we can make progress.

2. Subtasks mentioned in the project are for GSoC's whole timeline or for now before the application period starts?

Subtasks are for the full timeline.

[2] https://zhangyuc.github.io/files/wang10cikm.pdf
[3] https://github.com/ansh103/Search_Engine_Click_Models/blob/master/sparse_n_dim.py

Other things that I had in my mind regarding my Proposal:-
I was planning to attach a detailed analysis of our whole model(including all conclusions we are drawing in this Phabricator thread) and the way we'll proceed in this project. I'll post a link to that soon so that you can comment on it.
Apart from this, I am planning on getting the proposal draft ready by 20th/21st March. It won't be too late right?
[4] https://www.datacamp.com/community/tutorials/apache-spark-python

In T186742#4053235, @EBernhardson wrote:

In T186742#4041251, @ansh103 wrote:

Hi @EBernhardson
I am Ansh, a 3rd-year undergraduate student from India. Over the past week, I found my interests in Neural Networks matching with the project mentioned here. I have done “Load Forecasting Project using RNNs for my Institute’s hourly consumption of units” as a part of a project undertaken.
While going through the research article, I found that I need to understand the concepts of Information Retrieval on a better level. So I briefly went through this survey[1] which sums up all click models in a little more descriptive manner but at same time comprehensible enough.

Good initiative. The author of the book is also the author of the DBN implementation we used, so quite likely to be relevant. I looked over the introduction and it's worth noting that the book refers to click prediction / user simulation as the primary goal of a click models. That is likely true in general, but in this particular task we are more concerned with the ability of the model to predict relevance. The relevance predictions are then used as the target document orderings for an ML ranking function.

After having read the research article thoroughly, I did see how relevance prediction(which is what we are concerned about here in our project) was second part of their conclusions apart from click prediction tasks.

So it took quite a time to review all these material but I came to following conclusions:-

1. Wikimedia is interested in this research article, because “neural click model” proposed here is faster on both click prediction task and relevance prediction tasks. At the same time, it adds an additional feature i.e it can learn concepts which can’t be designed manually.

We aren't as worried about speed of relevance prediction, the main goal is for the accuracy of that prediction. The hypothesis is basically that neural click models can generate more accurate predictions of relevance from click data than the DBN. The paper shows, for their dataset, an improvement on ndcg@1 from 0.717 to 0.775 and ndcg@3 from 0.725 to 0.773. If even half of that improvement is seen on our own data that could lead directly to improvements in search results. One difficulty for this task will be that the researchers had manual relevance judgments for 5k queries to calculate those ndcg@k scores, but we don't have a similar quantity of "ground truth" data. We will need to determine some method of comparing the relevance predictions,

My bad here. 5.2 of Research Article clearly shows how NCM performs better than other PGM based click models on ndcg@1 and ndcg@3. They had 41,275 query-document pairs to back their relevance prediction results. But we dont have it as suggested by you.
There are two ways to solve it(correct me if I am wrong):-

1. Like in the research paper, the human generated binary relevance labels were used from Yandex's Relevance Prediction Set. This is an extravagant option. Russia's popular search engine had such options but here at WikiMedia we can't do that(assumption)
2. Estimated relevance data[2]. The paper clearly shows how close calculations are for estimated relevance data and editorial relevance data. NDCG scores are evaluated over estimated, partial editorial data and combined dataset as well.

2. With that being said, we need to see whether that works really or not.

Exactly.

3. In the starting point of the project discussion, “review the aggregation performed in the patch associated with this ticket, to verify that the input vectors are generated correctly according to the paper.” was set as starting point of the project.

Correct. There is a reasonable chance the aggregation i wrote generates correct sparse matrices, but code could always use review.

I am up for some of the todo tasks here. So I'll write codes according to this timeline and tasks with no timeline would be dealt with later:-

1. support n-dimensions 17th March
2. Use numpy arrays and resize? Would cut memory usage vs python list by 1/2. It might also allow faster copy operations from the vectors 18th March
3. Output a single file with all using np.savez But then we have to dig (only a little) into the sparse matrix implementation
4. Verify conclusions from research paper match our data. Although we don't care that much about click prediction. I suppose though we could try and compare results from a real AB test with that of a test that uses predicted clicks. See if they match up?
5. Instead of these generators and excessive error handling, what if we copy the whole directory at the outset and use a contextmanager to clean it up?
6. Evaluate with larger datasets line
7. mask out the first element of time series, it is only the query and has nothing to predict line
8. On my laptop (with only 2cpu) this is much slower, worth trying on a high core count machine though. workers=1, use_multiprocessing=True
9. Mini batches/Memory Sized Batches More training data? Activation Functions Units reduce_input_dim

4. That was not done, so it was assumed that datasets were generated correctly. And click data was made public with the attached Gerrit links.

Also correct. It might also be worth noting this is only a sample of the full dataset.

Before dealing with the click data generated, I would take up the task of reviewing the patch provided. After that, I'll provide an update here.

[1] https://clickmodels.weebly.com/uploads/5/2/2/5/52257029/mc2015-clickmodels.pdf

I had some doubts though:-

1. Regarding Implementation of n-dimensional matrix:- I am not sure of use of pyspark in existing code. Maybe because I dont know about it much. Here's what I created[3] using numpy only
2. Subtasks mentioned in the project are for GSoC's whole timeline or for now before the application period starts?

[2] https://zhangyuc.github.io/files/wang10cikm.pdf
[3] https://github.com/ansh103/Search_Engine_Click_Models/blob/master/sparse_n_dim.py

Hi @EBernhardson
I am Ansh, a 3rd-year undergraduate student from India. Over the past week, I found my interests in Neural Networks matching with the project mentioned here. I have done “Load Forecasting Project using RNNs for my Institute’s hourly consumption of units” as a part of a project undertaken.
While going through the research article, I found that I need to understand the concepts of Information Retrieval on a better level. So I briefly went through this survey[1] which sums up all click models in a little more descriptive manner but at same time comprehensible enough. So it took quite a time to review all these material but I came to following conclusions:-

1. Wikimedia is interested in this research article, because “neural click model” proposed here is faster on both click prediction task and relevance prediction tasks. At the same time, it adds an additional feature i.e it can learn concepts which can’t be designed manually.
2. With that being said, we need to see whether that works really or not.
3. In the starting point of the project discussion, “review the aggregation performed in the patch associated with this ticket, to verify that the input vectors are generated correctly according to the paper.” was set as starting point of the project.
4. That was not done, so it was assumed that datasets were generated correctly. And click data was made public with the attached Gerrit links.