In T190660#4720061, @Aklapper wrote:

@Kdhingra2210: Any news to share about the status of this task? Thanks in advance!

@srishakatux yes, testing is ongoing and I think it should be finished in next week or so.

@srishakatux the model is complete but it is in testing over the complete data, so those patches and the project should be completed once the testing is over. It should take more than a week to finish those tests and I am currently working on those patches.

@TJones and @EBernhardson

@EBernhardson you were right, I made a mistake while defining an object in cython code

@EBernhardson I did implement 3d sparse matrices in scipy too

@EBernhardson and @TJones during the past couple of weeks I have majorly worked in building different types of sparse matrices and understanding the research paper because there are certain minute details which can affect it largely.

Thanks, @EBernhardson, @TJones, @srishakatux and other fellow community members for accepting my proposal.

In T186742#4150338, @ansh103 wrote:

Congratulations @Kdhingra2210 on getting selected for this year's Google Summer of Code.

Thank you very much!

In T186742#4136657, @EBernhardson wrote:

Interesting that they have a SparseTensor but it's not supported.

Yepp, it has support just like normal sparse matrices but converting it to dense batches is very costly compared to scipy.sparse ones. (at least while computing using CPU)

Sure, i'll pull some data for 30 and 60 day aggregations and we can compare the datasets.

yep, I will look into these once uploaded

I have tried plenty of approaches to maintain 3d sparse matrices, but none have worked out well atleast till now

In T186742#4081478, @EBernhardson

I can certainly run a new aggregation of the data if we find a better way. Some script to convert scipy.sparse to the tensorflow format should be relatively short to evaluate that before hand.

In T186742#4081471, @EBernhardson wrote:

Due to how much SERP_SIZE changes the dimensions, i'm thinking that we need to stay pretty much at 10. For new data i was referring to the amount of time we aggregate the click data over. Within wikimedia we only keep the input data (click logs, queries, etc) necessary for the machine learned ranking for 90 days. What i was thinking about here is that if we train a model against 90 days of aggregated data, save it somewhere, and then later want to run new data against it, does that new data need to be for the same number of days? Many of the individual values of the feature vector are counts of clicks over the aggregation period so the scale of data might make a difference. I'm not sure this is the most important thing to figure out though, setting the aggregation period and keeping it constant is relatively easy.

Currently model support variable length of the input (in terms of days), Also we can save the trained model to work as a backup in case, we ever have to restart the whole model.

@EBernhardson for 3d sparse matrices, I looked for multiple posts and found that tensorflow has sparse input and it supports N dimensions, so we can pipeline the input to tensorflow sparse directly.

In T186742#4078432, @ansh103 wrote:

In T186742#4072583, @Kdhingra2210 wrote:

Also, for application over the different size of datasets, without retraining can be done using dynamic lstm cells (preferably gru), have worked over dynamic time series analysis and it provide quite better outputs but need relatively more training

Considering time would be an issue more or less, we would require to analyze which one would be better for this dataset. Here[1] it was concluded that both gating units perform better than traditional RNN which is pretty obvious. But the further conclusion said how LSTM performed better in one dataset whereas GRU performed better in another one. @EBernhardson could direct us from here.

[1] https://arxiv.org/pdf/1412.3555v1.pdf

@EBernhardson I ported this model to barebone tensorflow, though it doesn't support gridsearch or improve timedistributed dense functions, it does support dynamic indexing over the time dimension.

In T186742#4066359, @EBernhardson wrote:

In T186742#4064169, @Kdhingra2210 wrote:

@EBernhardson Can you share what results you got with that algorithm because I was getting fairly good confusion matrix.

The confusion matrices in your link look relatively similar to what I was getting. The confusion matrices are good for a quick look at how it's doing, but i think they sometimes make the results look better than they are. Precision, recall and F1 scores might be a reasonable way to look at the data. One of the strongest models in your link looks to be (roughly):

val_loss: 0.1629 - val_acc: 0.9404
[[255628   6692]
 [  6338  20592]]

This says in the ground truth there were 20592 + 6692 = 27284 clicks. The model predicted 26930 clicks.
This gives a few metrics:

• precision: 20592 / 26940 = 0.764
• recall: 20592 / 27284 = 0.754
• We could calculate F[0.5, 1, 2] from that, but the numbers are pretty close so call it 0.76.

This is certainly not a bad baseline, but seems like there is plenty of room for improvement.

I thought it to be in the opposite, that's why I asked about confusion matrix of yours,

@EBernhardson Can you share what results you got with that algorithm because I was getting fairly good confusion matrix.

In T186742#4019062, @EBernhardson wrote:

@Kdhingra2210 Great! I'm not really sure what the best entry point is. Without review on the data collection we have to mostly hope the aggregation I wrote is correct. I still put up a sample of the available click data on the public archive[1]. The data is perhaps deceptively small as it compresses very well. Contained there are around 900k observations (search sessions) . The X matrix has an overall shape of (899310, 11, 11265) giving ~111 billion individual values. I wrote a README file that hopefully explains the data format. The data format can also be reviewed in the gerrit link above which contains the code used to collect these sparse matrices.

[1] https://analytics.wikimedia.org/datasets/discovery/ncm-agg-clicks/

Hi @EBernhardson, I looked up this project from GSoC2018 and am interested in this, I have gone through the research paper and would like to work on this. I have already worked on time series models.