Currently the idea is to use metric, based on the Levenstein distance as distance between texts (examples in this notebook or this notebook for bigger cases). Current idea of closeness detection algorithm look like this like this:

These modifications are considered:

1. Normalised Levenstein distance might work way worse on "real" input data. In this case further analysis is considered, most likely including AST usage and comparison. They can be run in the background.
2. Calculating Levenstein distance requires walking through all the possible pairs, so it's quite costly, and speed is our main priority. Because of that it might be better to split initial input group into subgroups and run the code multiple times, shuffling subgroups between runs. Size of the group should be constant (as it's the thing affecting computational time the most), so the amount of runs should be calculated. Also, it's most likely possible to run different subgroups in different threads to speed everything up a lot.

What's left unanswered:

1. How to store predicted group belonging info and real (approved by users) group belonging info? Scripts table can be modified once more, but it is already huge, so maybe it would be better to create simple new tables with columns like pair_id - page_id1 - wiki1 - page_id2 - wiki2 - group_id (or predicted_group_id).
2. There are modules with the same titles across diffeent wikis, and it's usually OK to group them. But should we look into similar titles too, not only the same ones?

@LostEnchanter @gengh @dr0ptp4kt
I've attached my analysis and procedures so far. Some more tasks are added as todo. Tuning takes some time due to longer clustering time but good thing there's not much to tune.

@dr0ptp4kt @LostEnchanter @gengh

I've updated the pdf to include noise removal and tuning analysis. This should be it for now regarding similarity analysis. Feel free to send feedback.