For reference, we need more of these!

https://phabricator.wikimedia.org/T212129#4948690

It would be helpful if @jijiki were able to share their tools or code (if any) that they did for that previous analysis.

One algorithm for this might be the following:

• Make a hashtable for the pattern counts, mapping a pattern string to an integer

• For each line:
  • pattern = line
  • Replace substrings in pattern that match the code for of one of our wikis (like "en_wp") with "(WIKI)"
  • Replace everything after the last colon in pattern with "(ID)"
  • (Other transforms here)
  • Increment the count for this pattern in the big hashtable (or set to 1 if not exists)

• Dump comma-separated values (CSV) of hashtable to stdout as "count,pattern"

We can stop adding transforms when the resulting dumped CSV is less than 1000 lines long. Any dupes after that I can deal with manually.