To generate the original list I used:

1. download.sh257 BDownload
   to download the titles lists of the various wikis
2. shuffle.sh490 BDownload
   to shuffle the lists of titles and extract the top N
3. jsonify.js253 BDownload
   to turn the one-title-per-line files into a JSON file suitable for importing with Parsoid/tests/server/importJson.js

I still have the original shuffled files, so if you wanted to tweak our title set without losing all consistency with previous results I could extract the top M (for M != the original N) lines from those files.

Alternatively, we could use the "top 1000" articles, based on hit count. https://meta.wikimedia.org/wiki/Datasets lists a number of different sources for traffic data, but so far every one I've tried has been a dead link. :( I was pretty sure there was a WMF-generated page hit statistics data source somewhere, though.

gwicke suggested that an alternative to picking a random sample is to pick the most recently edited N pages for each wiki and that we can get those from /a/mw-log/runJobs.log on fluorine

But, so as not to skew the list towards the popular pages based on current events, we should probably pick half the pages from the recent edit list and pick the other half using random sampling.

Looking at the jobs logs on fluorine is not going to work since it is skewed by bot edits. Better to get the data from the RC stream via the API.

I updated the rt-testing database with a new set of titles.

• 28 wikis and 2 wiktionaries are now represented

• 30K titles from enwiki, 10K each from dewiki nlwiki frwiki itwiki ruwiki eswiki ... and progressively smaller from the rest -- 160K total (see full list below)

• 70% randomly chosen from the latest dump of titles; 30% from the recent changes stream -- gabriel made a pertinent observation that we should use pages from the RC stream since they are more representative of pages being edited. However, I didn't want to pick everything from the RC stream since we still want to support the full mix of pages on the wikis -- not just those being actively edited.

• I've restarted rt-testing to init the base data before we merge any new patches.

 30000 enwiki.all_titles.txt
 10000 dewiki.all_titles.txt
 10000 nlwiki.all_titles.txt
 10000 ruwiki.all_titles.txt
 10000 frwiki.all_titles.txt
 10000 eswiki.all_titles.txt
 10000 itwiki.all_titles.txt
  8000 svwiki.all_titles.txt
  8000 plwiki.all_titles.txt
  8000 jawiki.all_titles.txt
  7000 arwiki.all_titles.txt
  7000 kowiki.all_titles.txt
  7000 hiwiki.all_titles.txt
  7000 hewiki.all_titles.txt
  5000 zhwiki.all_titles.txt
  1000 enwiktionary.all_titles.txt
  1000 frwiktionary.all_titles.txt
  1000 mznwiki.all_titles.txt
  1000 uzwiki.all_titles.txt
  1000 iswiki.all_titles.txt
  1000 hywiki.all_titles.txt
  1000 kawiki.all_titles.txt
  1000 ukwiki.all_titles.txt
   956 pnbwiki.all_titles.txt
   948 ckbwiki.all_titles.txt
   859 cvwiki.all_titles.txt
   808 kaawiki.all_titles.txt
   715 lnwiki.all_titles.txt
   707 cuwiki.all_titles.txt
   705 lbewiki.all_titles.txt
160698 total

https://gerrit.wikimedia.org/r/#/c/225328/ has the scripts used to generate the new db -- useful for the next time we need to do this.

The new base rt-testing accuracy with the new db is 99.81% semantic, 74.2% syntactic. With the old db, the corresponding numbers were 99.95% and 85.2%. One of the reason for the bigger drop in syntactic accuracy is because over 2 years, the old db seems to have accumulated a lot of redirect pages which are trivial pages that rt perfectly. I expect the same thing to happen with this new db over time.