Some ideas we've kicked around:

• One hot encoding of popular templates/categories
• # of tokens in query
• # of tokens in title
• size of the document
• derivative features, such as wp10s # wikilinks / # content chars
• wp10 scores from ores
• Features used by wp10 in ores:

>>> from wikiclass.feature_lists import enwiki
>>> import pprint
>>> pprint.pprint(enwiki.wp10)
[<feature.wikitext.revision.chars>,
 <feature.wikitext.revision.content_chars>,
 <feature.wikitext.revision.ref_tags>,
 <feature.(wikitext.revision.ref_tags / max(wikitext.revision.content_chars, 1))>,
 <feature.wikitext.revision.wikilinks>,
 <feature.(wikitext.revision.wikilinks / max(wikitext.revision.content_chars, 1))>,
 <feature.wikitext.revision.external_links>,
 <feature.(wikitext.revision.external_links / max(wikitext.revision.content_chars, 1))>,
 <feature.wikitext.revision.headings_by_level(2)>,
 <feature.(wikitext.revision.headings_by_level(2) / max(wikitext.revision.content_chars, 1))>,
 <feature.wikitext.revision.headings_by_level(3)>,
 <feature.(wikitext.revision.headings_by_level(3) / max(wikitext.revision.content_chars, 1))>,
 <feature.enwiki.revision.image_links>,
 <feature.(enwiki.revision.image_links / max(wikitext.revision.content_chars, 1))>,
 <feature.enwiki.revision.category_links>,
 <feature.(enwiki.revision.category_links / max(wikitext.revision.content_chars, 1))>,
 <feature.enwiki.revision.cite_templates>,
 <feature.(enwiki.revision.cite_templates / max(wikitext.revision.content_chars, 1))>,
 <feature.(enwiki.revision.cite_templates / max(wikitext.revision.ref_tags, 1))>,
 <feature.max((wikitext.revision.ref_tags - enwiki.revision.cite_templates), 0)>,
 <feature.(max((wikitext.revision.ref_tags - enwiki.revision.cite_templates), 0) / max(wikitext.revision.content_chars, 1))>,
 <feature.enwiki.revision.non_cite_templates>,
 <feature.(enwiki.revision.non_cite_templates / max(wikitext.revision.content_chars, 1))>,
 <feature.enwiki.revision.infobox_templates>,
 <feature.(enwiki.revision.cn_templates + 1)>,
 <feature.(enwiki.revision.cn_templates / max(wikitext.revision.content_chars, 1))>,
 <feature.(enwiki.revision.cn_templates + 1)>,
 <feature.(enwiki.revision.cn_templates / max(wikitext.revision.content_chars, 1))>,
 <feature.enwiki.main_article_templates>,
 <feature.(enwiki.main_article_templates / max(wikitext.revision.content_chars, 1))>,
 <feature.(english.stemmed.revision.stems_length / max(wikitext.revision.content_chars, 1))>]

Some high level ideas to think about from the Yahoo LTR challange (http://www.jmlr.org/proceedings/papers/v14/chapelle11a/chapelle11a.pdf?WT.mc_id=Blog_MachLearn_General_DI):

Web graph

This type of features tries to determine the quality or the popularity of a
document based on its connectivity in the web graph. Simple features are functions
of the number of inlinks and outlinks while more complex ones involve some kind of
propagation on the graph. A famous example is PageRank (Page et al., 1999). Other
features include distance or propagation of a score from known good or bad documents
(Gy¨ongyi et al., 2004; Joshi et al., 2007).

Document statistics

These features compute some basic statistics of the document such
as the number of words in various fields. This category also includes characteristics
of the url, for instance the number of slashes.

Document classifier

Various classifiers are applied to the document, such as spam, adult,
language, main topic, quality, type of page (e.g. navigational destination vs informational).
In case of a binary classifier, the feature value is the real-valued output of the
classifier. In case of multiples classes, there is one feature per class.

Query

Features which help in characterizing the query type: number of terms, frequency
of the query and of its terms, click-through rate of the query. There are also result set
features, that are computed as an average of other features over the top documents
retrieved by a previous ranking function. For example, the average adult score of the
top documents retrieved for a query is a good indicator of whether the query is an
adult one or not.

Text match

The most important type of features is of course the textual similarity between
the query and the document; this is the largest category of features. The basic
features are computed from different sections of the document (title, body, abstract,
keywords) as well as from the anchor text and the url. These features are then aggregated
to form new composite features. The match score can be as simple as a count or
can be more complex such as BM25 (Robertson and Zaragoza, 2009). Counts can be
the number of occurrences in the document, the number of missing query terms or the
number of extra terms (i.e. not in the query). Some basic features are defined over
the query terms, while some others are arithmetic functions (min, max, or average)
of them. Finally, there are also proximity features which try to quantify how far in
the document are the query terms (the closer the better) (Metzler and Croft, 2005).

Topical matching

This type of feature tries to go beyond similarity at the word level and
compute similarity at the topic level. This can for instance been done by classifying
both the query and the document in a large topical taxonomy. In the context of
contextual advertising, details can be found in (Broder et al., 2007).

Click

These features try to incorporate the user feedback, most importantly the clicked
results (Agichtein et al., 2006). They are derived either from the search or the toolbar
logs. For a given query and document, different click probabilities can be computed:
probability of click, first click, last click, long dwell time click or only click. Also of
interest is the probability of skip (not clicked, but a document below is). If the given
query is rare, these clicks features can be computed using similar, but more frequent
queries. The average dwell time can be used as an indication of the landing page
quality. The geographic similarity of the users clicking on a page is a useful feature to
determine its localness. Finally, for a given host, the entropy of the click distribution
over queries is an indication of its specificity.

External references

For certain documents, some meta-information, such as Delicious
tags, is available and can be use to refine the text matching features. Also documents
from specific domains have additional information which can be used to evaluate
the quality of the page: for instance, the rating of an answer in Yahoo! Answers
documents.

Time

For time sensitive queries, the freshness of a page is important. There are several
features which measure the age of a document as well as the one of its inlinks and
outlinks. More information on such features can be found in (Dong et al., 2010,
Section 5.1).

Microsoft also has a list of 136 features, which are a subset of those used in bing and are "widely used in the research community." : https://www.microsoft.com/en-us/research/project/mslr/

These 136 features are actually a much smaller number of features, but applied multiple times. For example the IDF of the query terms in the body, anchor, title, url and whole document make up 5 separate features, but all are IDF. We would likely do similar, but with our fields: title, heading, opening text, etc. The deduplicated list follows, everything after the - is my own guesses about what these features mean, or data from the site describing the feature.

1. covered query term number - perhaps this is the number of terms from query that are matched by the document?
2. covered query term ratio - the percentage of terms from query that are matched by the document?
3. stream length - size in bytes/codepoints/something?
4. IDF
5. Sum of TF
6. Min of TF
7. Max of TF
8. Mean of TF
9. Variance of TF
10. sum of stream length normalized TF
11. min of stream length normalized TF
12. max of stream length normalized TF
13. mean of stream length normalized TF
14. variance of stream length normalized TF
15. sum of tf*idf
16. min of tf*idf
17. max of tf*idf
18. mean of tf*idf
19. variance of tf*idf
20. boolean model - ??? no clue
21. vector space model - This may be word2vec style vector space embeddings? not sure
22. BM25
23. LMIR.ABS - Site says "Language model approach for information retrieval (IR) with absolute discounting smoothing"
24. LMIR.DIR - Site says "Language model approach for IR with Bayesian smoothing using Dirichlet priors"
25. LMIR.JM - Site says "Language model approach for IR with Jelinek-Mercer smoothing"
26. Number of slash in URL - probably similar to a subpage depth in our case
27. Length of URL - Probably not relevant to use, the "stream length" of title is same
28. Inlink number
29. Outlink number
30. PageRank
31. SiteRank - Site says "Site level PageRank".
32. QualityScore - Site says "The quality score of a web page. The score is outputted by a web page quality classifier."
33. QualityScore2 - Site says "The quality score of a web page. The score is outputted by a web page quality classifier, which measures the badness of a web page."
34. Query-url click count - Site says "The click count of a query-url pair at a search engine in a period"
35. url click count - Site says " The click count of a url aggregated from user browsing data in a period"
36. url dwell time - Site says "The average dwell time of a url aggregated from user browsing data in a period"

@dcausse one thing to think about is how we could efficientlly extract some of the data above from lucene for use as feature vectors. Likely we wouldn't want to have a separate query for each of the sum/min/max/mean/variance of the TF (and again for a length-normalized version). It seems like all 10 features could come out of a single query some way or another, as they all have basically the same input data: The TF for each term in the query. These queries would still have to be duplicated against the set of fields we want to use, such as title, headings, opening_text, body, etc.

I was thinking that some of the generic text features of the query might be useful. Mikhail has an interesting list and some data in prevalence in his Zero To Hero report. Presence of a question mark, presence of quotes, and the presence of advanced syntax seem interesting—though as we discussed, these may be too low frequency to be useful.

Another one that just came to mind would be the presence of Unicode character classes (e.g., writing systems, very roughly) might be interesting. I'd expect that for any given wiki, it'd come down to "the character set of this wiki" and "not the character set of this wiki", but the generic features would be easy to implement.

I had the idea to take a list of popular categories and templates, train a model with them, and then use the results of that model to try and decide what to use.

I took all the categories that are on more than 1k pages, which amounts to 3166 categories, then trained a model with 1.2M samples using those 3166 categories as binary features along with our default set of features. This ends up being pretty expensive to train, the memory usage per-fold went from 186MB for the 9 feature model to 26GB for the 3175 feature model. Because of this instead of doing the full hyperparameter/cross validation pipeline i trained a single model using the same parameters as the best model previously trained with the same 1.2M sample and 9 features. This ended up using 66 categories in the final model. 45 of the features were used at least twice, and 34 of them at least three times. The data below only considers features with weight >= 3.

The tree statistics in graph form. Note that I had to put all the values on a log scale, otherwise the variations are incredibly hard to see due to the difference between largest and smallest values.

And same for templates:

Need to decide what to do with these exactly. For english i can certainly manually review the lists, take everything that seems sane, and build a combined model with all of them and re-review what happens. I think we also might need T162711 because talk page categories are probably as interesting as the main page categories. They will probably also change things.

There is also non-exact matches. Especially for talk page categories we might want matches like 'Grade A' or some such, rather than strictly matching specific categories to generalize things. Matching specific wikiproject tags regardless of quality might also be useful. I'm not sure how to generalize this all beyond english though, our best bet might be to do similar to ORES and work with communities to curate lists of sane things.

Thanks for all the data. (It imports into Libre Office very easily.) Those max/avg/min swings are huge—orders of magnitude!

Whether or not you should filter for things that seem sane isn't clear to me. Machine learning often doesn't seem sane. ;) What really matters is whether things are sufficiently stable to be predictive. I think the refresh period for the model might be more important. Things like category_All_Wikipedia_articles_written_in_American_English doesn't seem to have any theoretical value, but it might have real-world value, at least in the short term, because whoever is tagging that category is focusing on high-quality articles first, or something. If the model is refreshed often enough (for some value of "often enough") then the reason something is predictive doesn't matter too much, because the newer model will catch up with any shifts in predictive value. That also resolves the problem of dealing with languages & cultures we don't know.

I definitely agree that the talk page categories will be very valuable!

While I agree that matching on parts of categories, like "A Class" or "Good Articles" could boost the signal from the larger quality categories to the smaller ones, there's the problem of parsing them. There could be problems like "Good Articles for Deletion" (terrible category name, but you get the idea) in English, and there's the problem of languages we don't know (and some of those could have inflections of the words in titles that make parsing them even more annoying). So, maybe, it would make sense to try them without parsing and see how it goes. "History good articles" might be a significantly more valuable indicator than "Pokeman good articles". You could also try just the most obvious and easiest-to-extract non-exact matches to see how much they help over the full category names.

All very fair. I haven't automated the above selection of categories/templates, but i suppose i could. It's in code but it's code i pasted into a REPL as opposed to building into mjolnir directly.
. iIwas thinking though that perhaps some of these things that were chosen are basically proxies for some more specific signal. For example:

• category_Articles_with_unsourced_statements_from_[March|April|July]_2017 - Maybe this is a proxy for age of articles, or low quality new articles? Might be worth trying to use age as a feature directly and see if xgboost finds less value in these particular categories. We don't currently store the age of the oldest revision in elasticsearch but i think it would be pretty easy to inject.

• Template:Navbox - Almost all pages have this, pages that don't are perhaps lower quality pages? Maybe using ores wp10 scores would make this kind of proxy for page quality less useful?

• Module:WPMILHIST_Infobox_style - Maybe this is a proxy for general 'military history' pages? Not sure what better signal we could come up with though, this is probably pretty reasonable.

• category_Wikipedia_indefinitely_semi-protected_pages - This almost seems like a proxy for popularity or quality? Generally something is semi-protected due to spam which probably happens on either controversial or popular articles.

As with anything though I'm not sure that guessing at what the actual signal behind the choice is and trying to more directly capture that is going to be worthwhile. The above was relatively easy to do with just a computer stuffing numbers into a box and poping out an answer, whereas each individual signal will probably take a day or two of engineering time to do an initial validation on to see if it's of any use. And of course there is plenty of opportunity to be wrong about what signal it was trying to capture. The results of testing a signal and finding it to be useless is probably still useful information, but not as usefull as a new feature that works.

I think it's a great idea to look at English templates and categories like you have and try to get at some idea of a more generic signal. Article age and ORES score both seem like good ideas—but as you say, it may or may not be worth the engineering time to extract the info and test its utility in every conceivable case. OTOH, something like article age is universal across all wikis (that is, it's available and might be meaningful, unlike ORES scores, which are probably always meaningful, but not available for all wikis).

Some evaluations of current QI features, and the addition of wp10 and page_created_ts:

Surprisingly while both of our existing QI features are "good", the new features for evaluation have much smaller or no impact. wp10 doesn't look to be a useful feature either as a weighted sum or a one-hot encoding of article classes. The page created date has some small utility, but relatively little. Adding it to our elasticsearch docs would be relatively easy if we want to add it.

One more (or really two) new features, adding a token count of the query string with both the text_search and plain_search analyzers:

Will train the model with them individually to see if both are necessary.

Some other random ideas for features:

• Prefix match against title/redirect. It seems likely that a prefix match would be more important than a general match against title. May want to do this per-term? As in if any of the terms in the search query match the first term of the title.
• Age of last edit to page
• Age of last significant content change to page (so, disregarding edits to fix typos, or bots that swap a link out to point at internet archive, etc)
• Does the page contain multimedia, ratio of images to page length may be a quality signal (but wp10 turned out to not be useful, so maybe not as important or overshadowed by popularity?)
• Estimated reading level of a page? Seems it would somehow need to be combined with an estimated reading level of the query as well
• Number of Section? Average section length?
• Could category matches be influenced by the size of the category? Matching a very narrow category might be a better match than a large category.
• Search popularity of page (% of search clicks that go to that page)? Might provide a slightly different signal than overall popularity
• Backlink anchor text could be useful to index. It's possible that the choices of words people use to link to an article are different from the title/redirects that already exist.
• popularity velocity, or change over time. A page with increasing popularity may be more important than one with decreasing popularity. Or it might all be noise.
• backlink co-occurance, or the words that appear nearby a link to the article, may provide useful context.
• existence of a talk page for an article may be a quality signal (but again, wp10...hard to say)
• Page dwell time, or how long an "average" reading session on the page is.
• Does the page have an infobox?

Perhaps also interesting are the ranking features available in the newly released vespa, which yahoo used for many internal ranking and recommendation tasks: http://docs.vespa.ai/documentation/reference/rank-features.html

Another idea for a feature—some similarity measure between the search term and the matching term in a document. Though after talking to @dcausse it sounds too expensive because there's no good way to map matched terms to specific query terms, and even pulling out the matched terms is a pain.

But it could help with unexpected stemmer bugs—for example, the Polish stemmer is statistical and has a few really weird errors. It might penalize some stemmings of related but dissimilar words (e.g., the English stemmer stems Dutch and Holland as dutch, and the Ukrainian stemmer groups жене and гнали)—but those are very rare.