possibly write a new elasticsearch query that can combine and use in its scoring formula the number of tokens of its input query

Actually, we can do this with the ltr explore query today.

{
  "query": {
    "match_explorer": {
      "query": { "match": { "text.plain": { "query": "example words to count" } } },
      "type": "unique_terms_count"
    }
  },
  "size": 10,
  "_source": false
}

Another thing to consider would be T258055: [L] [SPIKE] Investigate traversing entities tree to include more entities with more detail & feedback is very much welcome (scroll to last post for findings)

@dcausse @matthiasmullie can we identify which tickets/experiments need to be done before the search team can get back to this task? I'd like to define a point when it's safe to make the handover.

The relevant tickets are already in the ticket description; I can't think of any others not already listed.

In addition to the previous recommendations:

• T252692 (search syntax and features) would benefit from completing the refactoring started in T185108. The MediaSearch query builder will be able to use parts of this work (the cirrus query AST) but will have to implement its own query components, depending on how both tasks advance there will be consolidation work to be done.
• T258055 (traversing the wikidata graph) is still being evaluated and its scope is relatively large so I don't think it's worth spending too much time on possible solutions but I believe this work might be integrated in a component that extract a subset of wikidata (possibly denormalized).

Summarizing what has been said so far I see:

• T268648: MediaSearch should use a dedicated service/query for doing its concept-lookup instead of the wikidata search API. This should encompass the work regarding rewriting (T258053 & T258052). But also the traversal of the wikidata graph (explored in T258055) even though this one is more complicated. Overall everything that requires joining wikidata or exploring the structure of the wikidata graph should be doable by constructing dedicated datasets offline.
• Work on refactoring the cirrus parsing can happen concurrently/later and can be re-prioritized depending on the possible blocking points that might be encountered on T252692. Supporting more complex syntax in the MediaSearch query builder will probably involve writing code that would have been better placed in CirrusSearch but we agreed to consolidate this when resuming work on refactoring the query building logic (T185108).
• If I understood correctly there are currently no plans to deploy MediaSearch on wikis other than commons and thus there is no longer the need to move this code out of the MediaInfo extension to break the wikibase dependencies (language fallbacks) therefor I don't think it's worth creating a ticket for this yet.
• Ranking: I think a lot of exploratory work has been done already by the SD team (e.g. T258054 and T262522). At this point I don't have specific recommendations to make except that the tooling around relevancy should be adapted to the specifics of MediaSearch and created T268653.
• Feature engineering: I think the features that the SD team wanted to explore have been captured in the tickets referenced in the description. There might be other features interesting for MediaSearch but without a better understanding of how to assess the quality of the results I think it is too early to discuss about this.

Thanks for this summary @dcausse!

In T260251#6648172, @dcausse wrote:

• T268648: MediaSearch should use a dedicated service/query for doing its concept-lookup instead of the wikidata search API. This should encompass the work regarding rewriting (T258053 & T258052). But also the traversal of the wikidata graph (explored in T258055) even though this one is more complicated. Overall everything that requires joining wikidata or exploring the structure of the wikidata graph should be doable by constructing dedicated datasets offline.

Does this mean that we should wait to do T258053 and T258052 until T268648 is complete? Or should we continue work on those tickets and then refactor once T268648 is complete?

• Work on refactoring the cirrus parsing can happen concurrently/later and can be re-prioritized depending on the possible blocking points that might be encountered on T252692. Supporting more complex syntax in the MediaSearch query builder will probably involve writing code that would have been better placed in CirrusSearch but we agreed to consolidate this when resuming work on refactoring the query building logic (T185108).

Should we put a note in T185108 about this so we don't forget whenever we return to it?

• If I understood correctly there are currently no plans to deploy MediaSearch on wikis other than commons and thus there is no longer the need to move this code out of the MediaInfo extension to break the wikibase dependencies (language fallbacks) therefor I don't think it's worth creating a ticket for this yet.

This is correct. One day we might expand beyond Commons to search the fair use images stored on other wikis, but there are no plans for this yet and I think we can create a ticket if that becomes a use case we need to support.

In T260251#6649364, @CBogen wrote:

Does this mean that we should wait to do T258053 and T258052 until T268648 is complete? Or should we continue work on those tickets and then refactor once T268648 is complete?

The features you planned to explore/implement should not be blocked on T268648. We should just coordinate while working on these tickets (i.e. reviews) so that we can discuss & adapt.

• Work on refactoring the cirrus parsing can happen concurrently/later and can be re-prioritized depending on the possible blocking points that might be encountered on T252692. Supporting more complex syntax in the MediaSearch query builder will probably involve writing code that would have been better placed in CirrusSearch but we agreed to consolidate this when resuming work on refactoring the query building logic (T185108).

Should we put a note in T185108 about this so we don't forget whenever we return to it?

Good idea, I've added a note.