Comparison Tool for Relevance Lab
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	TJones
	Oct 27 2015, 11:10 PM

Description

Depends on T116869, T116870, and T116871, but can start before either is finished by using current output or stubs for the various comppnents under development.

This is the top level Relevance Lab command line interface. As such, it's pretty complicated. It may make sense to specify a configuration file rather than try to cram all this into command line arguments, at the developer's discretion.

Inputs:

Comparison: specify a name, and two query runs:
- First query run: specify a name for the run, a query file, and optionally runSearchConf.json file
- Second query run: specify a name for the run, a query file, and optionally runSearchConf.json file
- (Should the relevancelab/ directory be in a canonical place, or specified by config file or command line? I suggest config file, but leave it to the developer's discretion)

Processing/Output:

make a reasonable directory name for the comparison and query runs based on their names (e.g., remove all non-alphanumerics)

set up an SSH tunnel between your local vagrant instance and the hypothesis testing cluster in labs
- (is this known or specified by command line argument or config file?)

run runSearch.php with the first query file and the first runSearchConf.json
- copy the runSearchConf.json to relevancelab/queries/<query1_dir>
- store the output of runSearch.php (one line of JSON for each query) in relevancelab/queries/<query1_dir>/results

run runSearch.php with the second query file and the second runSearchConf.json
- copy the runSearchConf.json to relevancelab/queries/<query2_dir>
- store the output of runSearch.php (one line of JSON for each query) in relevancelab/queries/<query2_dir>/results

run the diff tool (see T116870), with
- the two files relevancelab/queries/<query1_dir>/results and
  - relevancelab/queries/<query2_dir>/results
- output directory relevancelab/comparisons/<comparison_dir>/diffs

run the metrics/report tool (see T116871), with
- name "comparison name"
- the two files relevancelab/queries/<query1_dir>/results and
  - relevancelab/queries/<query2_dir>/results
- output directory relevancelab/comparisons/<comparison_dir>

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Initial version of relevance lab runner	mediawiki/extensions/CirrusSearch	master	+128 -0
	Relevancy runner in python	wikimedia/discovery/relevancylab	master	+103 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Smalyshev	T116872 Comparison Tool for Relevance Lab
Resolved	TJones	T116870 JSON Diff tool for Relevance Lab
Resolved	TJones	T116871 Metrics/Report tool for Relevance Lab
Resolved	EBernhardson	T116869 Enhance runSearch.php to output info needed for Relevance Lab

Event Timeline

TJones created this task.Oct 27 2015, 11:10 PM

TJones raised the priority of this task from to High.

TJones updated the task description. (Show Details)

TJones added a project: Discovery-Search (Current work).

TJones added subscribers: TJones, dcausse, EBernhardson, Smalyshev.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 27 2015, 11:10 PM

TJones added subtasks: T116869: Enhance runSearch.php to output info needed for Relevance Lab, T116870: JSON Diff tool for Relevance Lab, T116871: Metrics/Report tool for Relevance Lab.Oct 27 2015, 11:11 PM

@Smalyshev this would probably be good to prioritize over our hadoop work, we have a tentative goal to have a code freeze in december, so we need to focus on the language goals for this month and shift the hadoop work into december. Sorry to keep switching things up on you!

Smalyshev claimed this task.Nov 3 2015, 5:54 PM

Smalyshev set Security to None.

Proposal: the runner gets as input a file like this:

[settings]
labHost = suggesty.eqiad.wmflabs
searchCommand = sudo -u vagrant mwscript extensions/CirrusSearch/maintenance/runSearch.php 
workDir = /tmp/relevance/
jsonDiffTool = jsondiff.py
metricTool = metric

[test1]
name = Test 1
queries = test1.q

[test2]
name = Test 2
queries = test2.q
config = test2.json

Change 250909 had a related patch set uploaded (by Smalyshev):
Initial version of relevance lab runner

https://gerrit.wikimedia.org/r/250909

gerritbot added a project: Patch-For-Review.Nov 4 2015, 6:11 AM

EBernhardson moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.Nov 4 2015, 2:31 PM

I like the .ini config file! Otherwise the command line is untenably complex.

Change 251177 had a related patch set uploaded (by Smalyshev):
Relevancy runner in python

https://gerrit.wikimedia.org/r/251177

EBernhardson moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Nov 5 2015, 5:49 PM

Change 251177 merged by Smalyshev:
Relevancy runner in python

https://gerrit.wikimedia.org/r/251177

Smalyshev closed this task as Resolved.Nov 5 2015, 11:12 PM

Smalyshev moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.