Page MenuHomePhabricator
Create Task
Maniphest T273847

Create a elasticsearch/kibana index with queries to allow query completion candidate research
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

As a search developer I want to be able to analyze queries so that I can find a viable query completion candidates.

Since relforge is available through analytics network, we should be able to ship queries into kibana and analyze them.

AC:

  • search events from backend logs and frontend logs are joined via request_id
  • queries are normalized (to be determine how extensively)
  • joined events are available for analysis in Kibana

Details

SubjectRepoBranchLines +/-
Add code coverage to Sonarqube reportwikimedia/discovery/analyticsmaster+7 -2
Deploy the index setting filewikimedia/discovery/analyticsmaster+5 -2
Add elastic-template handlingwikimedia/discovery/analyticsmaster+1 -0
Set correct start date for export_queries_to_relforgewikimedia/discovery/analyticsmaster+3 -3
Add export_queries_to_relforge dagwikimedia/discovery/analyticsmaster+1 K -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Zbyszko created this task.Feb 4 2021, 8:01 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 4 2021, 8:01 AM
• Zbyszko added a parent task: T250436: [Epic] Query Completion.Feb 4 2021, 8:01 AM
• Zbyszko updated the task description. (Show Details)Feb 4 2021, 8:04 AM
• Zbyszko set the point value for this task to 8.
• Zbyszko moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.Feb 5 2021, 11:37 AM
• Zbyszko claimed this task.Feb 15 2021, 3:06 PM
• Zbyszko mentioned this in T274794: Establish a query result score from search satisfaction .Feb 15 2021, 3:09 PM
gerritbot added a comment.Feb 19 2021, 9:59 AM

Change 665310 had a related patch set uploaded (by ZPapierski; owner: ZPapierski):
[wikimedia/discovery/analytics@master] Add export_queries_to_relforge dag

https://gerrit.wikimedia.org/r/665310

gerritbot added a project: Patch-For-Review.Feb 19 2021, 9:59 AM
• Zbyszko moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Feb 23 2021, 9:14 AM
EBernhardson moved this task from Needs review to To Be Deployed on the Discovery-Search (Current work) board.Mar 4 2021, 8:01 PM
gerritbot added a comment.Mar 11 2021, 8:38 PM

Change 665310 merged by jenkins-bot:
[wikimedia/discovery/analytics@master] Add export_queries_to_relforge dag

https://gerrit.wikimedia.org/r/665310

Stashbot added a comment.Mar 11 2021, 8:44 PM

Mentioned in SAL (#wikimedia-operations) [2021-03-11T20:44:48Z] <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@cc478d4]: T273847 export queries to relforge dag deployment

Stashbot added a comment.Mar 11 2021, 8:46 PM

Mentioned in SAL (#wikimedia-operations) [2021-03-11T20:46:57Z] <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@cc478d4]: T273847 export queries to relforge dag deployment (duration: 02m 09s)

gerritbot added a comment.Mar 11 2021, 9:02 PM

Change 670942 had a related patch set uploaded (by ZPapierski; owner: ZPapierski):
[wikimedia/discovery/analytics@master] Set correct start date for export_queries_to_relforge

https://gerrit.wikimedia.org/r/670942

gerritbot added a comment.Mar 11 2021, 9:10 PM

Change 670942 merged by ZPapierski:
[wikimedia/discovery/analytics@master] Set correct start date for export_queries_to_relforge

https://gerrit.wikimedia.org/r/670942

Stashbot added a comment.Mar 11 2021, 9:12 PM

Mentioned in SAL (#wikimedia-operations) [2021-03-11T21:12:04Z] <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@3810277]: T273847 export queries to relforge dag deployment - correct start date

Stashbot added a comment.Mar 11 2021, 9:13 PM

Mentioned in SAL (#wikimedia-operations) [2021-03-11T21:13:57Z] <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@3810277]: T273847 export queries to relforge dag deployment - correct start date (duration: 01m 53s)

• Zbyszko mentioned this in rWDANe36cd6ae8f2c: Add export_queries_to_relforge dag.Mar 11 2021, 9:19 PM
• Zbyszko mentioned this in rWDAN38102777c7d2: Set correct start date for export_queries_to_relforge.
gerritbot added a comment.Mar 11 2021, 9:26 PM

Change 670955 had a related patch set uploaded (by ZPapierski; owner: ZPapierski):
[wikimedia/discovery/analytics@master] Add elastic-template handling

https://gerrit.wikimedia.org/r/670955

gerritbot added a comment.Mar 12 2021, 8:58 AM

Change 670955 merged by ZPapierski:
[wikimedia/discovery/analytics@master] Add elastic-template handling

https://gerrit.wikimedia.org/r/670955

Stashbot added a comment.Mar 12 2021, 8:59 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-12T08:59:33Z] <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling

• Zbyszko mentioned this in rWDAN9a408b260676: Add elastic-template handling.Mar 12 2021, 8:59 AM
Stashbot added a comment.Mar 12 2021, 8:59 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-12T08:59:43Z] <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling (duration: 00m 10s)

Stashbot added a comment.Mar 12 2021, 9:00 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-12T09:00:34Z] <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling

Stashbot added a comment.Mar 12 2021, 9:00 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-12T09:00:44Z] <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling (duration: 00m 09s)

Stashbot added a comment.Mar 12 2021, 9:05 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-12T09:05:26Z] <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling

Stashbot added a comment.Mar 12 2021, 9:07 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-12T09:07:01Z] <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@9a408b2]: T273847 export queries to relforge dag deployment - elastic-template handling (duration: 01m 35s)

Maintenance_bot removed a project: Patch-For-Review.Mar 12 2021, 9:10 AM
gerritbot added a comment.Mar 12 2021, 11:01 AM

Change 671115 had a related patch set uploaded (by ZPapierski; owner: ZPapierski):
[wikimedia/discovery/analytics@master] Deploy the index setting file

https://gerrit.wikimedia.org/r/671115

gerritbot added a project: Patch-For-Review.Mar 12 2021, 11:01 AM
gerritbot added a comment.Mar 15 2021, 4:56 PM

Change 672463 had a related patch set uploaded (by ZPapierski; owner: ZPapierski):
[wikimedia/discovery/analytics@master] Add code coverage to Sonarqube report

https://gerrit.wikimedia.org/r/672463

gerritbot added a comment.Mar 15 2021, 5:03 PM

Change 671115 merged by ZPapierski:
[wikimedia/discovery/analytics@master] Deploy the index setting file

https://gerrit.wikimedia.org/r/671115

gerritbot added a comment.Mar 15 2021, 5:03 PM

Change 672463 merged by ZPapierski:
[wikimedia/discovery/analytics@master] Add code coverage to Sonarqube report

https://gerrit.wikimedia.org/r/672463

• Zbyszko mentioned this in rWDANf65e5c972d37: Deploy the index setting file.Mar 15 2021, 5:13 PM
• Zbyszko mentioned this in rWDAN3bc7024dc4e2: Add code coverage to Sonarqube report.
• Zbyszko moved this task from To Be Deployed to Waiting on the Discovery-Search (Current work) board.Mar 16 2021, 9:40 AM
Stashbot added a comment.Apr 8 2021, 9:25 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-08T09:25:48Z] <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@d098717]: T273847 export queries to relforge dag deployment - sensor name fix

Stashbot added a comment.Apr 8 2021, 9:27 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-08T09:27:36Z] <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@d098717]: T273847 export queries to relforge dag deployment - sensor name fix (duration: 01m 48s)

Stashbot added a comment.Apr 8 2021, 10:09 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-08T10:09:44Z] <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@ff0137d]: T273847 export queries to relforge dag deployment - start date update

Stashbot added a comment.Apr 8 2021, 10:11 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-08T10:11:28Z] <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@ff0137d]: T273847 export queries to relforge dag deployment - start date update (duration: 01m 37s)

Stashbot added a comment.Apr 8 2021, 11:47 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-08T11:47:30Z] <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@25dad72]: T273847 export queries to relforge dag deployment - elastic index name fix

Stashbot added a comment.Apr 8 2021, 11:49 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-08T11:49:10Z] <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@25dad72]: T273847 export queries to relforge dag deployment - elastic index name fix (duration: 01m 39s)

Stashbot added a comment.Apr 8 2021, 11:57 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-08T11:57:34Z] <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@25dad72]: T273847 export queries to relforge dag deployment - elastic index name fix

Stashbot added a comment.Apr 8 2021, 11:57 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-08T11:57:43Z] <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@25dad72]: T273847 export queries to relforge dag deployment - elastic index name fix (duration: 00m 09s)

Stashbot added a comment.Apr 14 2021, 1:27 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-14T13:27:05Z] <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@825c60a]: T273847 export queries to relforge dag deployment - schedule change

Stashbot added a comment.Apr 14 2021, 1:29 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-14T13:29:14Z] <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@825c60a]: T273847 export queries to relforge dag deployment - schedule change (duration: 02m 08s)

Stashbot added a comment.Apr 14 2021, 2:09 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-14T14:09:06Z] <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@8ae53e3]: T273847 export queries to relforge dag deployment - start date update

Stashbot added a comment.Apr 14 2021, 2:11 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-14T14:11:20Z] <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@8ae53e3]: T273847 export queries to relforge dag deployment - start date update (duration: 02m 14s)

• Zbyszko moved this task from Waiting to In Progress on the Discovery-Search (Current work) board.May 17 2021, 3:31 PM
Stashbot added a comment.May 19 2021, 7:45 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-19T07:45:57Z] <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@5740956]: T273847 deploying export_queries_to_relforge - index setting changes

Stashbot added a comment.May 19 2021, 7:48 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-19T07:48:21Z] <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@5740956]: T273847 deploying export_queries_to_relforge - index setting changes (duration: 02m 23s)

Stashbot added a comment.May 19 2021, 8:10 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-19T08:10:54Z] <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@f514dd9]: T273847 deploying export_queries_to_relforge - starttime bump

Stashbot added a comment.May 19 2021, 8:13 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-19T08:13:19Z] <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@f514dd9]: T273847 deploying export_queries_to_relforge - starttime bump (duration: 02m 24s)

• Zbyszko moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.Jun 3 2021, 10:47 AM
Gehel closed this task as Resolved.Jun 9 2021, 1:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits