Page MenuHomePhabricator

Huge drop in search CTR as of Nov 30 2018
Closed, ResolvedPublic

Description

What's requested.

  • Fix or find an explanation as to why the CTR dropped suddenly in the search dashboards.

Why it's requested.

  • The metrics seem wrong

When it's requested.

  • asap so that the team can properly investigate the search stack if it's proven that the drop is real.

Any other helpful information, like relevant documentation.

  • Nov 30 is a Thursday so it can be related to a change in the code.
  • Further investigation in a comment below T211306#4809753.

Event Timeline

dcausse created this task.Dec 6 2018, 9:37 AM
Restricted Application edited projects, added Discovery-Search; removed Discovery-Search (Current work). · View Herald TranscriptDec 6 2018, 9:37 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
EBjune triaged this task as Normal priority.Dec 6 2018, 6:13 PM

I think it's due to mobile_session_counts.tsv not being updated.
Trying to debug I found that mobile_session_counts.R#L64 is failing with

Error: Duplicate identifiers for rows (2, 3, 4)

It seems to be related to https://github.com/tidyverse/tidyr/issues/426
A this point I have no clue if it's a consequence of another problem or simply because the R library was updated and is now more strict than before.
@chelsyx do you have an idea on what could cause such error message? Thank you!

dcausse updated the task description. (Show Details)Dec 11 2018, 10:21 AM
dcausse added a project: Product-Analytics.
dcausse updated the task description. (Show Details)
chelsyx moved this task from Triage to Doing on the Product-Analytics board.
chelsyx added a comment.EditedDec 13 2018, 12:17 AM

Hey @dcausse , I think the drop came from the missing of app's event data. Starting from Nov 30 2018, the app search events dashboard stop being updated. The reason is that user agent of the MobileWikiAppSearch table is not being parsed, and thus all fields in the useragent struct in NULL. Our dashboard script need to use the os_family field to consume the app events in the overall CTR calculation.

I've filed T211833 and ask analytics engineering to look into it.

Change 479366 had a related patch set uploaded (by Chelsyx; owner: Chelsyx):
[wikimedia/discovery/golden@master] Fix a bug that prevent mobile_session_counts from updating The issue stated in the bug ticket is not directly related to this fix, but one comment in this ticket is. See https://phabricator.wikimedia.org/T211306#4809753

https://gerrit.wikimedia.org/r/479366

Change 479366 merged by Chelsyx:
[wikimedia/discovery/golden@master] Fix a bug that prevent mobile_session_counts from updating The issue stated in the bug ticket is not directly related to this fix, but one comment in this ticket is. See https://phabricator.wikimedia.org/T211306#4809753

https://gerrit.wikimedia.org/r/479366

chelsyx moved this task from Doing to Epics on the Product-Analytics board.Dec 13 2018, 1:46 AM

I fixed the other bug with mobile_session_counts.tsv as stated in T211306#4809753. We should see the data in this table come back after report updater finish the cron job later today/tomorrow.

Moved this task to "Blocked" since we're waiting for T211833 to be fixed.

Many thanks for the investigation and the fixes!

moving to done as the CTR metric is now back to usual.
Thank you!

debt closed this task as Resolved.Dec 21 2018, 1:33 PM
debt awarded a token.
debt added a subscriber: debt.

Thanks for your help, @chelsyx and to @dcausse for discovering it!