Page MenuHomePhabricator

Fewer than expected articles have the 'has suggestion' flag set in elasticsearch
Closed, ResolvedPublicBUG REPORT

Description

After running the DAG with the latest snapshots, and importing the data into the search indices, we have far fewer articles with the recommendation.image/exists|1 flag set in weighted_tags in the article's search index doc than we'd expect. For ptwiki we expect around 130k articles, but we have only ~2.9k

Investigating, it looks like

  1. the search index deltas in hive produced by the SD team seem off
  2. the logs for importing the data into the search index also seem off - the ptwiki was imported in 1 second, which is way too fast

Event Timeline

Cparle added a subscriber: EBernhardson.

@EBernhardson re-imported the data, and now for ptwiki at least we have ~130k articles again. Still trying to see what went wrong

Update on part 1 from the description - we figured out what was wrong with the data on the SD side, and have merged a patch to fix the problem. Still don't know what went wrong with the import from the search side ...

Cparle claimed this task.

SD side fixed, and we have had successful search imports since, so closing