Page MenuHomePhabricator

NEW BUG REPORT 12 new wikis missing from the mediawiki_history dataset
Closed, ResolvedPublic1 Estimated Story PointsBUG REPORT

Description

Data Engineering Bug Report or Data Problem Form.

Please fill out the following

What kind of problem are you reporting?

  • Access related problem
  • Service related problem
  • Data related problem
For a data related problem:
  • Is this a data quality issue?
    • Yes
  • What datasets and/or dashboards are affected?
    • mediawiki_history and its relatives
  • What are the observed vs expected results? Please include information such as location of data, any initial assessments, sql statements, screenshots.
    • All public wikis should be included in mediawiki_history. However, new wikis are not regularly or automatically added to the sqoop list (T190700), which means data users have to notice and manually request the addition. This has become a periodic ritual for me 😁 (T220456, T299548, T329119).
    • In this case, the following new wikis are missing:
anpwiki
blkwiktionary
btmwiktionary
ckbwiktionary
fatwiki
fonwiki
gpewiki
guwwikinews
kbdwiktionary
kcgwiktionary
suwikisource
tlywiki
For the DE Team to fill out
Which systems does this effect?
  • Hive
  • Druid
  • Superset
  • Turnilo
  • WikiDumps
  • Wikistats
  • Airflow
  • HDFS
  • Goblin
  • Scqoop
  • Dashiki
  • DataHub
  • Spark
  • Jupyter
  • Modern Event Platform
  • Event Logging
  • Other
Impact Assessment:

Does this problem qualify as an incident?

  • Yes
  • No

Does this violate an SLO?

  • Yes
  • No
Value CalculatorRank
Will this improve the efficiency of a teams workflow?1-3
Does this have an effect of our Core Metrics?1-3
Does this align with our strategic goals?1-3
Is this a blocker for another team?1-3

Event Timeline

There are 4 more wikis missing from mediawiki_history, which brings the updated list of missing wikis (including those Neil listed above) to:

zghwiki 
bjnwikiquote 
dgawiki 
bbcwiki 
fonwiki 
tlywiki 
suwikisource 
blkwiktionary 
btmwiktionary 
gpewiki 
guwwikinews 
kcgwiktionary 
fatwikipedia 
kbdwiktionary 
ckbwiktionary 
anpwiki

Change 994178 had a related patch set uploaded (by Joal; author: Joal):

[analytics/refinery@master] Update sqoop list adding new wikis

https://gerrit.wikimedia.org/r/994178

lbowmaker set the point value for this task to 1.Jan 30 2024, 3:38 PM

Change 994178 merged by Phuedx:

[analytics/refinery@master] Update sqoop list adding new wikis

https://gerrit.wikimedia.org/r/994178