Weekly update:

• transform into python package for library use
• wip deploy airflow dags

updates:

• created wikimedia gitlab repo for ml related research code https://gitlab.wikimedia.org/FabianKaelin/research-ml

• added research-transform python package for reusable research focused code
• added wip knowledge gap code
• instructions interactive development, either from jupyter notebook or ipython/VS code

updates:

• generic spark job airflow dag template
• configurable airflow dag to download media from swift

Updates

• Research now has a dedicated hdfs data directory: /wmf/data/research
• Moved datasets: pageviews_daily (used by knowledge_gaps), html enterprise dumps, wikidiff
• Next up: article quality, content gaps

Updates

• moved articles features to research data directory, populated with an incremental airflow dag,
• the article features are available as en external hive research.article_features. the article quality scores are stored in research.article_quality_scores
• a library wrapper to access datasets generated by research via spark is available in https://gitlab.wikimedia.org/repos/research/research-common/-/blob/main/research_common/datasets.py