Making the outputs queryable by SQL is a huge win for analysis, and this data can be plugged into Superset dashboards.
Currently, the outputs are only in .ndjson and .csv files under https://analytics.wikimedia.org/published/datasets/one-off/html-dump-scraper-refs/, but these can only be analyzed after import or by using eg. `jq`.
- [] Plan Hive schema. Map data types, decide on naming and namespace,
- [] Hive adapter: https://gitlab.com/wmde/technical-wishes/apache_hive/-/merge_requests/1
- [] Scraper persists outputs in Hive. Start with `pipeline.ex`. Split hive persistence into a separate module.
to Hive when configured for production. Batch inserts.
- [] Document fields and provide example HiveQL queries.
- [] Airflow DAG takes over aggregation and alerts
== Hive adapter
There seems to be no existing driver for Hive in Elixir, so we've written a basic one: `apache_hive`
== Out of scope
There are no known external consumers of the published analytics files, so no obligation to keep producing the files. Old published files could even be deleted and json code run optionally depending on configuration.