A few extra complications:

• mjolnir has bits that take both the loaded dataframe and a schema, and makes sure they are compatible. That might be useful to bring into discolytics as an optional extension? It helps to document the shape of the data that is expected by the script.

• mjolnir reads a table that uses a User Defined Type (VectorUDT), these cannot be read from hive. Instead we use DataFrame.inputFiles to re-read the parquet files directly. That likely needs to stay on the mjolnir side, we've never seen another use case for the UDT's.

• mjolnir has some code in mjolnir.cli.helpers.Cli._resolve_partition_spec that magic's up the partitioning information used in the output tables. Will need to decide what to do with this.

• mjolnir doesn't write through hive directly, rather it writes parquet files and then adds partitions in hive that point at those parquet files. Not sure if this functionality required.

ebernhardson opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1246

WIP: search: Simplify mjolnir dag

ebernhardson opened https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/51

Draft: Add support to HivePartition for features needed by Mjolnir

ebernhardson opened https://gitlab.wikimedia.org/repos/search-platform/mjolnir/-/merge_requests/15

Draft: Migrate to discolytics HivePartition

dcausse merged https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/51

Add support to HivePartition for features needed by Mjolnir

ebernhardson merged https://gitlab.wikimedia.org/repos/search-platform/mjolnir/-/merge_requests/15

Migrate to discolytics HivePartition

ebernhardson merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1246

search: Simplify mjolnir dag

This has now been fully deployed, but we will want to make sure it runs to completion this week. It typically starts thursday at 00:00 and runs for 20-30h.