Change Details

StructTypes need to be recursively merged and reordered via SQL. This is done for StructType fields, but it is not done for ArrayTypes with elementType == StructType. Since no checking is done for ArrayType, the SQL for an array of struct field will look like: ``` CAST(d AS ARRAY<STRUCT<`db_string`: STRING, `da_long`: BIGINT>>) AS d ``` Which will not work if the source schema has the fields in a different order. This is causing bad refined data for mediawiki_cirrussearch_request, as reported in https://phabricator.wikimedia.org/P12200. We need to make convertToSchema smarter about converting an array of structs.

StructTypes need to be recursively merged and reordered via SQL. This is done for StructType fields, but it is not done for ArrayTypes with elementType == StructType. Since no checking is done for ArrayType, the SQL for an array of struct field will look like: ``` CAST(d AS ARRAY<STRUCT<`db_string`: STRING, `da_long`: BIGINT>>) AS d ``` Which will not work if the source schema has the struct fields in a different order, e.g. `db_long`, `db_string`. This is causing bad refined data for mediawiki_cirrussearch_request, as reported in https://phabricator.wikimedia.org/P12200. We need to make convertToSchema smarter about converting an array of structs.

StructTypes need to be recursively merged and reordered via SQL. This is done for StructType fields, but it is not done for ArrayTypes with elementType == StructType. Since no checking is done for ArrayType, the SQL for an array of struct field will look like: ``` CAST(d AS ARRAY<STRUCT<`db_string`: STRING, `da_long`: BIGINT>>) AS d ``` Which will not work if the source schema has the struct fields in a different order, e.g. `db_long`, `db_string`. This is causing bad refined data for mediawiki_cirrussearch_request, as reported in https://phabricator.wikimedia.org/P12200. We need to make convertToSchema smarter about converting an array of structs.