Change Details

We workaround a limitation in Spark that doesn't allow ALTER TABLE CHANGE COLUMN statements by issuing the DDL through a manually created Hive JDBC connection, instead of through `spark.sql()`. This recently caused a bug: {T209407}. We should fix this by submitting a fix for https://issues.apache.org/jira/browse/SPARK-23890 in Apache Spark upstream. WIP Pull Request here: https://github.com/apache/spark/pull/21012

We workaround a limitation in Spark that doesn't allow ALTER TABLE CHANGE COLUMN statements by issuing the DDL through a manually created Hive JDBC connection, instead of through `spark.sql()`. This recently caused a bug: {T209407}. We should fix this by submitting a fix for https://issues.apache.org/jira/browse/SPARK-23890 in Apache Spark upstream. WIP Pull Request here: https://github.com/apache/spark/pull/21012 If we do this, I believe that event schema column descriptions/comments will finally make their way into the Hive tables, and then up into DataHub.

We workaround a limitation in Spark that doesn't allow ALTER TABLE CHANGE COLUMN statements by issuing the DDL through a manually created Hive JDBC connection, instead of through `spark.sql()`. This recently caused a bug: {T209407}. We should fix this by submitting a fix for https://issues.apache.org/jira/browse/SPARK-23890 in Apache Spark upstream. WIP Pull Request here: https://github.com/apache/spark/pull/21012 If we do this, I believe that event schema column descriptions/comments will finally make their way into the Hive tables, and then up into DataHub.