Field timestamp is repeated with multiple non StructType types: StructField(timestamp,DoubleType,true) , StructField(timestamp,LongType,true))
This is an error message I'm getting from the Hive refinement step. It seems that sometimes clients are emitting the new timestamp field sometimes as integers, and sometimes as decimals. This passes JSONSchema validation because number validates both decimals and integers. However, this won't work for Hive. It is really important to know the exact type in strongly typed systems. This error means that the refinement failed for the hours of data that have different types in them. It looks like the first successful hour of data contained only decimal fields, so the timestamp field in Hive is of type double.
Can y'all make sure the client code always sends decimal numbers? Adding a .0 to the end of the number would be good enough.