Repository navigation
cannot import into bigint column from CSV file #39
Description
Hi,
Am trying to import a CSV file to Databricks (9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12) and even with a very simple 2-column example and tiny integer input, it fails with Failed to merge incompatible data types LongType and IntegerType
Here are my test commands:
dbsqlcli -e "create or replace table `datatype_test`(`bigint_col` bigint, `int_col` integer)"
dbsqlcli -e "COPY INTO datatype_test FROM '/tmp/datatype_test.csv' FILEFORMAT = CSV FORMAT_OPTIONS ('header'='true', 'inferSchema'='true') COPY_OPTIONS ('mergeSchema'='true')"
where /tmp/datatype_test.csv contains:
bigint_col,int_col
,123
123,
123,123
Any suggestions?
The error response header is:
org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.sql.AnalysisException: Failed to merge fields 'bigint_col' and 'bigint_col'. Failed to merge incompatible data types LongType and IntegerType
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:1019)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:759)
...
Activity