Home Custom Data Types for DataFrame columns when using Spark JDBC
Reply: 1

Custom Data Types for DataFrame columns when using Spark JDBC

horatio1701d
1#
horatio1701d Published in 2017-12-04 15:38:36Z

I know I can use a custom dialect for having a correct mapping between my db and spark but how can I create a custom table schema with specific field data types and lengths when I use spark's jdbc.write options? I would like to have granular control over my table schemas when I load a table from spark.

user8371915
2#
user8371915 Reply to 2017-12-04 15:49:01Z

There is a minimal flexibility for writes, implemented by

  • SPARK-10101 - Spark JDBC writer mapping String to TEXT or VARCHAR
  • SPARK-10849 - Allow user to specify database column type for data frame fields when writing data to jdbc data sources

but if you want

to have granular control over my table schemas when I load a table from spark.

you might have to implement your own JdbcDialect. It is internal developer API and as far as I can tell it is not plugable so you may need customized Spark binaries (it might be possible to registerDialect but I haven't tried this).

You need to login account before you can post.

About| Privacy statement| Terms of Service| Advertising| Contact us| Help| Sitemap|
Processed in 0.421639 second(s) , Gzip On .

© 2016 Powered by mzan.com design MATCHINFO