Spark SQL访问PostgreSQL

 .set("spark.driver.allowMultipleContexts", "true")  

 .set("spark.sql.shuffle.partitions","12") 

//本地启动  

 .setMaster("local[2]");

val sc = new SparkContext(sparkConf)
val sqlContext = new SQLContext(sc)

val url = "jdbc:postgresql://192.168.1.110:5432/demo?user=test&password=test"
val dataFrame = sqlContext.load("jdbc", Map(
"url" -> url,
"driver" -> "org.postgresql.Driver",
"dbtable" -> "tab_users"
))

dataFrame.registerTempTable("USERS")
val users = sqlContext.sql("select * from USERS")
users.foreach(println)
}
}

从性能角度考虑，还可以在创建SQLContext时，设置一些配置项，例如：

val sqlContext = new SQLContext(sc)
sqlContext.setConf("spark.sql.inMemoryColumnarStorage.batchSize", "10000")

Spark SQL访问PostgreSQL

标签：ring   ext   temp   erb   gis   frame   tab   dep   分享图片