Spark Learning: build SQL to meet the specified optimization rules

win10 compile Spark Source code
by SparkSql Add custom commands

One 、 Connect local oracle database

import org.apache.spark.sql.DataFrame
val dfOracle:DataFrame = spark.read.format("jdbc").option("driver", "oracle.jdbc.driver.OracleDriver").option("url", "jdbc:oracle:thin:@192.168.0.101:1521:orcl").option("user", "scott").option("password","scott").option("numPartitions", 20).option("dbtable", "EMP").load()
// Connect scott Among users emp Table to test 
// Create a temporary table 
dfOracle.createTempView("t")
// bring apply Which rules are printed 
spark.sql("SET spark.sql.planChangeLog.level=WARN")

Two 、 Rule three

1、 Rule understanding

• CombineFilters: Merge two adjacent Filters, A form of predicate pushdown , Put the... Of the same table Filter Merge together , Such as 1000 < sql < 2000
• CollapseProject: Merge two adjacent project, I understand the same as column pruning , Unnecessary columns do not project
• BooleanSimplification: Boolean expression simplification , The expression in the condition satisfies true, You can omit , Such as sql in 1 = 1

2、 Code implementation

val query:String = "select a.ename from (select ename,sal from t where 1 = 1 and sal > 1000) a where a.sal < 2000)
spark.sql(query).explain(true)

2、 Log information

3、 ... and 、 Rule five

1、 Rule understanding

• ConstantFolding: Constant collapse , Constants pre evaluate or remove expressions from constants , As in the example 1 = 1
• PushDownPredicates: Predicate push-down ,Filter Conditions are pushed to the data source , Reduce the amount of data , As in the example sal > 1000
• ReplaceDistinctWithAggregate: take distinct De duplication to group by Aggregation operation , As in the example distinct
• ReplaceExceptWithAntiJoin: take Except Difference set into Left-Anti Join Left half connection , As in the example except
• FoldablePropagation: Foldable propagation , It's not clear yet …

2、 Code implementation

//oracle Create a new table in the library 
create table emp1 as select * from emp where deptno = 20;
// establish DataFrame
val dfOracle1:DataFrame = spark.read.format("jdbc").option("driver", "oracle.jdbc.driver.OracleDriver").option("url", "jdbc:oracle:thin:@192.168.0.101:1521:orcl").option("user", "scott").option("password","scott").option("numPartitions", 20).option("dbtable", "EMP").load()
// Create a temporary table 
dfOracle1.createTempView("t1")
val query:String = "select distinct a.job from (select * from t where sal > 1000 and 1 = 1) a where a.deptno = 10 except select job from t1 where sal > 2000"
spark.sql(query).explain(true)

3、 Log information