Spark related FAQ summary

One .SparkSQL relevant

Question 1 unresolvedAdderssException

• In execution Spark Throw in the process ：Failed to bingdate001:33381,caused by：java.nio.channels.unresolvedAdderssException

• reason ： The reason is due to hosts Not configured , Cause no recognition

• resolvent ： Modify the corresponding machine host that will do

Question two IndexOutOfBoundsException

• In execution Sparksql operation orc Thrown when a table of type ：java.lang.IndexOutOfBoundsException perhaps java.lang.NullPointerException

• reason ： There is an empty partition or table orc file . The BUG stay Spark2.3.0 And then fix it

• resolvent ： Circumvention solution . modify ORC The default segmentation strategy for is ：hive.exec.orc.split.strategy=BI Solve .Orc The score of split Yes 3 Strategies （ETL、BI、HYBIRD）, The default is HYBIRD( Mixed mode , Automatically select according to the size and number of files ETL still BI Pattern ),BI The mode is divided according to the number of files split.
  Spark2.1.0 Permanent functions are not supported , This is because Spark2.2.0 Reading was not supported before hdfs above jar package .

Question 3 socketTimeOutException:read

• Saprk-sql and ThriftServer Use times wrong ：Java.net.socketTimeOutException:read time out

• reason ： Is due to hivemetastore Too busy or gc Cause connection timeout

• resolvent ：spark-sql solve ：hive.metastore.client.socket.timeout Increase the parameter .ThriftServer terms of settlement ： Get a Connection And before ：DriverManager.setLoginTimeout(100)

Question 4 Spark-sql Running too slow

• Spark-sql When executing, a Very small files Split into 25 individual task Run , The running speed is too slow .

• reason ： Is due to HaddopRDD In the process of generation partitions Yes, I can take parameters mapreduce.job.maps , or mapred.map.tasks（25） and spark Default number of partitions (2) Compare the maximum value , Therefore, the default is 25

• resolvent ： Modify this parameter to set task Come down .

Question five StackOverflowError

• stay Spark SQL Running in SQL If the sentence is too complicated , There will be java.lang.StackOverflowError abnormal

• reason ： This is because when the program is running Stack Size greater than JVM Set the size of

• resolvent ： By starting Spark-sql When it's time to add --driver-java-options "-Xss10m" Option to solve this problem

Question 6 OutOfMemoryError

• Sparksql In use Executor End throw ：java.lang.OutOfMemoryError: GC overhead limit exceeded

• reason ： This is because most of the events are GC, Lead to OOM.

• resolvent ： Increase actuator memory , modify GC Strategy spark.executor.extraJavaOptions -XX:+UseG1GC

Two .Spark core relevant

Question seven No space left on device

• In the use of Spark In the process ：java.io.IOException: No space left on device

• reason ： Generally due to Spark Of tmp The directory is full, causing

• resolvent ： You can set the directory space larger , It supports separating multiple directories by commas ：spark.local.dir

Question 8 Java heap space

• common OOM：java.lang.OutOfMemoryError: Java heap space

• reason ：1. Too much data , Applied Executor Resources are not enough to support .2. The amount of data in a single partition is too large , And too many partitions lead to execution task and job Too much information stored leads to Driver OutOfMemoryError

• resolvent ：1、 Try not to use collect operation .2、 Check whether the data is tilted , increase shuffle Parallelism of , enlarge Executor Memory

Question 9 ClassNotFoundException

• jar When the package version conflicts ：java.lang.ClassNotFoundException: XXX

• reason ： Generally, it may be the user jar and Spark jar Conflict

• resolvent ：1、 The best and Spark dependent jar ADAPTS .2、 If not, you can use parameters ：spark.driver.userClassPathFirst and spark.executor.userClassPathFirst Set to true

3、 ... and 、Streaming relevant

Question 10 OffsetOutOfRangeException

• consumption kafka when , Error in reading message ：OffsetOutOfRangeException

• reason ： Read the offsetRange Beyond the Kafka Message range , If it is less than, that is kafka The saved message has been disposed （log.retention.hours）. Or beyond Kafka The existing offset

• resolvent ： In the reading offset Correct first , Get offset Of earliestOffset and lastestOffset

Question 11 consumption kafka

• consumption kafka when , first job Read existing All the news , Lead to the first Job Handle it too long or even fail

• reason ：auto.offset.reset Set up in order to earliest From the earliest offset Start spending , There's no setup spark.streaming.kafka.maxRatePerPartition Parameters

• resolvent ： Specify to start from the data consumed before ： Set up offsetRange. And set the parameter to ：auto.offset.reset=latest Set up Spark The rate of each partition .