Several methods of spark parameter configuration

1.Spark attribute
Spark The operation of the application is controlled by external parameters , Whether the parameters are set correctly , Good and bad will directly affect the performance of the application , This will affect the performance of our entire cluster . The following methods are used for parameter control ：
（1） Set it directly to SparkConf, Pass to... In the form of parameters SparkContext, Achieve the goal of control .（ adopt set() Methods the incoming key-value Yes ） such as ：

val conf = new SparkConf()
　　　　.setMaster(”local[2]“) # Be careful 1
　　　　.setAppName(”test“)
　　　　.set(“spark.cores.max”, “10”) ######set() Method to pass in attribute parameters
val sc = new SparkContext(conf)
# Be careful 1 Local open 2 Threads to work , Pay attention , The local working mode is in addition to collection generation RDD、 Read local files and hdfs The file can start a thread and work outside , In other cases, at least two threads can work normally . This is because in addition to the above ,Spark Will open a reciver To accept the data , If there is only one thread ,reciver Just occupy the resources of the only thread , And data processing and other operations will have no resources to execute .

（2） Dynamic loading Spark attribute . Not hard coded for attributes such as application name and clustering mode , Flexible application . You can do this ：val sc = new SparkContext(new SparkConf()). adopt spark-submit Add necessary parameters when . Such as ：

./bin/spark-submit --name “My app” --master local[4] --conf spark.eventLog.enabled=false
–conf “spark.executor.extraJavaOptions=-XX:&#