Install spark + run Scala related projects with commands + crontab scheduled execution

One 、 preparation ：

1. Software needed ：java install jre1.8    spark-2.2.0.tar.gz( It has been configured ）

2. What is needed to run the project ：maven The project hit jar package --prodmge-mapreduce-0.0.1-SNAPSHOT.jar, Required for operation of the project lib,bjtds.sql

Two 、 install spark And configuration spark The variable of ： Mainly for reference https://blog.csdn.net/weixin_46995451/article/details/108709068

Pay attention to spark The version should be consistent with the version used in the project , Otherwise it will go wrong .

Modify environment variables ：vim /etc/profile（ There is no need to modify in the project , Ignore this step for now ）

After adding ,source /etc/profile 

After running, only the current window can be used directly in any position spark The order of , It will not work until it is restarted .

3、 ... and 、 Use the command to run the project

1. stay spark-2.2.0/bin Execute under directory No profile

./spark-submit  --driver-class-path /home/sy/spark/mysql-connector-java-5.1.6-bin.jar  --jars $(files=(/home/sy/single-node/3.docker-service-image-build/lib/*.jar); IFS=,; echo "${files[*]}")  --class com.bjtds.ProdmgeDataStastics --master local  /home/sy/single-node/3.docker-service-image-build/test.jar

2. After configuring environment variables , Direct use spark-submit command

spark-submit  --driver-class-path /home/sy/spark/mysql-connector-java-5.1.6-bin.jar  --jars $(files=(/home/sy/single-node/3.docker-service-image-build/lib/*.jar); IFS=,; echo "${files[*]}")  --class com.bjtds.ProdmgeDataStastics --master local  /home/sy/single-node/3.docker-service-image-build/test.jar

Four 、 Write commands and configurations to shell Script

vim data.sh

Add the following to the script

#!/bin/bash

set mysql-host-mapreduce=192.168.1.242:3306

set mysql-database-bjtds=bjtds

set mysql-username=root

set mysql-passwd=bjtds2019

cd /home/sy/spark/spark-2.2.0/bin && ./spark-submit  --driver-class-path /home/sy/spark/mysql-connector-java-5.1.6-bin.jar  --jars $(files=(/home/sy/single-node/3.docker-service-image-build/lib/*.jar); IFS=,; echo "${files[*]}")  --class com.bjtds.ProdmgeDataStastics --master local  /home/sy/single-node/3.docker-service-image-build/test.jar

chmod +x data.sh

5、 ... and 、 Use shell Scripts automatically add scheduled tasks to crontab 

vim execCrontab.sh

Add content ：

#!/bin/sh

crontab -l > conf && echo  "* * * * * /bin/bash /home/sy/spark/test.sh>>/home/sy/spark/prodmgeDataStastics.log 2>&1 " >> conf && crontab conf && rm -f conf

chmod +x execCrontab.sh // Set the permissions

./execCrontab.sh // Execute the script

After the execution , see

crontab -e // Edit view crontab, The original command does not delete , Append to the back .

6、 ... and 、 Summary of problems that have occurred

1.ip Wrong address , Project configuration ip Follow the running ip atypism , There was this mistake .

Finally, the reason is found to be configured in the project code mysql-host, In the system mysql Already configured ip, terms of settlement ： Changed the project code mysql-host by mysql-host-xxx.

2. There is no... In the command mysql The driver reports an error spark-submit    --jars $(files=(/home/sy/single-node/3.docker-service-image-build/lib/*.jar); IFS=,; echo "${files[*]}")  --class com.bjtds.ProdmgeDataStastics --files /home/sy/spark/prodmgeDataStastics.xml --master local  /home/sy/single-node/3.docker-service-image-build/test.jar

After adding the driver spark-submit  --driver-class-path /home/sy/spark/mysql-connector-java-5.1.6-bin.jar  --jars $(files=(/home/sy/single-node/3.docker-service-image-build/lib/*.jar); IFS=,; echo "${files[*]}")  --class com.bjtds.ProdmgeDataStastics --files /home/sy/spark/prodmgeDataStastics.xml --master local  /home/sy/single-node/3.docker-service-image-build/test.jar Run correctly .

3、crontab Timing execution

crontab -e  Enter editing status , Add the last line

service cron restart To restart .

But I found that I couldn't find spark-submit command ,spark The environment variable has been configured , Still in crontab perform shell Could not find problem while scripting , So find out why

Change to the following command , Can operate correctly .

First clear the log file

Log files are also output normally