Using restcloud ETL shell component to schedule dataX offline tasks

 

RestCloud ETL Community Edition is a data integration tool , Provide visual multi data pipeline construction 、 Data source management 、 Operation monitoring and authority management functions .

1. The scene that ：

For some offline data synchronization tools that are already using Alibaba DataX Users of , Want to run regularly every day （ Or manual operation ）job The task is very difficult , You need to enter the control interface and click the command line , Is it simpler 、 The quick way ？ Yes ！ Use RestCloud ETL Shell Component build process , Realize timing / Run the task manually .

2. Overall flow chart ：

3. Implementation steps ：

3.1 Create a process , Pull an execution from the script execution component category Shell Script components

3.2 To configure Shell The script content

Here, the remote server is called , Fill in the... Of the remote server IP、 The logged in user ID、 Password and what needs to be executed shell Script .

Shell The script can use variables here , Dynamically obtain the data in the process or http Request incoming data .

Shell The sample script code is as follows ：

============

#!/bin/bash

#!/usr/bin/env python

source /etc/profile

python /usr/wgx/datax/bin/datax.py -p "-DlastSuccessEndTime='2022-05-12 18:37:27'" \

/usr/wgx/datax/job/mysql-job3.json

============

DataX Of Job The sample code is as follows :

3.3 Configure assertion execution logic

The running results will be stored in shellResult variable , Therefore, it can be obtained and judged in the flow line .

If you return 0 On behalf of success , Other returns are unsuccessful , If it is not successful, we can send a nailing message to alert the message .

Successful routing line configuration

Failed routing line configuration

After the failure, it will be sent through the nailing message ：

 

Nail configuration

such , The whole process has been configured .

3.4 Configure scheduled scheduling

 

We can return to the process interface , Modify process configuration .

Here we use Only the primary server can run that will do .

Operation mode ： Here we choose the mode of scheduled scheduling to run
Scheduling strategy ： Select the scheduling strategy according to your own needs
Task scheduling mode ：

Task queue collection mode ： Means to put the task in the queue , Run freely by the server

All cluster servers can run simultaneously ： Indicates that all servers execute
Only the primary server can run ： It is applicable when the performance of the primary server is higher than that of other servers , Or just one server Run time usage

It is uniformly dispatched by the dispatching server ： It means that the dispatcher of the system uniformly schedules

 4. Execution results

We can manually run and view the scheduling records

The specific process operation flow line can be viewed by clicking graphic control

We enter datax You can view the log files generated by the operation

  When executed Shell When the failure , Then send a pinning message

  Come here , Completed through RestCloud ETL Shell Component to realize remote scheduling datax Mission .

Thank you for your attention and support ！ For more features , Please join RestCloud ETL Community free download experience , Download address ：https://etl.restcloud.cn