Restcloud ETL extracting dynamic library table data

RestCloud ETL Community Edition is a data integration tool , Provide visual multi data pipeline construction 、 Data source management 、 Operation monitoring and authority management functions .

1. The scene that ：

For some business systems, a new data table is generated automatically every day , Hope to pass 1 Data flow automatically reads the data table of the day every day . Such as 2022 year 6 month 1 The name of daily production table is “table_20220601”,2022 year 6 month 2 The name of daily production table is “table_20220602”, And so on ; For such dynamic tables , How to use RestCloud ETL Simple 、 Convenient data extraction .

2. Realization principle ：

First, we need to automatically generate the name of the table according to the date , And transfer the parameter to the library table input component . The implementation flow chart is as follows ：

The prerequisite for this service is that the library table input component supports receiving variable transmission , This RestCloud ETL Unparalleled advantages .

3. Implementation steps ：

3.1 To configure " perform Java Script "

Right click " perform Java Script ", Click on “ Node properties ” Open the configuration edit page , Click next , edit Java Code . Add the generated table name code to the code , The output parameters are ： p_tablename  , Here's the picture ：

The code is as follows ：

package cn.restcloud.etl.rule.ext;

import org.apache.commons.lang3.StringUtils;

import org.bson.Document;

import java.sql.Connection;

import cn.restcloud.framework.core.context.*;

import cn.restcloud.etl.base.IETLBaseEvent;

import cn.restcloud.etl.base.IETLBaseProcessEngine;

import cn.restcloud.framework.core.util.*;

import cn.restcloud.framework.core.util.db.rdb.*;

import java.util.*;

import java.text.SimpleDateFormat;

/**

indoc Stream data

Successful execution must return characters 1, return 0 Means to terminate the process

*/

public classETL_T00005_1INWBJPUXVB implements IETLBaseEvent {

@Override

public String execute(IETLBaseProcessEngine engine, Document modelNodeDoc, Document indoc,String fieldId,String params) throws Exception {

Date date = newDate();

        SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd");

        String table = "table_" + sdf.format(date);

        PrintUtil.o(table);

        indoc.put("p_tablename",table);

return "1";

}

}   

Complete code editing , Click on 【 Compile and save 】 Exit configuration .

3.2 Library table input configuration

Right click " Library table input ", Click on “ Node properties ” Open the configuration edit page .

Select source table data source , Click next ：

Edit read data SQL sentence ,select * from The name of the table , Table name input receiving parameters , Parameter format ：${ Parameters },${p_tablename}, Click next ：

Click on 【 New fields 】, Input extraction data table fields , Click Save to complete the configuration .

3.3 Library table output configuration

Right click " Library table input ", Click on “ Node properties ” Open the configuration edit page .

Enter the configuration in sequence , Select data source , Load database tables , Select database table .

Click on 【 Read from the table 】, Check keyword ( Primary key ), Click next .

Select the corresponding journal policy as required , Click Save to complete the configuration .

4. Run the test

Click on 【 function 】

Click on 【 Began to run 】

5. View the run results

Click library table output , Data transmission statistics