How to handle 2gcsv files that cannot be opened? Use byzer

Project needs , It is necessary to export the data of the customer's environment for reconciliation analysis , Customer data is secret , Cannot provide a calling interface , You can only csv In the form of documents .
But one 2G Of csv file , Most machines will crash themselves . Another way is to split the tool , But it is troublesome to analyze data after splitting .

There is one Byzer Tools can meet this requirement

Official website address ：

https://docs.byzer.org/#/

Support private environment deployment , Ensure data privacy , The construction environment is as follows ：

1、 Prepare one liunx machine , Machine configuration 2 nucleus 8G about

2、 download byzer And install

wget https://download.byzer.org/byzer/2.3.0.1/byzer-lang-all-in-one-linux-amd64-3.1.1-2.3.0.1.tar.gz

tar -zxvf byzer-lang-all-in-one-linux-amd64-3.1.1-2.3.0.1.tar.gz

cd byzer-lang-all-in-one-linux-amd64-3.1.1-2.3.0.1

3、 start-up byzer

./bin/byzer.sh start

4. You can visit byzer, But not very friendly , The supporting visualization tools can be installed notebook

5、 Download and install notebook

wget https://download.byzer.org/byzer-notebook/1.2.0/Byzer-Notebook-1.2.0.tar.gz

tar -xvf Byzer-Notebook-1.2.0.tar.gz

6、notebook rely on mysql, It needs to be installed in advance mysql

have access to docker-compose install , Other installation methods , Baidu for a while

docker-compose.yml file

version: "2"

services:
  mysql:
    container_name: mysql
    image: mysql:5.7
    restart: always
    volumes:
      - ./mysql/data:/var/lib/mysql
      - ./mysql/init:/docker-entrypoint-initdb.d
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3306:3306"
    environment:
      MYSQL_ROOT_PASSWORD: "XXXXX"
      TZ: "Asia/Shanghai"
    command: --max_allowed_packet=32505856

7、 adjustment notebook About mysql Configuration of

Path address

cd /home/Byzer-Notebook-1.2.0/conf

8、 start-up notebook

./bin/notebook.sh start

9. After registration, you can access and use

10、 Click upload , Upload your own 2G Of csv file

the reason being that 2G The file of , Upload is slow , Here we need to wait patiently

11、 New notebook , View the file

12、 Will just export csv Convert to table structure

load csv.`/tmp/upload/billing.csv` where header="true" as r3;

If it is xlsx file

load excel.`/tmp/upload/billing.xlsx` where header="true" as r4;

Be careful

Here are the back quotes , Not single quotes

13、 Now you can view and use

select * from r3 limit 10 as 2022_06_24_r3;

 Insert a code chip here 

Be careful to take it with you at the back “as The table alias ” This may be byzer A special way of marking .

select sum(BlendedCost) from r3 where payerAccountId=417966497442 as 417966497442_count;

Support all sql A query

Note appended ：

If this error occurs , Check the environment JDK To configure

If there is no access 9002,9003 port , Check the security group settings