数据湖架构之Hudi编译篇

前言

说起编译hudi，从第一遍过之后，再回过头来看，发现就是第一遍不熟悉，出现的一切问题可以总结为maven仓库没配置好。一开始我只是配置了阿里云仓库，但是后面不断报错，然后百度谷歌找原因，再调整配置，再编译，最后就成功了，所以整体来说编译不复杂，只要配置正确，那我把最后可以通过的配置贴出来，这也是我觉得可以帮助到大部分同学的地方。

版本与源码

hudi迭代还是比较快的，因为同时也依赖了hadoop和spark，为了组合使用，我使用的是0.9.0版本，对应地址：[https://hudi.apache.org/releases/release-0.9.0](https://hudi.apache.org/releases/release-
0.9.0)
源码部分可以点击download的部分，即可下载

环境准备

[[email protected] target]# mvn -v
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00)
Maven home: /apps/svr/maven
Java version: 1.8.0_144, vendor: Oracle Corporation, runtime: /apps/svr/jdk1.8.0_144/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "3.10.5-3.el6.x86_64", arch: "amd64", family: "unix"

编译

编译过程其实就是普通maven项目，不是很复杂

mvn clean install -DskipTests -DskipITs -Dscala-2.12 -Dspark3 

前面也提到了，陆陆续续才成功，最后的结果，还是需要亮相一下:

[INFO] Hudi 0.9.0 ......................................... SUCCESS [  2.758 s]
[INFO] hudi-common ........................................ SUCCESS [ 20.652 s]
[INFO] hudi-timeline-service .............................. SUCCESS [  3.375 s]
[INFO] hudi-client ........................................ SUCCESS [  0.208 s]
[INFO] hudi-client-common ................................. SUCCESS [ 12.704 s]
[INFO] hudi-hadoop-mr ..................................... SUCCESS [  5.637 s]
[INFO] hudi-spark-client .................................. SUCCESS [ 24.567 s]
[INFO] hudi-sync-common ................................... SUCCESS [  1.197 s]
[INFO] hudi-hive-sync ..................................... SUCCESS [  6.125 s]
[INFO] hudi-spark-datasource .............................. SUCCESS [  0.107 s]
[INFO] hudi-spark-common_2.12 ............................. SUCCESS [ 13.649 s]
[INFO] hudi-spark3_2.12 ................................... SUCCESS [ 11.451 s]
[INFO] hudi-spark_2.12 .................................... SUCCESS [ 45.515 s]
[INFO] hudi-utilities_2.12 ................................ SUCCESS [ 23.751 s]
[INFO] hudi-utilities-bundle_2.12 ......................... SUCCESS [ 51.554 s]
[INFO] hudi-cli ........................................... SUCCESS [ 32.192 s]
[INFO] hudi-java-client ................................... SUCCESS [  3.458 s]
[INFO] hudi-flink-client .................................. SUCCESS [ 12.356 s]
[INFO] hudi-spark2_2.12 ................................... SUCCESS [ 17.489 s]
[INFO] hudi-dla-sync ...................................... SUCCESS [  3.055 s]
[INFO] hudi-sync .......................................... SUCCESS [  0.131 s]
[INFO] hudi-hadoop-mr-bundle .............................. SUCCESS [  6.229 s]
[INFO] hudi-hive-sync-bundle .............................. SUCCESS [  2.009 s]
[INFO] hudi-spark3-bundle_2.12 ............................ SUCCESS [ 14.460 s]
[INFO] hudi-presto-bundle ................................. SUCCESS [  9.588 s]
[INFO] hudi-timeline-server-bundle ........................ SUCCESS [  7.371 s]
[INFO] hudi-hadoop-docker ................................. SUCCESS [  0.852 s]
[INFO] hudi-hadoop-base-docker ............................ SUCCESS [01:08 min]
[INFO] hudi-hadoop-namenode-docker ........................ SUCCESS [  0.178 s]
[INFO] hudi-hadoop-datanode-docker ........................ SUCCESS [  0.124 s]
[INFO] hudi-hadoop-history-docker ......................... SUCCESS [  0.112 s]
[INFO] hudi-hadoop-hive-docker ............................ SUCCESS [  0.541 s]
[INFO] hudi-hadoop-sparkbase-docker ....................... SUCCESS [  0.121 s]
[INFO] hudi-hadoop-sparkmaster-docker ..................... SUCCESS [  0.128 s]
[INFO] hudi-hadoop-sparkworker-docker ..................... SUCCESS [  0.181 s]
[INFO] hudi-hadoop-sparkadhoc-docker ...................... SUCCESS [  0.171 s]
[INFO] hudi-hadoop-presto-docker .......................... SUCCESS [  0.231 s]
[INFO] hudi-integ-test .................................... SUCCESS [01:05 min]
[INFO] hudi-integ-test-bundle ............................. SUCCESS [02:29 min]
[INFO] hudi-examples ...................................... SUCCESS [  9.459 s]
[INFO] hudi-flink_2.12 .................................... SUCCESS [  9.703 s]
[INFO] hudi-flink-bundle_2.12 0.9.0 ....................... SUCCESS [ 24.891 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:02 min
[INFO] Finished at: 2022-05-04T00:53:46+08:00
[INFO] ------------------------------------------------------------------------

关键maven配置

这份配置是不断报错，找资料最后解决的，所以大家编译的时候直接把我这份配置放在settings.xml里面就可以。

 <mirrors>
 <mirror>
    <id>alimaven</id>
    <name>aliyun maven</name>
    <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>aliyunmaven</id>
    <mirrorOf>*</mirrorOf>
    <name>阿里云spring插件</name>
    <url>https://maven.aliyun.com/repository/spring-plugin</url>
</mirror>
<mirror>
    <id>repo2</id>
    <name>Mirror from Maven Repo2</name>
    <url>https://repo.spring.io/plugins-release/</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>UK</id>
    <name>UK Central</name>
    <url>http://uk.maven.org/maven2</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>jboss- public-repository- group</id>
    <name>JBoss Public Repository Group</name>
    <url>http://repository.jboss.org/nexus/content/groups/public</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>CN</id>
    <name>OSChina Central</name>
    <url>http: //maven.oschina.net/content/groups/public/</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>google-maven-centralk</id>
    <name>GCS Maven Central mirror Asia Pacific</name>
    <url>https://maven-central-asia.storage-download.googleapis.com/maven2/</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>confluent</id>
    <name>confluent maven</name>
    <url>http://packages.confluent.io/maven/</url>
    <mirrorOf>confluent</mirrorOf>
</mirror>
<mirror>
    <id>jboss- public-repository-group</id>
    <name>JBoss Public Repository Group</name>
    <url>http://repository.jboss.org/nexus/content/groups/public</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>CN</id>
    <name>OSChina Central</name>
    <url>http://maven.oschina.net/content/groups/public/</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>google-maven- centralk</id>
    <name>GCS Maven Central mirror Asia Pacific</name>
    <url>https://maven-central-asia.storage-download.googleapis. com/maven2/</url>
    <mirrorOf>central</mirrorOf>
</mirror>
<mirror>
    <id>confluent</id>
    <name>confluent maven</name>
    <url>http://packages.confluent.io/maven/</url>
    <mirrorOf>confluent</mirrorOf>
</mirror>
 </mirrors>

后记

hudi编译会比其他的简单很多 ~~