编译hive-3.1.3（3.0.0）和编译tez-0.10.2适配hadoop-3.3.4以及hadoop-3.2.2（附安装包）

一.搭建编译环境

1.准备一台linux虚拟机，我安装 Centos8.0 系统（带桌面环境）

安装一些基础环境需要的依赖

yum -y install gcc gcc-c++ kernel-devel libtirpc-devel
yum -y install rpm-build
yum -y install net-tools
yum -y install openssl openssl-devel patch
yum -y install wget
yum install -y bash-completion
yum install -y unzip zip
yum install -y openssh-clients
yum install -y openssh-server
yum  -y install rsync  

2.安装 JDK

1.下载jdk,选择版本
     下载地址 https://www.openlogic.com/openjdk-downloads
      我选择的是 openlogic-openjdk-8u272-b10-linux-x64.tar.gz
      
2.卸载现有 JDK 
     [appview@master software]$ sudo rpm -qa | grep -i java | xargs -n1 sudo rpm -e --nodeps 
     
3.将 JDK 上传到虚拟机的/opt/software 文件夹下面 
​
4.解压 JDK 到/opt/module 目录下 
    [appview@master software]$  tar -zxvf openlogic-openjdk-8u272-b10-linux-x64.tar.gz -C /opt/module/ 
        #大写的C 指定解压目录
        
5.配置 JDK 环境变量 
   （1）[appview@master software]$  vim ~/.bashrc 
添加如下内容，然后保存（:wq）退出 
#JAVA_HOME 
export JAVA_HOME=/opt/module/openjdk-8u272 
export PATH=$PATH:$JAVA_HOME/bin 
 （2） 让环境变量生效 
   [appview@master software]$ source ~/.bashrc  
   
6.测试 JDK 是否安装成功 
[appview@master software]$ java -version 
如果能看到以下结果，则 Java 正常安装 
openjdk version "1.8.0-272"
OpenJDK Runtime Environment (build 1.8.0-272-b10)
OpenJDK 64-Bit Server VM (build 25.71-b10, mixed mode)

3.安装maven

1.将 Maven 安装包上传到虚拟机/opt/software 目录（下载地址随便百度一下，有点懒了~^^~）
​
2.解压 Maven 到/opt/module 目录下 
   [appview@master software] tar -zxvf apache-maven-3.8.8-bin.tar.gz -C /opt/module/ 
3.配置 Maven 环境变量 
 (1)编辑  ~/.bashrc 文件 
    [appview@master software]$  vim ~/.bashrc 
    追加以下内容 
# MAVEN_HOME 
export MAVEN_HOME=/opt/module/maven-3.8.8 
export PATH=$PATH:$MAVEN_HOME/bin 
 （2） 让环境变量生效 
[appview@master software]$ source ~/.bashrc 
​
4.检测 Maven 是否安装成功 
[appview@master software]$ mvn -version 
如果能看到以下结果，则 Maven 正常安装 
Apache Maven 3.8.8 (4c87b05d9aedce574290d1acc98575ed5eb6cd39)
Maven home: /opt/moudle/maven-3.8.8
Java version: 1.8.0-272, vendor: OpenLogic-OpenJDK, runtime: /opt/moudle/openjdk-8u272/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "4.18.0-80.el8.x86_64", arch: "amd64", family: "unix"
​
5.配置仓库镜像 
(1) 修改 Maven 配置文件 
[appview@master software]$ vim /opt/module/apache-maven3.8.8/conf/settings.xml
​
(2) 在节点中增加以下内容 

      alimaven
      aliyun maven
      http://maven.aliyun.com/nexus/content/groups/public/
      central

​

    aliyunmaven
    *
    阿里云公共仓库
    https://maven.aliyun.com/repository/public

​

    aliyunmaven
    *
    spring-plugin
    https://maven.aliyun.com/repository/spring-plugin
 
​
 
    repo2
    Mirror from Maven Repo2
    https://repo.spring.io/plugins-release/
    central
 
​

4.安装 IDEA

步骤和上面的操作一样，就不详细列举了，下载、上传、解压、配置环境变量、最后source一下环境变量让启生效

启动 IDEA（在图形化界面启动）

[appview@master software]$ nohup /opt/module/idea-IU211.7628.21/bin/idea.sh >/dev/null 2>&1 &

打开配置IDEA（在图形化界面启动 ）点击feile----settings-----Build,Execution,Deployment --Build-toos-- maven 配置maven路径

同样file—Project Structure 配置SDK 选择以上安装的java_home路径

5.hive代码拉取

可以选择安装git拉取代码,或者下载到本地上传linux,我选择的是后一种，因为之前我用centos7的时候已经下得源码包了，方便自己想删就删再解压就OK了，现在换成centos8,重新部署一下编译环境写下得操作文档，我发现写操作文档比安装部署还累

将源码包解压到指定目录，打开idea，打开项目，粘贴解压的hive源码路径，先打开终端命令行terminal,执行以下命令，测试以下编译环境可行性

输入打包命令

[appview@master apache-hive-3.1.3-src]$ mvn clean package -Pdist -DskipTests -Dmaven.javadoc.skip=true

打包成功的标志如下：

[INFO] Hive HPL/SQL ....................................... SUCCESS [ 11.966 s]
[INFO] Hive Streaming ..................................... SUCCESS [  6.496 s]
[INFO] Hive Llap External Client .......................... SUCCESS [  6.153 s]
[INFO] Hive Shims Aggregator .............................. SUCCESS [  0.104 s]
[INFO] Hive Kryo Registrator .............................. SUCCESS [  6.021 s]
[INFO] Hive TestUtils ..................................... SUCCESS [  0.168 s]
[INFO] Hive Packaging ..................................... SUCCESS [01:40 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  14:37 min
[INFO] Finished at: 2024-04-14T12:50:17+08:00
[INFO] ------------------------------------------------------------------------
[appview@master apache-hive-3.1.3-src]$ 

6.编译环境过程遇到的坑以及解决办法

（1）因为编辑配置maven的/opt/module/apache-maven3.8.8/conf/settings.xml仓库有点问题报如下错误，

appview@master apache-hive-3.1.3-src]$ mvn clean package -Pdist -DskipTests -Dmaven.javadoc.skip=true
[INFO] Scanning for projects...
Downloading from repo2: https://repo.spring.io/plugins-release/org/apache/apache/18/apache-18.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.apache.hive:hive:3.1.3: Could not transfer artifact org.apache:apache:pom:18 from/to repo2 (https://repo.spring.io/plugins-release/): authentication failed for https://repo.spring.io/plugins-release/org/apache/apache/18/apache-18.pom, status: 401 Unauthorized and 'parent.relativePath' points at wrong local POM @ line 17, column 11
 @ 
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project org.apache.hive:hive:3.1.3 (/opt/software/scrmake/apache-hive-3.1.3-src/pom.xml) has 1 error
[ERROR]     Non-resolvable parent POM for org.apache.hive:hive:3.1.3: Could not transfer artifact org.apache:apache:pom:18 from/to repo2 (https://repo.spring.io/plugins-release/): authentication failed for https://repo.spring.io/plugins-release/org/apache/apache/18/apache-18.pom, status: 401 Unauthorized and 'parent.relativePath' points at wrong local POM @ line 17, column 11 -> [Help 2]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException

（2）查询了很多资料有的说加以下内容能解决，我本来就加了的，博友说多试了几次就成功或重启idea，我就多次了几次也都没有用甚至删包重新打开项目都不行，后面我查询资料阿里云的仓库资料发现了以下这个仓库加上就可以继续往下走了

[appview@master software]$ vim /opt/module/apache-maven3.8.8/conf/settings.xml
###  在节点中增加以下内容 
​

    aliyunmaven
    *
    spring-plugin
    https://maven.aliyun.com/repository/spring-plugin
 
 
    repo2
    Mirror from Maven Repo2
    https://repo.spring.io/plugins-release/
    central
 

后面我查询资料阿里云的仓库资料发现了以下这个仓库加上就可以继续往下走了

[appview@master software]$ vim /opt/module/apache-maven3.8.8/conf/settings.xml
###  在节点中增加以下内容 

      alimaven
      aliyun maven
      http://maven.aliyun.com/nexus/content/groups/public/
      central

​

不过马上又遇到新的问题了，提示找不到这个pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar

[INFO] Hive Packaging ..................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  21.756 s
[INFO] Finished at: 2024-04-14T11:30:41+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project hive-upgrade-acid: Could not resolve dependencies for project org.apache.hive:hive-upgrade-acid:jar:3.1.3: Could not find artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde in aliyunmaven (https://maven.aliyun.com/repository/public) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[appview@master apache-hive-3.1.3-src]$

网友说的解决办法是加入上面6.~~~~（2），多次几次或者下载本地包上传到目标目录，但却没有相对的目录，我找了好久的目录才发现是我们本地仓库的目标路径，一切都在报错内容中，所以不要忽略报错信息，因为报错中就含有解决问题的方法：如上面内容：[ERROR] Failed to execute goal on project hive-upgrade-acid: Could not resolve dependencies for project org.apache.hive:hive-upgrade-acid:jar:3.1.3: Could not find artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde in aliyunmaven

路径就在这条信息中，首先找到本地maven的仓库，我的默认为/home/appview/.m2/repository，我在里面找到了这个jar包，如下

，把这两个文件的后缀名.lastUpdated去掉,或者直接上传这两个文件到该路径下也ok。

[appview@master 5.1.5-jhyde]$ pwd
/home/appview/.m2/repository/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde
[appview@master 5.1.5-jhyde]$
[appview@master 5.1.5-jhyde]$
[appview@master 5.1.5-jhyde]$
[appview@master 5.1.5-jhyde]$ ll
总用量 8
-rw-rw-r-- 1 appview appview  404 4月  14 12:30 pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar.lastUpdated
-rw-rw-r-- 1 appview appview 1020 4月  14 12:30 pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom.lastUpdated
[appview@master 5.1.5-jhyde]$ mv pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar.lastUpdated pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
​
[appview@master 5.1.5-jhyde]$ mv pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom.lastUpdated pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom

然后我再执行编译打包命令就成功了

[appview@master apache-hive-3.1.3-src]$ mvn clean package -Pdist -DskipTests -Dmaven.javadoc.skip=true

 Hive Service ....................................... SUCCESS [ 15.281 s]
[INFO] Hive Accumulo Handler .............................. SUCCESS [  9.926 s]
[INFO] Hive JDBC .......................................... SUCCESS [ 44.256 s]
[INFO] Hive Beeline ....................................... SUCCESS [  8.778 s]
[INFO] Hive CLI ........................................... SUCCESS [  6.941 s]
[INFO] Hive Contrib ....................................... SUCCESS [  4.701 s]
[INFO] Hive Druid Handler ................................. SUCCESS [ 33.979 s]
[INFO] Hive HBase Handler ................................. SUCCESS [ 17.064 s]
[INFO] Hive JDBC Handler .................................. SUCCESS [  4.867 s]
[INFO] Hive HCatalog ...................................... SUCCESS [  1.189 s]
[INFO] Hive HCatalog Core ................................. SUCCESS [  8.743 s]
[INFO] Hive HCatalog Pig Adapter .......................... SUCCESS [  7.251 s]
[INFO] Hive HCatalog Server Extensions .................... SUCCESS [  6.315 s]
[INFO] Hive HCatalog Webhcat Java Client .................. SUCCESS [  6.714 s]
[INFO] Hive HCatalog Webhcat .............................. SUCCESS [ 23.539 s]
[INFO] Hive HCatalog Streaming ............................ SUCCESS [  9.005 s]
[INFO] Hive HPL/SQL ....................................... SUCCESS [ 11.966 s]
[INFO] Hive Streaming ..................................... SUCCESS [  6.496 s]
[INFO] Hive Llap External Client .......................... SUCCESS [  6.153 s]
[INFO] Hive Shims Aggregator .............................. SUCCESS [  0.104 s]
[INFO] Hive Kryo Registrator .............................. SUCCESS [  6.021 s]
[INFO] Hive TestUtils ..................................... SUCCESS [  0.168 s]
[INFO] Hive Packaging ..................................... SUCCESS [01:40 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  14:37 min
[INFO] Finished at: 2024-04-14T12:50:17+08:00
[INFO] ------------------------------------------------------------------------
[appview@master apache-hive-3.1.3-src]$ 

7.编译环境已经搭建完成了

然后根据自己需求对应的版本，编译选择对应hive版本修改pom.xml里面的版本对应的Hadoop.version改为3.3.4、spark.version、tez.version改为0.10.2、guava-19.0 改为guava-27.0-jre、等版本，以及其他需要的软件版本好等，在执行编译打包命令，一步步解决问题直到打包成功出现如下，方能成功

[INFO] Hive Kryo Registrator .............................. SUCCESS [  6.021 s]
[INFO] Hive TestUtils ..................................... SUCCESS [  0.168 s]
[INFO] Hive Packaging ..................................... SUCCESS [01:40 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

今天没有时间，详细写出编译的步骤了，后期有时间的话重新上编译的详细过程以及ambari搭建hadoop集群以及高可用、kafka集群、redis集群、elasticsearch集群，doris集群等更多的技术，下面特附上两套已经编译好的的hive-3.1.3、hadoop-3.3.4、tez-0.10.2以及hive-3.0.0、hadoop-3.2.2、tez-0.10.2 的编译包，为什么是两套呢，因为公司对操作系统的不同，软件对应的版本也不同，第一套对应的是X86_64的，后面一套是arm操作系统的，其实都可以用的，不过是公司规定，所以选择的版本不同而已，下面附属安装包，其中没有hadoop-3.2.2的安装包自己网上下，hadoop不需要编译，后面还会上hive on tez 的操作步骤，如急需请留言我看到会及时回复的。

8.结语

之所以写下这一篇文章呢，主要是给予我这样的遇到同样问题的朋友们节约一些时间，提高效率，同时记录一下自己的学习历程，当然有不足之处，还请各位朋友多多指点。感谢！！！！！！！！！！！

安装包链接

hadoop-3.2.2的链接：百度网盘 请输入提取码 提取码：qwer

hadoop-3.3.4的链接：百度网盘 请输入提取码 提取码：qwer