HIVE-27030 : Upgrade HIVE to Hadoop 3.3.4

[amanraj2520]

JIRA link : https://issues.apache.org/jira/browse/HIVE-27030

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

[amanraj2520]

@abstractdog @ayushtkn @zabetak @kgyrtkirk Can you please review this.

[zabetak]

LGTM, will merge soon!

[ayushtkn]

Is it working with Tez, last time when we tried to move to 3.3.3, we moved back because of Tez.
It got me a green build though.
#3279 (comment)

[amanraj2520]

@ayushtkn You are right when we try to integrate it with Tez, things can break. I agree fully (as you pointed out). But in that case, we can ignore this commit in Hive 4.x release. We can still have this PR merged to Hive master I think.

The failure you are talking about is because Tez 0.10.2 uses 3.3.1 Hadoop and Hive will use 3.3.4 if this PR goes through. This is because Tez 0.10.2 brings in certain older jars in the classpath while a Hive job is executed as transitive dependencies for Hadoop 3.3.1.

This was exactly why I was pushing for Tez 0.10.3 release which will have Hadoop 3.3.4 as we were discussing on this thread - apache/tez#213. Personally, I have added internal commits in Tez 0.10.2 just for testing in my local cluster where I just upgraded Hadoop to 3.3.4 (in Tez 0.10.2) and used that SNAPSHOT jar in the Hive master pom. Those errors were resolved. So basically, what I am again suggesting is that we can have a Tez 0.10.3 release (with Hadoop 3.3.4), if everyone agrees.
So that would mean Hive master, Tez 0.10.3 and Hadoop 3.3.4 can form a proper stack for end to end integration. Also, this will help in the ongoing hive-3.2.0 release that we are putting our efforts to where we are trying to use Hadoop 3.3.4 as the latest stack.

Please let me know your opinion.

cc @zabetak @abstractdog

[ayushtkn]

But in that case, we can ignore this commit in Hive 4.x release. We can still have this PR merged to Hive master I think.

This is not how things work, 4.x release is going to be from master branch, so why we will merge this. Secondly, a lot of developer develop on the master branch, we can't make the master 'knowingly' unstable.

Secondly, This is dupe to #3578, and has a mention on the Tez ticket as well

[amanraj2520]

Got it Ayush. Will see if we can make a Tez release 0.10.3 first and then re-open this if required. Thanks for the review.

[amanraj2520]

Closing this for now. Will reopen if required. @zabetak @ayushtkn @abstractdog