chore: Enable Comet shuffle with AQE coalesce partitions

[viirya]

Which issue does this PR close?

Closes #387.

Rationale for this change

What changes are included in this PR?

How are these changes tested?

[viirya]

Some Spark tests need to be updated. Marked this as a draft for now.

[viirya]

All Comet and TPCDS, TPCH tests are passed. Next, I will fix Spark SQL tests.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 25.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 33.99%. Comparing base (4fe43ad) to head (3562b39).
Report is 7 commits behind head on main.

Files	Patch %	Lines
...ain/java/org/apache/comet/parquet/BatchReader.java	0.00%	1 Missing ⚠️
...l/comet/execution/arrow/CometArrowConverters.scala	0.00%	1 Missing ⚠️
...org/apache/comet/CometSparkSessionExtensions.scala	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #651      +/-   ##
============================================
+ Coverage     33.94%   33.99%   +0.05%     
+ Complexity      874      858      -16     
============================================
  Files           112      112              
  Lines         42916    42875      -41     
  Branches       9464     9461       -3     
============================================
+ Hits          14567    14576       +9     
+ Misses        25379    25304      -75     
- Partials       2970     2995      +25     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[viirya]

For the negative refcount issue reported by Java Arrow, I cannot reproduce it locally or even on internal CI (different platform other than ubuntu).

I'm not sure if it is related to the ubuntu platform the GitHub Actions uses. Because GitHub Actions only supports ubuntu for linux, I cannot try other linux os. Currently I can only try to use older ubuntu (20.04) and see it helps.

[viirya]

Close in favor of #834.