feat: sort file groups by statistics during sort pushdown

[adriangb]

Summary

• Sort files within each file group by min/max statistics during sort pushdown to better align with the requested ordering
• When files are non-overlapping and within-file ordering is guaranteed (Parquet returns Exact), the SortExec is completely eliminated
• When files overlap, best-effort statistics-based reordering is applied with SortExec retained for correctness (Inexact)
• ParquetSource::try_pushdown_sort now returns Exact when the file's natural ordering already satisfies the request, enabling sort elimination for the same-direction case
• Add SLT integration tests covering non-overlapping sort elimination, overlapping files, reverse scan with mixed file naming, and multi-group merging

Related to #19724

Test plan

• cargo test -p datafusion-sqllogictest --test sqllogictests -- sort_pushdown passes
• cargo test -p datafusion-sqllogictest --test sqllogictests -- parquet_sorted_statistics passes
• New SLT tests verify EXPLAIN plans show correct optimizer behavior (sort elimination, SortExec retention, reverse_row_groups, file ordering)
• New SLT tests verify query result correctness for all scenarios

🤖 Generated with Claude Code

[adriangb]

I'm going to wait to sequence this after #20329. I think this also fixes the bug but I'd rather do bugfix and feature independently.

[adriangb]

run benchmark clickbench_partitioned

[alamb-ghbot]

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing sort-groups (29ee72e) to a5f490e diff using: clickbench_partitioned
Results will be posted here when complete

[alamb-ghbot]

🤖: Benchmark completed

Details

Comparing HEAD and sort-groups
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ sort-groups ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     2.57 ms │     2.64 ms │     no change │
│ QQuery 1  │    49.81 ms │    51.68 ms │     no change │
│ QQuery 2  │   163.21 ms │   170.26 ms │     no change │
│ QQuery 3  │   164.68 ms │   170.01 ms │     no change │
│ QQuery 4  │  1118.70 ms │  1176.20 ms │  1.05x slower │
│ QQuery 5  │  1304.76 ms │  1420.54 ms │  1.09x slower │
│ QQuery 6  │     6.61 ms │     6.80 ms │     no change │
│ QQuery 7  │    55.17 ms │    55.77 ms │     no change │
│ QQuery 8  │  1571.66 ms │  1604.62 ms │     no change │
│ QQuery 9  │  1958.40 ms │  2008.11 ms │     no change │
│ QQuery 10 │   341.86 ms │   361.48 ms │  1.06x slower │
│ QQuery 11 │   397.43 ms │   416.61 ms │     no change │
│ QQuery 12 │  1243.33 ms │  1307.39 ms │  1.05x slower │
│ QQuery 13 │  2051.70 ms │  2126.89 ms │     no change │
│ QQuery 14 │  1243.47 ms │  1305.09 ms │     no change │
│ QQuery 15 │  1280.20 ms │  1336.19 ms │     no change │
│ QQuery 16 │  2690.15 ms │  2789.21 ms │     no change │
│ QQuery 17 │  2667.07 ms │  2732.10 ms │     no change │
│ QQuery 18 │  5484.33 ms │  5328.09 ms │     no change │
│ QQuery 19 │   130.29 ms │   131.86 ms │     no change │
│ QQuery 20 │  1874.78 ms │  1914.24 ms │     no change │
│ QQuery 21 │  2157.82 ms │  2310.22 ms │  1.07x slower │
│ QQuery 22 │  3721.26 ms │  3963.04 ms │  1.06x slower │
│ QQuery 23 │ 11896.30 ms │ 12766.66 ms │  1.07x slower │
│ QQuery 24 │   203.36 ms │   215.42 ms │  1.06x slower │
│ QQuery 25 │   421.89 ms │   465.67 ms │  1.10x slower │
│ QQuery 26 │   202.41 ms │   199.01 ms │     no change │
│ QQuery 27 │  2722.68 ms │  2720.60 ms │     no change │
│ QQuery 28 │ 25510.30 ms │ 25425.30 ms │     no change │
│ QQuery 29 │  1061.23 ms │  1081.94 ms │     no change │
│ QQuery 30 │  1277.06 ms │  1328.88 ms │     no change │
│ QQuery 31 │  1385.31 ms │  1425.15 ms │     no change │
│ QQuery 32 │  5106.58 ms │  4828.95 ms │ +1.06x faster │
│ QQuery 33 │  5655.99 ms │  5886.97 ms │     no change │
│ QQuery 34 │  6464.05 ms │  6298.75 ms │     no change │
│ QQuery 35 │  2069.78 ms │  2076.51 ms │     no change │
│ QQuery 36 │   186.71 ms │   193.12 ms │     no change │
│ QQuery 37 │    71.32 ms │    75.21 ms │  1.05x slower │
│ QQuery 38 │   109.19 ms │   114.37 ms │     no change │
│ QQuery 39 │   329.77 ms │   344.86 ms │     no change │
│ QQuery 40 │    43.15 ms │    38.72 ms │ +1.11x faster │
│ QQuery 41 │    37.98 ms │    35.94 ms │ +1.06x faster │
│ QQuery 42 │    32.57 ms │    30.90 ms │ +1.05x faster │
└───────────┴─────────────┴─────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary          ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)          │ 96466.87ms │
│ Total Time (sort-groups)   │ 98241.97ms │
│ Average Time (HEAD)        │  2243.42ms │
│ Average Time (sort-groups) │  2284.70ms │
│ Queries Faster             │          4 │
│ Queries Slower             │         10 │
│ Queries with No Change     │         29 │
│ Queries with Failure       │          0 │
└────────────────────────────┴────────────┘