Migrate Avro reader to arrow-avro and remove internal conversion code by getChan · Pull Request #17861 · apache/datafusion

getChan · 2025-10-01T14:10:36Z

Which issue does this PR close?

Closes Use arrow-avro for performance and improved type support #14097

Rationale for this change

DataFusion previously maintained custom Avro-to-Arrow conversion logic.
This PR migrates Avro reading to arrow-avro to align behavior with upstream Arrow and remove duplicated implementation.

What changes are included in this PR?

Switched DataFusion Avro reader path to arrow-avro (ReaderBuilder)
Removed internal/legacy Avro conversion paths that are no longer needed
Updated crate wiring to use arrow-avro and removed prior apache-avro dependency usage in affected paths
Updated Avro projection flow to use arrow-avro projection support
Added/updated upgrade documentation for Avro API and behavior changes

Are these changes tested?

Yes.

Added/updated Avro reader unit tests in datafusion/datasource-avro (including projection and timestamp logical types)
Updated SQL logic tests in datafusion/sqllogictest/test_files/avro.slt
Integration is covered by existing CI/test suites for affected crates

Are there any user-facing changes?

Yes.

DataFusionError::AvroError is removed.
From<apache_avro::Error> for DataFusionError is removed.
Re-export changed from datafusion::apache_avro to datafusion::arrow_avro.
Avro feature wiring changed:
- datafusion crate avro feature no longer enables datafusion-common/avro
- datafusion-proto crate avro feature no longer enables datafusion-common/avro
Avro decoding behavior now follows arrow-avro semantics, including:
- Avro string values being read as Arrow Binary in this path
- timestamp-* logical types read as UTC timezone-aware timestamps (Timestamp(..., Some("+00:00")))
- local-timestamp-* remaining timezone-naive (Timestamp(..., None))

Upgrade notes are documented in:
docs/source/library-user-guide/upgrading/53.0.0.md

alamb · 2025-10-02T10:10:04Z

❤️ amazing! Thank you @getChan
FYI @jecsand838 and @nathaniel-d-ef

alamb · 2025-10-19T20:04:52Z

Hi @getChan -- I am preparing to make an arrow release -- have you hit any blockers while integrating the new arrow-avro crate into DataFusion?

getChan · 2025-10-19T23:10:10Z

Hi @getChan -- I am preparing to make an arrow release -- have you hit any blockers while integrating the new arrow-avro crate into DataFusion?

No, not yet. Thanks for release.

nathaniel-d-ef · 2025-10-20T15:21:27Z

Thanks for jumping on this @getChan; let me know if I can help!

# Conflicts: # Cargo.lock # Cargo.toml # datafusion/common/Cargo.toml

alamb · 2025-10-29T11:08:03Z

FYI I merged the arrow 57 upgrade to DataFusion -- so if you rebase this PR against main you'll have access to the new arrow-avro crate

# Conflicts: # Cargo.lock

Co-authored-by: Connor Sanders <170039284+jecsand838@users.noreply.github.com>

# Conflicts: # Cargo.lock # datafusion/common/Cargo.toml

datafusion/datasource-avro/src/source.rs

docs/source/library-user-guide/upgrading/53.0.0.md

alamb

Thanks @getChan and @jecsand838 -- this is pretty epic work. The fact that all the tests pass is pretty incredible

I had a few small comments on the upgrade guide (I can help make them too if you want). Otherwise I think this is ready to go.

I also took the liberty of merging up from main to resolve a conflict

docs/source/library-user-guide/upgrading/53.0.0.md

Cargo.lock

# Conflicts: # Cargo.toml

alamb · 2026-03-25T19:26:08Z

Any other thoughts @jecsand838?

FYI @Igosuki -- this is a long time since you contributed the original avro reader. It is fun to see how far things have come

jecsand838

@alamb This LGTM!

alamb · 2026-03-26T21:28:29Z

Merged up to resolve a conflict

alamb · 2026-03-26T22:14:45Z

go go go go!

alamb · 2026-03-26T22:19:11Z

Epic work @getChan

draft: using arrow-avro

e27e1f8

github-actions bot added common Related to common crate datasource Changes to the datasource crate labels Oct 1, 2025

draft: using arrow-avro

027e79d

getChan added 3 commits October 19, 2025 17:15

Merge remote-tracking branch 'origin/main' into arrow-avro

2bf2c89

cargo fmt

2c7e364

check public API's can be replaced

7c9f976

getChan added 3 commits October 27, 2025 23:52

revert arrow dependency

7bc24ea

Merge remote-tracking branch 'origin/main' into arrow-avro

185ede5

# Conflicts: # Cargo.lock # Cargo.toml # datafusion/common/Cargo.toml

apply arrow-avro 57.0.0

51557fe

github-actions bot removed the common Related to common crate label Oct 27, 2025

timsaucer mentioned this pull request Oct 28, 2025

Switch from deprecated xz2 to libzma #18331

Closed

getChan added 2 commits October 30, 2025 00:36

BufReader

8e12f47

remove apache-avro dependencies

83ec01d

github-actions bot added core Core DataFusion crate common Related to common crate proto Related to proto crate labels Oct 29, 2025

getChan added 8 commits October 30, 2025 01:16

Merge remote-tracking branch 'origin/main' into arrow-avro

f37fcd3

# Conflicts: # Cargo.lock

remove apache-avro dependencies

b12f563

remove apache-avro dependencies

581ecae

fix test result

74792ff

pass projected schema to arrow-avro

151ea95

testing submodule checkout

78df065

taplo format

c7b3b63

replace with arrow-avro

2556591

getChan and others added 10 commits March 17, 2026 00:44

Apply suggestions from code review - strip mtdt

f42726d

Co-authored-by: Connor Sanders <170039284+jecsand838@users.noreply.github.com>

avro: strip metadata recursively for union types

36cf9e2

avro: preserve projection compatibility via name-based schema coercion

0019184

avro: restore missing-column test behavior to pre-17861 semantics

6831da1

avro: remap projection pushdown to writer schema ordinals

a0755b7

avro: restore compatibility docs and sqllogictest expectations

3b67bd8

Merge branch 'main' into arrow-avro

8805368

# Conflicts: # Cargo.lock # datafusion/common/Cargo.toml

avro: fix clippy in projection coercion path

43c6b6c

fix: remove unintended ahash workspace dependency changes

d5d615f

Merge branch 'main' into arrow-avro

bb0325f

alamb mentioned this pull request Mar 24, 2026

Release DataFusion 54.0.0 (Apr 2026 / May 2026) #21080

Open

19 tasks

adriangb reviewed Mar 24, 2026

View reviewed changes

datafusion/datasource-avro/src/source.rs Outdated Show resolved Hide resolved

alamb reviewed Mar 24, 2026

View reviewed changes

docs/source/library-user-guide/upgrading/53.0.0.md Outdated Show resolved Hide resolved

Merge remote-tracking branch 'apache/main' into arrow-avro

b45ac7f

alamb approved these changes Mar 24, 2026

View reviewed changes

getChan added 6 commits March 25, 2026 21:19

docs: move avro upgrade note to 54 guide

d7f1d31

docs: refine avro upgrade note wording

7958154

test: remove avro sqllogictest conflict marker

da7d51a

Merge branch 'main' into arrow-avro

6c4362e

# Conflicts: # Cargo.toml

build: refresh Cargo.lock

eadd337

replace coerce_batch_to_schema to BatchAdapter

471abd3

jecsand838 approved these changes Mar 26, 2026

View reviewed changes

Merge remote-tracking branch 'apache/main' into arrow-avro

add2b3b

alamb added this pull request to the merge queue Mar 26, 2026

Merged via the queue into apache:main with commit 627faba Mar 26, 2026
34 checks passed

Conversation

getChan commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Oct 2, 2025

Uh oh!

alamb commented Oct 19, 2025

Uh oh!

getChan commented Oct 19, 2025

Uh oh!

nathaniel-d-ef commented Oct 20, 2025

Uh oh!

alamb commented Oct 29, 2025

Uh oh!

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb commented Mar 25, 2026

Uh oh!

jecsand838 left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Mar 26, 2026

Uh oh!

alamb commented Mar 26, 2026

Uh oh!

Uh oh!

alamb commented Mar 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

getChan commented Oct 1, 2025 •

edited

Loading