Skip to content

Conversation

@010Soham
Copy link
Contributor

What changed

  • Add a client-side validation error when a PyArrow null type is used with format version < 3, including the field path and a hint to use a concrete type or format-version 3.
  • Add a unit test to verify the clear error message when converting a PyArrow schema with a null field.

Why

  • Prevents misleading REST errors like “Cannot parse type string… unknown” and points directly to the offending field.

Testing

  • make lint
  • uv run pytest tests/catalog/test_base.py -k rejects_null_type -v
  • make test (timed out at ~42%)

Closes #2539

Copy link
Contributor

@kevinjqliu kevinjqliu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks pa.null() has been a long standing issue. This should provide some clarity

Comment on lines +215 to +217
assert "Null type" in message
assert "n1" in message
assert "format-version=3" in message
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the full error message

'Null type (pa.null()) is not supported in Iceberg format version 2. Field: n1. Use a concrete type (string, int, boolean, etc.) or set format-version=3.'

@kevinjqliu kevinjqliu requested a review from Fokko December 24, 2025 18:48
Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this check, I think this will help a lot of users.

We can merge this after resolving @geruh's comment 👍

@010Soham
Copy link
Contributor Author

010Soham commented Jan 8, 2026

@Fokko
Have added the changes prescribed @geruh
let me know if that sounds good

@kevinjqliu kevinjqliu merged commit 3855f64 into apache:main Jan 8, 2026
10 checks passed
@kevinjqliu
Copy link
Contributor

Thanks for the PR @010Soham and @geruh @Fokko for the review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

error message for failed table creation due to null columns is misleading

4 participants