Skip to content

Fix read from multiple s3 regions #1453

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kevinjqliu merged 26 commits into from
Jan 6, 2025
Merged

Fix read from multiple s3 regions #1453

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
7f52a1a
Take netloc into account for s3 filesystem when calling `_initialize_fs`
jiakai-li Dec 19, 2024
b53e89f
Fix unit test for s3 fileystem
jiakai-li Dec 20, 2024
7699a76
Merge branch 'main' into fix-read-from-multiple-s3-regions
jiakai-li Dec 21, 2024
eb5e491
Update ArrowScan to use different FileSystem per file
jiakai-li Dec 21, 2024
0c61ac8
Add unit test for `PyArrorFileIO.fs_by_scheme` cache behavior
jiakai-li Dec 21, 2024
327dbac
Add error handling
jiakai-li Dec 22, 2024
b4fccf2
Update tests/io/test_pyarrow.py
jiakai-li Dec 23, 2024
48bb811
Update `s3.region` document and a test case
jiakai-li Dec 23, 2024
8404e6b
Add test case for `PyArrowFileIO.new_input` multi region
jiakai-li Dec 24, 2024
53951f5
Shuffle code location for better maintainability
jiakai-li Dec 24, 2024
51fb6ff
Comment for future integration test
jiakai-li Dec 24, 2024
0cd06c4
Typo fix
jiakai-li Dec 24, 2024
64fbdab
Document wording
jiakai-li Dec 24, 2024
37d9ec2
Add warning when the bucket region for a file cannot be resolved (for…
jiakai-li Dec 29, 2024
74e78ae
Merge branch 'main' into fix-read-from-multiple-s3-regions
jiakai-li Dec 29, 2024
4ff4a7d
Fix code linting
jiakai-li Dec 29, 2024
b56f2ee
Update mkdocs/docs/configuration.md
jiakai-li Jan 3, 2025
2a4cee1
Merge branch 'main' into fix-read-from-multiple-s3-regions
jiakai-li Jan 4, 2025
9cc3a30
Code refactoring
jiakai-li Jan 4, 2025
ba5ef76
Unit test
jiakai-li Jan 4, 2025
8f06a15
Code refactoring
jiakai-li Jan 4, 2025
e5cac02
Test cases
jiakai-li Jan 4, 2025
9652baf
Code format
jiakai-li Jan 4, 2025
bc2adc7
Merge branch 'main' into fix-read-from-multiple-s3-regions
jiakai-li Jan 6, 2025
4b83fc0
Code tidy-up
jiakai-li Jan 6, 2025
7f207bf
Update pyiceberg/io/pyarrow.py
jiakai-li Jan 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 15 additions & 15 deletions mkdocs/docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,21 +102,21 @@ For the FileIO there are several configuration options available:

<!-- markdown-link-check-disable -->

| Key | Example | Description |
|----------------------|----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| s3.endpoint | <https://10.0.19.25/> | Configure an alternative endpoint of the S3 service for the FileIO to access. This could be used to use S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud. |
| s3.access-key-id | admin | Configure the static access key id used to access the FileIO. |
| s3.secret-access-key | password | Configure the static secret access key used to access the FileIO. |
| s3.session-token | AQoDYXdzEJr... | Configure the static session token used to access the FileIO. |
| s3.role-session-name | session | An optional identifier for the assumed role session. |
| s3.role-arn | arn:aws:... | AWS Role ARN. If provided instead of access_key and secret_key, temporary credentials will be fetched by assuming this role. |
| s3.signer | bearer | Configure the signature version of the FileIO. |
| s3.signer.uri | <http://my.signer:8080/s3> | Configure the remote signing uri if it differs from the catalog uri. Remote signing is only implemented for `FsspecFileIO`. The final request is sent to `<s3.signer.uri>/<s3.signer.endpoint>`. |
| s3.signer.endpoint | v1/main/s3-sign | Configure the remote signing endpoint. Remote signing is only implemented for `FsspecFileIO`. The final request is sent to `<s3.signer.uri>/<s3.signer.endpoint>`. (default : v1/aws/s3/sign). |
| s3.region | us-west-2 | Sets the region of the bucket |
| s3.proxy-uri | <http://my.proxy.com:8080> | Configure the proxy server to be used by the FileIO. |
| s3.connect-timeout | 60.0 | Configure socket connection timeout, in seconds. |
| s3.force-virtual-addressing | False | Whether to use virtual addressing of buckets. If true, then virtual addressing is always enabled. If false, then virtual addressing is only enabled if endpoint_override is empty. This can be used for non-AWS backends that only support virtual hosted-style access. |
| Key | Example | Description |
|----------------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| s3.endpoint | <https://10.0.19.25/> | Configure an alternative endpoint of the S3 service for the FileIO to access. This could be used to use S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud. |
| s3.access-key-id | admin | Configure the static access key id used to access the FileIO. |
| s3.secret-access-key | password | Configure the static secret access key used to access the FileIO. |
| s3.session-token | AQoDYXdzEJr... | Configure the static session token used to access the FileIO. |
| s3.role-session-name | session | An optional identifier for the assumed role session. |
| s3.role-arn | arn:aws:... | AWS Role ARN. If provided instead of access_key and secret_key, temporary credentials will be fetched by assuming this role. |
| s3.signer | bearer | Configure the signature version of the FileIO. |
| s3.signer.uri | <http://my.signer:8080/s3> | Configure the remote signing uri if it differs from the catalog uri. Remote signing is only implemented for `FsspecFileIO`. The final request is sent to `<s3.signer.uri>/<s3.signer.endpoint>`. |
| s3.signer.endpoint | v1/main/s3-sign | Configure the remote signing endpoint. Remote signing is only implemented for `FsspecFileIO`. The final request is sent to `<s3.signer.uri>/<s3.signer.endpoint>`. (default : v1/aws/s3/sign). |
| s3.region | us-west-2 | Configure the default region used to initialize an `S3FileSystem`. `PyArrowFileIO` attempts to automatically resolve the region for each S3 bucket, falling back to this value if resolution fails. |
| s3.proxy-uri | <http://my.proxy.com:8080> | Configure the proxy server to be used by the FileIO. |
| s3.connect-timeout | 60.0 | Configure socket connection timeout, in seconds. |
| s3.force-virtual-addressing | False | Whether to use virtual addressing of buckets. If true, then virtual addressing is always enabled. If false, then virtual addressing is only enabled if endpoint_override is empty. This can be used for non-AWS backends that only support virtual hosted-style access. |

<!-- markdown-link-check-enable-->

Expand Down
Loading
Loading