feat: Add Amazon S3 Vectors document store integration

[dotKokott]

Related Issues

• fixes Amazon S3 Vectors (DocStore) #2110

Proposed Changes:

Adds an Amazon S3 Vectors document store integration — a serverless vector storage capability native to S3.

Components:

• S3VectorsDocumentStore — full DocumentStore protocol (write, count, filter, delete)
• S3VectorsEmbeddingRetriever — embedding-based retrieval with server-side metadata filtering

Key design decisions:

• Content stored as non-filterable metadata (AWS-recommended pattern for large text)
• Cosine distance converted to similarity score (1 - distance) for Haystack convention
• Blob data uses base64 encoding for round-trip fidelity
• filter_documents() uses list_vectors(returnData=True, returnMetadata=True) with client-side filtering (warning logged) since S3 Vectors has no standalone filter API
• Batch existence checks for DuplicatePolicy.SKIP/NONE (batches of 100)

Known limitations (documented in README):

• top_k capped at 100 (service limit)
• query_vectors does not return embedding data
• 40KB total metadata per vector, 2KB filterable
• Only float32, cosine/euclidean, eventual consistency

How did you test it?

• 26 unit tests — serialization, score conversion, filter conversion, duplicate policy logic, document conversion (mocked boto3)
• 12 integration tests — full lifecycle against live AWS S3 Vectors, with pytestmark credential guard for CI
• hatch run test:all, hatch run fmt, hatch run test:types
• Example script (examples/example.py) verified against live AWS

Notes for the reviewer

This PR was fully generated with an AI assistant. I have reviewed the changes and run the relevant tests.
Structure and test style follow the Pinecone integration pattern.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

[dotKokott]

CI: Integration tests need AWS credential setup

The integration tests currently run unconditionally in CI with no AWS credentials configured. The tests have a pytestmark = pytest.mark.skipif(not _aws_credentials_available(), ...) guard so they silently skip (0 collected), but this means:

• Integration tests never actually run in CI — only locally by developers with AWS credentials
• The "combined" coverage badge will just reflect unit test coverage

What needs to happen

The workflow should match the amazon_bedrock.yml pattern — add an OIDC role assumption step and gate the integration test run on its success:

# Do not authenticate on PRs from forks and on PRs created by dependabot
- name: AWS authentication
  id: aws-auth
  if: github.event_name == 'schedule' || (github.event.pull_request.head.repo.full_name == github.repository && !startsWith(github.event.pull_request.head.ref, 'dependabot/'))
  uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37
  with:
    aws-region: us-east-1
    role-to-assume: ${{ secrets.AWS_S3_VECTORS_CI_ROLE_ARN }}

- name: Run integration tests
  if: success() && steps.aws-auth.outcome == 'success'
  run: hatch run test:integration-cov-append-retry

Prerequisites (maintainer action required)

1. Create an IAM role with s3vectors:* permissions (scoped to haystack-test-* bucket names)
2. Configure the role's trust policy for GitHub OIDC (token.actions.githubusercontent.com)
3. Add the role ARN as a repository secret (e.g. AWS_S3_VECTORS_CI_ROLE_ARN)

[anakin87]

@dotKokott I'll try to take a look in the next few days.

Have you tried the integration yourself in a real-world setting with AWS?

[dotKokott]

@dotKokott I'll try to take a look in the next few days.

Have you tried the integration yourself in a real-world setting with AWS?

I have tried all integration tests and examples on my AWS account.

However I did not try with any large datasets. That might be next thing to validate: does this work as expected with real load.