[KafkaIO] Share size and offset estimators among ReadFromKafkaDoFn instances per PTransform

[sjvanrossum]

This change adds shared caches among unique instances of ReadFromKafkaDoFn, identified by a static AtomicLong which is stored as an instance field and incremented during construction of the DoFn. The constructor is bypassed during deserialization at execution time, thus providing a shared identifier for all instances of ReadFromKafkaDoFn for a single transform.

Note: this PR is stacked on top of #32921, #32920 and #32889

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[sjvanrossum]

assign set of reviewers

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @Abacn for label java.
R: @Abacn for label io.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[johnjcasey]

LGTM, please fix the conflicts and this should be good to go

[sjvanrossum]

Done, should be good to go once #32921 is merged.

[sjvanrossum]

@johnjcasey rebased and updated where required. PreCommit Spotless is failing, so I ran its action (./gradlew spotlessCheck checkStyleMain checkStyleTest :buildSrc:spotlessCheck) locally and that passed.

@Abacn any thoughts?

[sjvanrossum]

Run Spotless PreCommit

[sjvanrossum]

PreCommit Spotless passed: https://github.com/apache/beam/actions/runs/11817154535

Not sure why the check overview doesn't update.

[Abacn]

Details

The UI is misleading. It's a GitHub bug and isn't an actual failure. It's due to some actions in workflow publish check back, but it does not name the report, and it's goes to a random workflow.

[Abacn]

Done, should be good to go once #32921 is merged.

#32921 is merged, need to rebase this PR onto the current master branch

[sjvanrossum]

Run Spotless PreCommit