Support managed jdbc io (Postgres)

[shunping]

Managed JDBCIO (Part 1) - Postgres

The first PR to make JDBCIO into managed IO.

Notice that we are still using the GCP IO expansion service jar. I will move it to the designated expansion service jar after finishing all the four supported databases.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[ahmedabu98]

Will take a look later, but from a quick skim, it looks like we're missing translation logic in this PR

[github-actions]

Assigning reviewers:

R: @liferoad for label python.
R: @Abacn for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[shunping]

Will take a look later, but from a quick skim, it looks like we're missing translation logic in this PR

Thanks for taking a look. I've been able to call both the managed and unmanaged versions of these transforms from Java and Python. Could you provide some pointers on where this translation logic is used?

[shunping]

Will take a look later, but from a quick skim, it looks like we're missing translation logic in this PR

Thanks for taking a look. I've been able to call both the managed and unmanaged versions of these transforms from Java and Python. Could you provide some pointers on where this translation logic is used?

Added translation logic. However for my educational purpose, where is this translation called?

[ahmedabu98]

where is this translation called

when converting the transform to its proto representation:

beam/sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/PTransformTranslation.java

Line 499 in d28fd64

FunctionSpec spec = payloadTranslator.translate(appliedPTransform, components);

[shunping]

where is this translation called

when converting the transform to its proto representation:

beam/sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/PTransformTranslation.java

Line 499 in d28fd64

FunctionSpec spec = payloadTranslator.translate(appliedPTransform, components);

Then why could my previous integration tests and my manual call in python work without it?

[ahmedabu98]

Then why could my previous integration tests and my manual call in python work without it?

Because it can still build a viable transform proto just fine, and run in a normal pipeline.

The extra translation logic is just needed to add more information that the Dataflow managed service can make use of to do upgrades.

[ahmedabu98]

Left a few comments, but looks good overall! Thanks @shunping

[shunping]

Then why could my previous integration tests and my manual call in python work without it?

Because it can still build a viable transform proto just fine, and run in a normal pipeline.

The extra translation logic is just needed to add more information that the Dataflow managed service can make use of to do upgrades.

Thanks for clarifying this. cc'ed @liferoad

[liferoad]

Then why could my previous integration tests and my manual call in python work without it?

Because it can still build a viable transform proto just fine, and run in a normal pipeline.
The extra translation logic is just needed to add more information that the Dataflow managed service can make use of to do upgrades.

Thanks for clarifying this. cc'ed @liferoad

We should have the readme file to document all these requirements.