The PostCommit Java Jpms Dataflow Java17 job is flaky

[github-actions]

The PostCommit Java Jpms Dataflow Java17 is failing over 50% of the time.
Please visit https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Jpms_Dataflow_Java17.yml?query=is%3Afailure+branch%3Amaster to see all failed workflow runs.
See also Grafana statistics: http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1&viewPanel=4&var-Workflow=PostCommit%20Java%20Jpms%20Dataflow%20Java17

[Amar3tto]

Job has been stable for 2 days. Closing as resolved.

[github-actions]

Reopening since the workflow is still flaky

[Amar3tto]

Possible culprit: #33086.
@scwhittle PTAL

[scwhittle]

Do you know where to find the worker logs?

In the test failure I just see java.lang.RuntimeException: SDK harness sdk-0-0 disconnected. This usually means that the process running the pipeline code has crashed. Inspect the Worker Logs and the Diagnostics tab to determine the cause of the crash.SDK harness sdk-0-0 disconnected.

[Amar3tto]

Do you know where to find the worker logs?

In the test failure I just see java.lang.RuntimeException: SDK harness sdk-0-0 disconnected. This usually means that the process running the pipeline code has crashed. Inspect the Worker Logs and the Diagnostics tab to determine the cause of the crash.SDK harness sdk-0-0 disconnected.

Here is the link to the job: https://console.cloud.google.com/dataflow/jobs/us-central1/2025-03-21_03_05_55-12385013227422836580

[scwhittle]

The workers are failing with error:

java.lang.NoSuchMethodError: 'boolean org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.multiplexElements(org.apache.beam.model.fnexecution.v1.BeamFnApi$Elements)'
	at org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:528)
	at org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:150)
	at org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:115)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run(UnboundedScheduledExecutorService.java:163)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

My PR did change the signature of that method. However it's unclear why it is attempting to call the previous version from ProcessBundleHandler. Is there something odd about the jar setup in this test causing that?

[Amar3tto]

The workers are failing with error:

java.lang.NoSuchMethodError: 'boolean org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.multiplexElements(org.apache.beam.model.fnexecution.v1.BeamFnApi$Elements)'
	at org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:528)
	at org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:150)
	at org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:115)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run(UnboundedScheduledExecutorService.java:163)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

My PR did change the signature of that method. However it's unclear why it is attempting to call the previous version from ProcessBundleHandler. Is there something odd about the jar setup in this test causing that?

I think we just need to upload and then bump dataflow java container version, as it is currently at March 12th:
ext.dataflowFnapiContainerVersion = 'beam-master-20250312'