GA Migration Java PostCommit Dataflow V2 & Dataflow V2 Streaming

.github/workflows/job-postcommit-java-vr-dataflow-v2-streaming.yml

@@ -0,0 +1,53 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java VR Dataflow V2 Streaming PostCommit
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '0 */6 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: ['v*']
+permissions: read-all
+
+jobs:
+  java_vr_dataflow_v2:
+    name: Run Java Dataflow V2 ValidatesRunner Streaming
+    runs-on: [self-hosted, ubuntu-20.04]
+    timeout-minutes: 510
+    steps:
+      - name: Get number of available processors
+        id: get_available_processors
+        run: |
+          CORES=$(grep -c ^processor /proc/cpuinfo)
+          BALANCE_CORES=$((3*CORES))
+          echo $BALANCE_CORES
+          echo "::set-output name=BALANCE_CORES::$BALANCE_CORES"
+      - name: Checkout code
+        uses: actions/checkout@v3
+        with:
+          persist-credentials: false
+      - name: Setup Environment
+        uses: ./.github/actions/setup-self-hosted-action
+      - name: Run :runners:google-cloud-dataflow-java:validatesRunnerV2Streaming
+        uses: ./.github/actions/gradle-command-self-hosted-action
+        with:
+          gradle-command: :runners:google-cloud-dataflow-java:validatesRunnerV2Streaming
+          max-workers: ${{ steps.get_available_processors.outputs.BALANCE_CORES }}

.github/workflows/job-postcommit-java-vr-dataflow-v2.yml

@@ -0,0 +1,53 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java VR Dataflow V2 PostCommit
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '0 */6 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: ['v*']
+permissions: read-all
+
+jobs:
+  java_vr_dataflow_v2:
+    name: Run Java Dataflow V2 ValidatesRunner
+    runs-on: [self-hosted, ubuntu-20.04]
+    timeout-minutes: 390
+    steps:
+      - name: Get number of available processors
+        id: get_available_processors
+        run: |
+          CORES=$(grep -c ^processor /proc/cpuinfo)
+          BALANCE_CORES=$((3*CORES))
+          echo $BALANCE_CORES
+          echo "::set-output name=BALANCE_CORES::$BALANCE_CORES"
+      - name: Checkout code
+        uses: actions/checkout@v3
+        with:
+          persist-credentials: false
+      - name: Setup Environment
+        uses: ./.github/actions/setup-self-hosted-action
+      - name: Run :runners:google-cloud-dataflow-java:validatesRunnerV2
+        uses: ./.github/actions/gradle-command-self-hosted-action
+        with:
+          gradle-command: :runners:google-cloud-dataflow-java:validatesRunnerV2
+          max-workers: ${{ steps.get_available_processors.outputs.BALANCE_CORES }}

.test-infra/jenkins/README.md

@@ -107,8 +107,6 @@ Beam Jenkins overview page: [link](https://ci-beam.apache.org/)
 | beam_PostCommit_Java_ValidatesRunner_Spark | [cron](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/), [phrase](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_PR/) | `Run Spark ValidatesRunner` | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark) |
 | beam_PostCommit_Java_ValidatesRunner_Spark_Java11 | [cron](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_ValidatesRunner_Spark_Java11/), [phrase](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_ValidatesRunner_Spark_Java11_PR) | `Run Spark ValidatesRunner Java 11` | [![Build Status](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_ValidatesRunner_Spark_Java11/badge/icon)](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_ValidatesRunner_Spark_Java11/) |
 | beam_PostCommit_Java_ValidatesRunner_ULR | [cron](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_ValidatesRunner_ULR/), [phrase](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_ValidatesRunner_ULR_PR) | `Run ULR Loopback ValidatesRunner` | [![Build Status](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_ValidatesRunner_ULR/badge/icon)](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_ValidatesRunner_ULR/) |
-| beam_PostCommit_Java_VR_Dataflow_V2 | [cron](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_VR_Dataflow_V2/), [phrase](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_VR_Dataflow_V2_PR/) | `Run Java Dataflow V2 ValidatesRunner` | [![Build Status](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_VR_Dataflow_V2/badge/icon)](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_VR_Dataflow_V2/) |
-| beam_PostCommit_Java_VR_Dataflow_V2_Streaming | [cron](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/), [phrase](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming_PR/) | `Run Java Dataflow V2 ValidatesRunner Streaming` | [![Build Status](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/badge/icon)](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/) |
 | beam_PostCommit_Javadoc | [cron](https://ci-beam.apache.org/job/beam_PostCommit_Javadoc/), [phrase](https://ci-beam.apache.org/job/beam_PostCommit_Javadoc_PR/) | `Run Javadoc PostCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Javadoc/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Javadoc) |
 | beam_PostCommit_PortableJar_Flink | [cron](https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/), [phrase](https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink_PR/) | `Run PortableJar_Flink PostCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink) |
 | beam_PostCommit_PortableJar_Spark | [cron](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_PortableJar_Spark/), [phrase](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_PortableJar_Spark_PR/) | `Run PortableJar_Spark PostCommit` | [![Build Status](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_PortableJar_Spark/badge/icon)](https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_PortableJar_Spark/) |

CI.md

@@ -140,7 +140,8 @@ Service Account shall have following permissions ([IAM roles](https://cloud.goog
 ### PostCommit Workflows
 | Workflow                                                                           | Description             | Requires GCP Credentials |
 |------------------------------------------------------------------------------------|-------------------------|--------------------------|
-| [job-postcommit-placeholder.yml](.github/workflows/job-postcommit-placeholder.yml) | Description placeholder | Yes/No                   |
+| [job-postcommit-java-vr-dataflow-v2.yml](.github/workflows/job-postcommit-java-vr-dataflow-v2.yml) | Runs Java ValidatesRunner suite on the Dataflow runner V2 | Yes/No                   |
+| [job-postcommit-java-vr-dataflow-v2-streaming.yml](.github/workflows/job-postcommit-java-vr-dataflow-v2-streaming.yml) | Runs Java ValidatesRunner suite on the Dataflow runner V2 forcing streaming mode | Yes/No                   |
 
 ### GitHub Action Tips
 

runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java

@@ -182,7 +182,7 @@ private List<String> gcsCredentialArgs() {
     // TODO(https://github.com/apache/beam/issues/19061): Allow this to be disabled manually.
     if (Files.exists(Paths.get(localGcloudConfig))) {
       return ImmutableList.of(
-          "--mount",
+          "-v",
           String.format("type=bind,src=%s,dst=%s", localGcloudConfig, dockerGcloudConfig));
     } else {
       return ImmutableList.of();