[FLINK-37881] Drop gosu in favour of Dockerfile USER

[avi-sanwal]

This PR removes the gosu utility from Flink Docker images, replacing privilege-dropping at runtime with the Dockerfile USER directive. This change improves security by following the principle of least privilege and aligns with modern container security best practices.

Breaking Changes

Images now start as the flink user (UID 9999) instead of root. This affects users who build custom Dockerfiles inheriting from official Flink images.

Migration Required for Custom Dockerfiles

Before this change:

FROM flink:1.20
RUN apt-get update && apt-get install -y custom-package

After this change:

FROM flink:1.20
USER root
RUN apt-get update && apt-get install -y custom-package
USER flink

Changes in This PR

Dockerfile Template (Dockerfile-ubuntu.template)

• Removed gosu installation and GPG verification (~20 lines)
• Added USER flink directive before entrypoint configuration

Entrypoint Script (docker-entrypoint.sh)

• Replaced drop_privs_cmd() with check_priv_user() that warns when running as root
• Removed $(drop_privs_cmd) from all exec commands
• Container now runs as flink user throughout its lifecycle

Security Benefits

1. Reduced attack surface: Eliminates runtime privilege escalation mechanism
2. Supply chain security: One fewer third-party binary to verify and maintain
3. Compliance: Aligns with CIS Docker Benchmark and Kubernetes Pod Security Standards
4. Simpler security posture: No privileged operations after image build

Impact Assessment

Affected Users

• Organizations with custom Dockerfiles that inherit from flink:* images
• Deployments that assume containers start as root
• CI/CD pipelines building derived images

Common Migration Patterns

1. System Package Installation

FROM flink:2.0
USER root
RUN apt-get update && apt-get install -y curl vim \
    && rm -rf /var/lib/apt/lists/*
USER flink

2. File Copying with Ownership

FROM flink:2.0
USER root
COPY --chown=flink:flink ./custom-libs/*.jar /opt/flink/lib/
USER flink

3. Directory Creation

FROM flink:2.0
USER root
RUN mkdir -p /custom/path && chown flink:flink /custom/path
USER flink

Kubernetes Considerations

If you have customised kubernetes entrypoint scripts, and you happen to have a volume mounts that the script was accessing, the volume mounts may require fsGroup configuration:

apiVersion: v1
kind: Pod
spec:
  securityContext:
    fsGroup: 9999
  containers:
  - name: flink
    image: flink:2.0
    volumeMounts:
    - name: data
      mountPath: /opt/flink/data

Testing

A comprehensive test suite has been added to validate the changes:

• Container starts as flink user (UID 9999)
• gosu binary is not present
• JobManager and TaskManager start successfully
• Custom Dockerfile inheritance works correctly
• Volume mount permissions function properly
• Environment variable configuration works
• Root user warning is displayed when appropriate

Test script: testing/test-gosu-removal.sh

Rollback Plan

Users requiring the old behavior can:

• Pin to Flink 1.19.x images which retain gosu
• Build custom images with gosu if needed
• Explicitly add gosu to derived Dockerfiles

Affected Versions

• Flink 2.0 and later
• Flink 1.20 and later (if backported)

Previous versions (1.19.x and earlier) are unaffected.

Documentation Updates

This breaking change will be documented in:

• Flink release notes for affected versions
• Docker Hub image descriptions
• JIRA ticket FLINK-37881
• Announcements to dev@flink.apache.org mailing list

Related Work

• JIRA: FLINK-37881
• Previous discussion: [Link to mailing list thread if applicable]

Checklist

• Changes applied to template files in dev-master branch
• Test suite added and passing
• Breaking changes documented
• Migration guide reviewed by maintainers
• Announcement drafted for mailing list
• Release notes entry prepared

[gaborgsomogyi]

I've gone through the code and one suggestion is that we should add migration patterns into the readme on master + we should give the context to users why it must be done. Please rebase to the latest code.

[gaborgsomogyi]

In the meantime I've made extensive testing on the code, namely:

• Created a docker image before and after change and compared file owner and rights. They are the same except gosu which makes sense.
• Started standalonesession from both before and after docker images. Similar output with the timestamp and temporary ID differences which also makes sense.

[gaborgsomogyi]

I think this will be good to go if nobody has comments and after rebase to latest.

[avi-sanwal]

rebased

[gaborgsomogyi]

Waiting couple of days before merge to see whether somebody has comments

[avi-sanwal]

Thank you, @gaborgsomogyi