Docker compose v2 support in local mode

[sateeshmannar]

Describe the bug
Local mode execution currently just uses "docker-compose" and dosen't use the newer "docker compose".
The _compose function @ image,py should attempt to find both docker compose executables and use the one thats available with v2 (docker compose) being the first option.

relevant code to change:

sagemaker-python-sdk/src/sagemaker/local/image.py

Lines 96 to 103 in 6a7002e

	# check if docker-compose is installed
	if find_executable("docker-compose") is None:
	raise ImportError(
	"'docker-compose' is not installed. "
	"Local Mode features will not work without docker-compose. "
	"For more information on how to install 'docker-compose', please, see "
	"https://docs.docker.com/compose/install/"
	)

sagemaker-python-sdk/src/sagemaker/local/image.py

Lines 712 to 730 in 6a7002e

	def _compose(self, detached=False):
	"""Invokes the docker compose command.
	Args:
	detached:
	"""
	compose_cmd = "docker-compose"
	command = [
	compose_cmd,
	"-f",
	os.path.join(self.container_root, DOCKER_COMPOSE_FILENAME),
	"up",
	"--build",
	"--abort-on-container-exit" if not detached else "--detach", # mutually exclusive
	]
	logger.info("docker command: %s", " ".join(command))
	return command

To reproduce
Install the latest version of SageMaker SDK and attempt to run a processing or training task in local mode.

• Run local mode WITHOUT installing package "docker-compose" (docker-compose 1.29.2)

Expected behavior
local mode execution should attempt to find both docker compose executables ("docker compose" , "docker-compose" in the listed order) and use the one thats available. "docker compose" the newer version should be the preferred option if both executables are found in the runtime environment.

System information

• SageMaker Python SDK version: 2.173.0
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): SageMaker Processing
• Python version: 3.11.4
• CPU or GPU: CPU
• Custom Docker image (Y/N): Yes

Additional context
The custom docker image was working in local mode until the dependency package (docker-compose 1.29.2) was removed local requirements as part of release v2.164.0

[lorenzwalthert]

Also see #3613.

[jmahlik]

In case anyone runs in to this, depending how docker compose is installed one has to add a shim for docker-compose after installing compose v2. Mostly when it's installed via a package manager (the recommended way to install so it's updatable). #3913 doesn't really call this out.

It would be ideal if the sdk could find the new executable to avoid the shim. Users could install compose as normal then.

Right now it looks like below. We could remove the shim if the sdk found docker compose.

 sudo yum install -y yum-utils
 sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
 sudo yum install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# We have to add this script
cat << EOF | sudo tee /usr/local/bin/docker-compose
#!/bin/bash
docker compose --compatibility "\$@"
EOF
chmod +x  /usr/local/bin/docker-compose

[Ce11an]

Any movement on this? I have tried several different methods in my CI pipeline to no avail.

[jmahlik]

Any movement on this? I have tried several different methods in my CI pipeline to no avail.

What kind of issues are you running in to? Do you have a traceback? Did you add a shim in to map compose to docker-compose or directly download the binary?