Make it possible to run init/systemd inside the build container

[aressem]

I would like to start the build container with a working systemd inside. This means having /sbin/init as PID 1.

Currently there is no good way to do this as the container command is hardcoded:

agent-stack-k8s/internal/controller/scheduler/scheduler.go

Line 248 in 2bf2584

c.Command = []string{"/workspace/buildkite-agent"}

It would be nice to be able to override this command to be able to start /sbin/init and then the buildkite-agent with its arguments.

We could of course hack this into a container image derived from the official one, but would rather not.

Any help making this work would be highly appreciated.

[moskyb]

g'day @aressem! currently, we hardcode the the container command for the exec container, as we need to be able to explicitly control how the agent is started in that container, as there's a bit of a dance involved in booting k8s agents. we don't anticipate changing this very soon, but we might look into it in the future.

i'd like to know more about your use case of systemd inside a k8s container, however. what are the problems you're trying to solve?

[aressem]

My use case is that I would like to start a set of services with systemd (e.g. dockerd) and also test systemd units as part of our builds. To successfully start systemd the /sbin/init command must have PID 1.

Currently the buildkite-agent grabs the PID 1 as it is the first command in the container. We could do some nasty stuff like swapping the PIDs after startup if possible, but it would be much cleaner to allow the command of the build container to be a wrapper. This wrapper would then start the buildkite-agent with the provided arguments and then exec itself into PID 1.

Another approach would be to allow an init container to be inserted after the copy-agent init container. This way we could replace the /workspace/build-agent used by the build containers to be the above mentioned wrapper.

Any workarounds or ideas that would help us to have this wrapper in place would be helpful.

[aressem]

We have a workaround for this where we have used Kyverno to add a MutatingAdmissionWebhook that fires up systemd with PID 1 and executed the build-agent with its arguments. This works well.

Just saw that the new feature with the podSpecPatch using strategic merge patches and it might be that this could solve this issue as well?