[Enhancement]: Add systemd units for production deployments on Podman

[fzdarsky]

What would you like to be added:

systemd units for running MicroShift on Podman on RHEL in production. This includes creation of the Podman volumes for keeping MicroShift state, least privileges (seccomp et al.), ensuring MicroShift restarts upon failure, etc. We should also suggest an auto-update strategy. That production deployment should use an external CRI-O instance, so MicroShift container restarts/redeployments don't disrupt workloads (however, we may want to also provide an example for the "all-in-one" MicroShift container with CRI-O).

Potentially useful: systemd for Podman, auto-updates, and seccomp for containers.

Why is this needed:

We need a prescriptive way how to do this, so we a) maximise chances users successfully deploy without requiring lots of upfront reading and b) we can provide better test coverage for this scenario.

[sallyom]

scenarios:

• all-in-one podman image and a systemd unit file with the recommended run command
• podman image with crio and kubelet mounted from the host unit file with recommended run command
• identify recommended seccomp profile
• document update process, auto-update process - (feature within podman)

[sallyom]

@husky-parul we get to use this! 😁 https://podman.io/blogs/2019/10/15/generate-seccomp-profiles.html#the-oci-seccomp-bpf-hook

[fzdarsky]

@sallyom in the second scenario, the kubelet would still have to run in the container (i.e. just CRI-O on the host). As least until we support multi-node.

[rootfs]

Background

Using podman to start containerized microshift can ensure timely image update to pick up the latest Microshift release.
Additionally, using crio service on host allows workload continunity during Microshift upgrade.

PoC

Start containerized microshift

podman run --privileged --ipc=host --network=host  \
         -d --rm \
        -v /var/run:/var/run \
        -v /sys:/sys:ro \
         -v /var/lib:/var/lib:rw,rshared \
         -v /lib/modules:/lib/modules \
         -v /etc:/etc \
         -v /run/containers:/run/containers \
         -v /var/log:/var/log \
         quay.io/microshift/microshift:4.7.0-0.microshift-2021-08-31-224727-linux-amd64 

Demo

Check pods via kubectl:

[root@rhel microshift]# export KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig
[root@rhel microshift]# kubectl get pods -A
NAMESPACE                       NAME                                  READY   STATUS    RESTARTS   AGE
kube-system                     kube-flannel-ds-xlj66                 1/1     Running   0          38s
kubevirt-hostpath-provisioner   kubevirt-hostpath-provisioner-bfjvw   1/1     Running   0          39s
openshift-dns                   dns-default-s624v                     3/3     Running   0          38s
openshift-ingress               router-default-6d8c9d8f57-8n5qs       1/1     Running   0          38s
openshift-service-ca            service-ca-64547678c6-fd4bq           1/1     Running   0          39s

Check containerized microshift via podman:

[root@rhel microshift]# podman ps
CONTAINER ID  IMAGE                                                                           COMMAND     CREATED             STATUS                 PORTS       NAMES
428a0bfb8b5f  quay.io/microshift/microshift:4.7.0-0.microshift-2021-08-31-224727-linux-amd64  run         About a minute ago  Up About a minute ago              fervent_kare

Check crio pods:

[root@rhel microshift]# crictl ps
CONTAINER           IMAGE                                                                                                           CREATED             STATE               NAME                            ATTEMPT             POD ID
e1343381ab088       ebb673575b8b1081b417eb000208b5818cd7031ccbf7fdc583d003611a8f98f4                                                4 minutes ago       Running             dns-node-resolver               0                   58d75053d6191
21e6deca5b878       18248adc08dc2d2dbf1a7c192dc0a707bbc81766aecf8b9b615b8769d7cc3f8a                                                4 minutes ago       Running             kube-rbac-proxy                 0                   58d75053d6191
c4c9ac7080ac3       d3704d0b0df2e397b83dccbf31f0405b74a578e19eb16618adb04a7910d608ed                                                4 minutes ago       Running             dns                             0                   58d75053d6191
d1a6fab76e06e       52866d4b65279ebbba6c33b48d7aa08be85624a71d845b56ade6646749c8236a                                                4 minutes ago       Running             router                          0                   202b48eab1812
be1f092238e58       cfbfdcba5a82ceff418e260fbde0131a711e03a762a1e6f4062302ca6987fb4e                                                4 minutes ago
       Running             service-ca-controller           0                   825821bf835dc
99df18e71c3a0       quay.io/kubevirt/hostpath-provisioner@sha256:8cf5c2186a0357a70b835f2ef76430306e91da5743f375f87693f61a470ac75f   4 minutes ago       Running             kubevirt-hostpath-provisioner   0                   1cbc645342688
50be83ccf9e10       8522d622299ca431311ac69992419c956fbaca6fa8289c76810c9399d17c69de                                                4 minutes ago       Running             kube-flannel                    0                   6a8b86de590d2

TODO

• Finalize microshift startup command
• Production ready systemd unit for containerized MicroShift

[husky-parul]

@fzdarsky @sallyom @rootfs
The secomp filter are as following

{"defaultAction":"SCMP_ACT_ERRNO","architectures":["SCMP_ARCH_X86_64"],
"syscalls":[{"names":["accept4","access","arch_prctl","bind","brk","capget","capset","chdir","clone","close","connect","dup2","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","execve","exit_group","fallocate","fchdir","fcntl","fdatasync","flock","fstat","ftruncate","futex","getdents64","getegid","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getsockname","getsockopt","gettid","getuid","inotify_add_watch","inotify_init1","ioctl","listen","lseek","madvise","mkdirat","mmap","mount","mprotect","munmap","nanosleep","newfstatat","openat","pipe2","pivot_root","prctl","pread64","prlimit64","pwrite64","read","readlink","readlinkat","recvfrom","recvmsg","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","select","sendmsg","sendto","set_robust_list","set_tid_address","sethostname","setresgid","setresuid","setrlimit","setsid","setsockopt","sigaltstack","socket","stat","statfs","statx","sysinfo","tgkill","time","umask","umount2","uname","unlinkat","wait4","waitid","write"],"action":"SCMP_ACT_ALLOW","args":[],"comment":"","includes":{},"excludes":{}}]}