[BUG] Observed a panic: runtime error: invalid memory address or nil pointer dereference

[rbaumgar]

What happened:

microshift panic error during startup in loop.

What you expected to happen:

microshift get ready

How to reproduce it (as minimally and precisely as possible):

1. installed Fedora IOT 36 on raspberry 3B. Fedora Linux 36.20220731.0 (IoT Edition)
2. install cri-o 1.21, microshift-4.8.0-2022_04_20_141053.fc36.aarch64
3. systemctl enable crio microshift

Anything else we need to know?:

Environment:

• Microshift version (use microshift version):
  MicroShift Version: 4.8.0-0.microshift-2022-04-20-141053
  Base OKD Version: 4.8.0-0.okd-2021-10-10-030117

• Hardware configuration:
  Architecture: aarch64
  CPU op-mode(s): 32-bit, 64-bit
  Byte Order: Little Endian
  CPU(s): 4
  On-line CPU(s) list: 0-3
  Vendor ID: ARM
  BIOS Vendor ID: Unknown
  Model name: Cortex-A53
  BIOS Model name: Unknown CPU @ 0.0GHz
  BIOS CPU family: 2
  Model: 4
  Thread(s) per core: 1
  Core(s) per cluster: 4
  Socket(s): 1
  Cluster(s): 1
  Stepping: r0p4
  CPU(s) scaling MHz: 68%
  CPU max MHz: 1400.0000
  CPU min MHz: 600.0000
  BogoMIPS: 38.40
  Flags: fp asimd evtstrm crc32 cpuid
  Caches (sum of all):
  L1d: 128 KiB (4 instances)
  L1i: 128 KiB (4 instances)
  L2: 512 KiB (1 instance)
  NUMA:
  NUMA node(s): 1
  NUMA node0 CPU(s): 0-3
  Vulnerabilities:
  Itlb multihit: Not affected
  L1tf: Not affected
  Mds: Not affected
  Meltdown: Not affected
  Mmio stale data: Not affected
  Spec store bypass: Not affected
  Spectre v1: Mitigation; __user pointer sanitization
  Spectre v2: Not affected
  Srbds: Not affected
  Tsx async abort: Not affected
  [root@fedora ~]# lsblk
  NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
  mmcblk0 179:0 0 28.9G 0 disk
  ├─mmcblk0p1 179:1 0 501M 0 part /boot/efi
  ├─mmcblk0p2 179:2 0 1G 0 part /boot
  └─mmcblk0p3 179:3 0 27.4G 0 part /var
  /usr
  /
  /sysroot
  zram0 252:0 0 925M 0 disk [SWAP]

• OS (e.g: cat /etc/os-release):
  NAME="Fedora Linux"
  VERSION="36.20220731.0 (IoT Edition)"
  ID=fedora
  VERSION_ID=36
  VERSION_CODENAME=""
  PLATFORM_ID="platform:f36"
  PRETTY_NAME="Fedora Linux 36.20220731.0 (IoT Edition)"
  ANSI_COLOR="0;38;2;60;110;180"
  LOGO=fedora-logo-icon
  CPE_NAME="cpe:/o:fedoraproject:fedora:36"
  HOME_URL="https://fedoraproject.org/"
  DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f36/system-administrators-guide/"
  SUPPORT_URL="https://ask.fedoraproject.org/"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"
  REDHAT_BUGZILLA_PRODUCT="Fedora"
  REDHAT_BUGZILLA_PRODUCT_VERSION=36
  REDHAT_SUPPORT_PRODUCT="Fedora"
  REDHAT_SUPPORT_PRODUCT_VERSION=36
  PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
  VARIANT="IoT Edition"
  VARIANT_ID=iot
  OSTREE_VERSION='36.20220731.0'

• Kernel (e.g. uname -a):
  Linux microshift 5.18.13-200.fc36.aarch64 Init #1 SMP PREEMPT_DYNAMIC Fri Jul 22 13:44:25 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Relevant Logs

[microshift.log]
(https://github.com/openshift/microshift/files/9232500/microshift.log)

[thinkahead]

The microshift binary from Apr 20, 2022 installed in /usr/bin by the rpm installer does not work with Fedora 36. It causes a crash-loop as seen with journalctl command. Replacing the microshift binary installed by the rpm with a newer binary from May 11, 2022 downloaded from https://github.com/openshift/microshift/releases/ works. Note that the microshift-linux-arm64 binaries from 05-11-2022 Latest Nightly Build or 04-20-2022 (both 4.8.0-0.microshift-2022-04-20-182108 and 4.8.0-0.microshift-2022-04-20-141053) work.
The microshift version installed by rpm shows 4.8.0-0.microshift-2022-04-20-141053 but it does not work. I built the microshift binary from sources and that works too.

I also mentioned this in my blog https://community.ibm.com/community/user/cloud/blogs/alexei-karve/2022/06/20/microshift-21 search for "Replace MicroShift binary"

[rbaumgar]

I used this a workaround: (problem Fedora IOT is only read-only FS)
curl -L https://github.com/openshift/microshift/releases/download/nightly/microshift-linux-arm64 > /usr/local/bin/microshift
chmod +x /usr/local/bin/microshift
/usr/local/bin/microshift version
sed -i "s|/usr/bin|/usr/local/bin|" /etc/systemd/system/microshift.service
systemctl daemon-reload

[rbaumgar]

Now I receive another Fatalf..
microshift.log
.

[thinkahead]

I had used cri-o:1.23

rpm-ostree ex module enable cri-o:1.23 # Experimental enable
rpm-ostree install cri-o cri-tools microshift

[rbaumgar]

Now I have a raspberry 4 with 4GB.
Installed Fedora IOT 36
Installed cri-o 1.23
Microshift nightly build
--> working!

oc get node -o wide
NAME         STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                   KERNEL-VERSION            CONTAINER-RUNTIME
microshift   Ready    <none>   4m13s   v1.21.0   192.168.1.4   <none>        Fedora Linux 36.20220618.0 (IoT Edition)   5.18.5-200.fc36.aarch64   cri-o://1.23.0

[mangelajo]

Right, use the nightly build for now.

Fedora36 provided the wrong golang compiler version that seems to make some of the components crash. This was fixed on the nightly build by enforcing an specific golang version.

We are working on the product MicroShift, afterwards you should see activity again on the community version.

[dhellmann]

We're focusing on RHEL support for the moment and the main branch has moved on quite a bit from the 4.8 release branch. You could try compiling the main branch, but it may be a few months before the team returns to focusing on Fedora support.