[BUG] kubectl is unable to contact the apiserver

[osilkin98]

What happened:

Running into a weird issue. I'm running an interactive terminal from the microshift:aio-cuda-11.4.2-base-ubi8 and microshift is definitely running; however, kubectl is unable to connect to the apiserver and returns a connection refused message.

The output from ps aux | grep microshift:

[root@ushift /]# ps aux | grep microshift
root      612447 14.6  3.2 13429504 1017592 ?    Ssl  Oct03 131:31 /usr/local/bin/microshift run

And the Microshift ports are definitely bound. See the output from netstat -tulpn | grep microshift:

[root@ushift /]#netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      612447/microshift   
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      612447/microshift   
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      612447/microshift   
tcp        0      0 10.88.0.5:2379          0.0.0.0:*               LISTEN      612447/microshift   
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      612447/microshift   
tcp        0      0 10.88.0.5:2380          0.0.0.0:*               LISTEN      612447/microshift   
tcp        0      0 127.0.0.1:2381          0.0.0.0:*               LISTEN      612447/microshift   
tcp6       0      0 :::10256                :::*                    LISTEN      612447/microshift   
tcp6       0      0 :::10259                :::*                    LISTEN      612447/microshift   
tcp6       0      0 :::8444                 :::*                    LISTEN      612447/microshift   
tcp6       0      0 :::8445                 :::*                    LISTEN      612447/microshift   
tcp6       0      0 :::10250                :::*                    LISTEN      612447/microshift   
tcp6       0      0 :::10251                :::*                    LISTEN      612447/microshift   
tcp6       0      0 :::6443                 :::*                    LISTEN      612447/microshift   
tcp6       0      0 :::10252                :::*                    LISTEN      612447/microshift   

The microshift ports are responsive, curl https://localhost:6443 produces a JSON result complaining about an unauthorized request:

[root@ushift /]# curl --insecure https://localhost:6443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

What you expected to happen:

kubectl get all should have returned a list of all resources in the default namespace.

How to reproduce it (as minimally and precisely as possible):

To reproduce it, you need to setup Microshift within a GPU environment following the guide here

Anything else we need to know?:

Environment:

• Microshift version (use microshift version): 4.7.0-0.microshift-2021-08-31-224727
• Hardware configuration: g3s.xlarge ec2 cluster
• OS (e.g: cat /etc/os-release):

  • NAME="Red Hat Enterprise Linux"
    VERSION="8.4 (Ootpa)"
    ID="rhel"
    ID_LIKE="fedora"
    VERSION_ID="8.4"
    PLATFORM_ID="platform:el8"
    PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
    ANSI_COLOR="0;31"
    CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
    HOME_URL="https://www.redhat.com/"
      DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
    BUG_REPORT_URL="https://bugzilla.redhat.com/"
    
    REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
    REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
    REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
    REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
    

• Kernel (e.g. uname -a): Linux ushift 4.18.0-305.19.1.el8_4.x86_64 #1 SMP Tue Sep 7 07:07:31 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
• Others: The above output was recorded within a privileged podman container

Relevant Logs

[osilkin98]

Solved this by setting the KUBECONFIG environment variable, as outlined here