diff --git a/apiserver-config-flow.md b/apiserver-config-flow.md
new file mode 100644
index 0000000000..665a5cd8a2
--- /dev/null
+++ b/apiserver-config-flow.md
@@ -0,0 +1,397 @@
+# API Server Config Data Flow
+
+This document traces the data path from the `apiServer` field in `/etc/microshift/config.yaml` down to where the values are ingested by the kube-apiserver.
+
+## UML Sequence Diagram
+
+```mermaid
+sequenceDiagram
+ participant ConfigFile as config.yaml
+ participant ActiveConfig as config.ActiveConfig()
+ participant FillDefaults as config.fillDefaults()
+ participant IncorporateSettings as config.incorporateUserSettings()
+ participant UpdateComputed as config.updateComputedValues()
+ participant KASRun as KubeAPIServer.Run()
+ participant KASConfigure as KubeAPIServer.configure()
+ participant Merge as resourcemerge.MergePrunedProcessConfig()
+ participant TempFile as Create Temp File
+ participant NewCmd as kubeapiserver.NewAPIServerCommand()
+ participant Execute as cmd.ExecuteContext()
+ participant GetOpenshiftConfig as enablement.GetOpenshiftConfig()
+ participant ConfigToFlags as openshiftkubeapiserver.ConfigToFlags()
+ participant ParseFlags as cmd.ParseFlags()
+ participant KASMain as Run() [kube-apiserver]
+
+ Note over ConfigFile: User configuration in
/etc/microshift/config.yaml
apiServer:
advertiseAddress: "10.43.0.2"
port: 6443
auditLog:
profile: "Default"
tls:
minVersion: "VersionTLS12"
cipherSuites: [...]
featureGates:
featureSet: "CustomNoUpgrade"
customNoUpgrade:
enabled: ["UserNamespacesSupport"]
+
+ ConfigFile->>+ActiveConfig: Read config.yaml
[files.go:120]
+
+ ActiveConfig->>+FillDefaults: Set default values
[config.go:92]
+ Note right of FillDefaults: Sets default ApiServer values:
- Port: 6443
- URL: "https://localhost:6443"
- AuditLog defaults
- SubjectAltNames
+ FillDefaults-->>-ActiveConfig: Config with defaults
+
+ ActiveConfig->>+IncorporateSettings: Merge user settings
[config.go:195]
+ Note right of IncorporateSettings: Merges user-provided values:
- ApiServer.AdvertiseAddress
- ApiServer.AuditLog.*
- ApiServer.TLS.*
- ApiServer.NamedCertificates
- ApiServer.FeatureGates.*
+ IncorporateSettings-->>-ActiveConfig: Config with user settings
+
+ ActiveConfig->>+UpdateComputed: Compute derived values
[config.go:434]
+ Note right of UpdateComputed: Computes:
- AdvertiseAddresses (from ServiceNetwork)
- TLS.UpdateValues() - cipher suites
- SkipInterface flag
+ UpdateComputed-->>-ActiveConfig: Final Config
+
+ ActiveConfig-->>-ConfigFile: *config.Config
+
+ Note over KASRun: MicroShift creates and starts
kube-apiserver controller
+
+ KASRun->>+KASConfigure: configure(ctx, cfg)
[kube-apiserver.go:97]
+
+ Note right of KASConfigure: Uses cfg.ApiServer fields to build
KubeAPIServerConfig overrides:
- advertise-address
- audit-policy-file
- audit-log-*
- tls-cert-file
- tls-private-key-file
- tls-min-version
- tls-cipher-suites
- feature-gates
- service-cluster-ip-range
- disable-admission-plugins
- enable-admission-plugins
+
+ KASConfigure->>+Merge: Merge config layers
[kube-apiserver.go:267]
+ Note right of Merge: Merges 3 layers:
1. defaultconfig.yaml (embedded)
2. config-overrides.yaml (embedded)
3. Runtime overrides from cfg.ApiServer
Creates KubeAPIServerConfig struct
with all settings merged
+ Merge-->>-KASConfigure: kasConfigBytes (marshaled YAML)
+
+ KASConfigure-->>-KASRun: Configuration complete
+
+ KASRun->>TempFile: Create temp config file
[kube-apiserver.go:363]
+ Note right of TempFile: Write kasConfigBytes to:
/tmp/kube-apiserver-config-*.yaml
+
+ KASRun->>+NewCmd: NewAPIServerCommand()
[kube-apiserver.go:391]
+ Note right of NewCmd: Initialize kube-apiserver command
from kubernetes/cmd/kube-apiserver
+ NewCmd-->>-KASRun: cmd *cobra.Command
+
+ KASRun->>+Execute: cmd.ExecuteContext(ctx)
[kube-apiserver.go:404]
+ Note right of Execute: Args:
--openshift-config /tmp/kube-apiserver-config-*.yaml
-v [verbosity]
+
+ Execute->>+GetOpenshiftConfig: Read config file
[intialization.go:27]
+ Note right of GetOpenshiftConfig: Reads temp config file and
deserializes into
KubeAPIServerConfig struct
Resolves file paths
Sets recommended defaults
+ GetOpenshiftConfig-->>-Execute: *KubeAPIServerConfig
+
+ Execute->>+ConfigToFlags: Convert to CLI flags
[flags.go:18]
+ Note right of ConfigToFlags: Converts KubeAPIServerConfig.APIServerArguments
to command-line flags:
Map[string][]string -> []string
Examples:
"advertise-address": ["10.43.0.2"]
-> --advertise-address=10.43.0.2
"feature-gates": ["UserNamespacesSupport=true"]
-> --feature-gates=UserNamespacesSupport=true
Also converts:
- ServingInfo -> tls-* flags
- AuditConfig -> audit-* flags
- AdmissionConfig -> admission-* flags
+ ConfigToFlags-->>-Execute: []string (CLI flags)
+
+ Execute->>+ParseFlags: Parse merged flags
[server.go:122]
+ Note right of ParseFlags: Re-parses command flags with
values from OpenShift config
This updates the ServerRunOptions
with all the ApiServer settings
+ ParseFlags-->>-Execute: Updated options
+
+ Execute->>+KASMain: Run(ctx, completedOptions)
[server.go:153]
+ Note right of KASMain: Kube-apiserver starts with
all configuration applied:
- TLS settings
- Audit logging
- Feature gates
- Admission plugins
- Serving configuration
+
+ Note over KASMain: Kube-apiserver running with
all config.ApiServer values
applied via command-line flags
+```
+
+## Key Data Structures
+
+### 1. MicroShift Config (`pkg/config/apiserver.go`)
+
+```go
+type ApiServer struct {
+ SubjectAltNames []string
+ AdvertiseAddress string
+ NamedCertificates []NamedCertificateEntry
+ AuditLog AuditLog
+ TLS TLSConfig
+ FeatureGates FeatureGates
+ URL string
+ Port int
+ AdvertiseAddresses []string
+}
+```
+
+### 2. KubeAPIServerConfig (`vendor/github.com/openshift/api/kubecontrolplane/v1/types.go`)
+
+```go
+type KubeAPIServerConfig struct {
+ GenericAPIServerConfig configv1.GenericAPIServerConfig
+ APIServerArguments map[string]Arguments
+ ServiceAccountPublicKeyFiles []string
+ ServicesNodePortRange string
+ // ... other fields
+}
+```
+
+### 3. APIServerArguments (Key-Value Map)
+
+```go
+APIServerArguments: map[string]Arguments{
+ "advertise-address": {cfg.ApiServer.AdvertiseAddress},
+ "audit-log-maxage": {strconv.Itoa(cfg.ApiServer.AuditLog.MaxFileAge)},
+ "tls-min-version": {cfg.ApiServer.TLS.MinVersion},
+ "tls-cipher-suites": cfg.ApiServer.TLS.CipherSuites,
+ "feature-gates": {"UserNamespacesSupport=true", ...},
+ // ... many more
+}
+```
+
+## Configuration Flow Steps
+
+### Step 1: Config Loading (`pkg/config/files.go`)
+
+**Function**: `ActiveConfig()`
+**Line**: [120-127](file:///home/microshift/microshift/pkg/config/files.go#L120-L127)
+
+1. Reads `/etc/microshift/config.yaml`
+2. Reads YAML files from `/etc/microshift/config.d/*.yaml`
+3. Merges all YAML files using JSON patch merge
+4. Deserializes into `Config` struct
+
+### Step 2: Config Initialization (`pkg/config/config.go`)
+
+**Function**: `fillDefaults()`
+**Lines**: [92-190](file:///home/microshift/microshift/pkg/config/config.go#L92-L190)
+
+Sets default values for `ApiServer`:
+```go
+c.ApiServer = ApiServer{
+ SubjectAltNames: subjectAltNames,
+ URL: "https://localhost:6443",
+ Port: 6443,
+}
+c.ApiServer.AuditLog = AuditLog{
+ MaxFileAge: 0,
+ MaxFiles: 10,
+ MaxFileSize: 200,
+ Profile: "Default",
+}
+```
+
+**Function**: `incorporateUserSettings()`
+**Lines**: [195-429](file:///home/microshift/microshift/pkg/config/config.go#L195-L429)
+
+Merges user-provided values:
+```go
+if u.ApiServer.AdvertiseAddress != "" {
+ c.ApiServer.AdvertiseAddress = u.ApiServer.AdvertiseAddress
+}
+if u.ApiServer.AuditLog.Profile != "" {
+ c.ApiServer.AuditLog.Profile = u.ApiServer.AuditLog.Profile
+}
+// ... TLS settings
+// ... FeatureGates
+```
+
+**Function**: `updateComputedValues()`
+**Lines**: [434-516](file:///home/microshift/microshift/pkg/config/config.go#L434-L516)
+
+Computes derived values:
+- `AdvertiseAddress` (if not set)
+- `AdvertiseAddresses` (for dual-stack)
+- `TLS.UpdateValues()` - cipher suites normalization
+
+### Step 3: KubeAPIServer Configuration (`pkg/controllers/kube-apiserver.go`)
+
+**Function**: `configure()`
+**Lines**: [97-297](file:///home/microshift/microshift/pkg/controllers/kube-apiserver.go#L97-L297)
+
+Creates `KubeAPIServerConfig` overrides:
+```go
+overrides := &kubecontrolplanev1.KubeAPIServerConfig{
+ APIServerArguments: map[string]Arguments{
+ "advertise-address": {s.advertiseAddress},
+ "audit-policy-file": {...},
+ "audit-log-maxage": {strconv.Itoa(cfg.ApiServer.AuditLog.MaxFileAge)},
+ "audit-log-maxbackup": {strconv.Itoa(cfg.ApiServer.AuditLog.MaxFiles)},
+ "audit-log-maxsize": {strconv.Itoa(cfg.ApiServer.AuditLog.MaxFileSize)},
+ "service-cluster-ip-range": {strings.Join(cfg.Network.ServiceNetwork, ",")},
+ "disable-admission-plugins": {...},
+ "enable-admission-plugins": {},
+ "feature-gates": {"UserNamespacesSupport=true", ...},
+ },
+ GenericAPIServerConfig: configv1.GenericAPIServerConfig{
+ ServingInfo: configv1.HTTPServingInfo{
+ ServingInfo: configv1.ServingInfo{
+ MinTLSVersion: cfg.ApiServer.TLS.MinVersion,
+ CipherSuites: cfg.ApiServer.TLS.CipherSuites,
+ NamedCertificates: namedCerts,
+ },
+ },
+ },
+}
+```
+
+**Function**: `resourcemerge.MergePrunedProcessConfig()`
+**Lines**: [267-291](file:///home/microshift/microshift/pkg/controllers/kube-apiserver.go#L267-L291)
+
+Merges three configuration layers:
+1. `defaultconfig.yaml` (embedded baseline)
+2. `config-overrides.yaml` (embedded OpenShift defaults)
+3. Runtime `overrides` (from cfg.ApiServer)
+
+Result: `kasConfigBytes` - marshaled YAML config
+
+### Step 4: Starting Kube-APIServer (`pkg/controllers/kube-apiserver.go`)
+
+**Function**: `Run()`
+**Lines**: [315-413](file:///home/microshift/microshift/pkg/controllers/kube-apiserver.go#L315-L413)
+
+1. **Line 363-381**: Creates temporary config file
+ ```go
+ fd, err := os.CreateTemp("", "kube-apiserver-config-*.yaml")
+ io.Copy(fd, bytes.NewBuffer(s.kasConfigBytes))
+ ```
+
+2. **Line 391-395**: Creates kube-apiserver command
+ ```go
+ cmd := kubeapiserver.NewAPIServerCommand()
+ cmd.SetArgs([]string{
+ "--openshift-config", fd.Name(),
+ "-v", strconv.Itoa(s.verbosity),
+ })
+ ```
+
+3. **Line 404**: Executes command
+ ```go
+ errorChannel <- cmd.ExecuteContext(ctx)
+ ```
+
+### Step 5: Kube-APIServer Ingestion (`deps/.../cmd/kube-apiserver/app/server.go`)
+
+**Function**: `RunE` (cobra command handler)
+**Lines**: [96-154](file:///home/microshift/microshift/deps/github.com/openshift/kubernetes/cmd/kube-apiserver/app/server.go#L96-L154)
+
+1. **Line 110**: Read OpenShift config file
+ ```go
+ openshiftConfig, err := enablement.GetOpenshiftConfig(s.OpenShiftConfig)
+ ```
+
+2. **Line 116-119**: Convert to command-line flags
+ ```go
+ args, err := openshiftkubeapiserver.ConfigToFlags(openshiftConfig)
+ // Example: ["--advertise-address=10.43.0.2", "--audit-log-maxage=0", ...]
+ ```
+
+3. **Line 122**: Parse flags to update options
+ ```go
+ cmd.ParseFlags(args)
+ ```
+
+4. **Line 153**: Start API server with merged options
+ ```go
+ return Run(ctx, completedOptions)
+ ```
+
+### Step 6: Config to Flags Conversion (`deps/.../openshiftkubeapiserver/flags.go`)
+
+**Function**: `ConfigToFlags()`
+**Lines**: [18-46](file:///home/microshift/microshift/deps/github.com/openshift/kubernetes/openshift-kube-apiserver/openshiftkubeapiserver/flags.go#L18-L46)
+
+Converts `KubeAPIServerConfig` to CLI flags:
+
+```go
+func ConfigToFlags(kubeAPIServerConfig *KubeAPIServerConfig) ([]string, error) {
+ args := unmaskArgs(kubeAPIServerConfig.APIServerArguments)
+
+ // Extract from APIServerArguments map
+ // "advertise-address": ["10.43.0.2"] -> --advertise-address=10.43.0.2
+
+ // Add additional flags from other config fields
+ configflags.SetIfUnset(args, "bind-address", host)
+ configflags.SetIfUnset(args, "tls-cipher-suites",
+ kubeAPIServerConfig.ServingInfo.CipherSuites...)
+ configflags.SetIfUnset(args, "tls-min-version",
+ kubeAPIServerConfig.ServingInfo.MinTLSVersion)
+
+ return configflags.ToFlagSlice(args), nil
+}
+```
+
+**Function**: `ToFlagSlice()`
+**Lines**: [29-43](file:///home/microshift/microshift/deps/github.com/openshift/kubernetes/vendor/github.com/openshift/apiserver-library-go/pkg/configflags/helpers.go#L29-L43)
+
+Converts map to flag array:
+```go
+func ToFlagSlice(args map[string][]string) []string {
+ var flags []string
+ for key, values := range args {
+ for _, value := range values {
+ flags = append(flags, fmt.Sprintf("--%s=%v", key, value))
+ }
+ }
+ return flags
+}
+```
+
+### Step 7: OpenShift Config Reading (`deps/.../enablement/intialization.go`)
+
+**Function**: `GetOpenshiftConfig()`
+**Lines**: [27-58](file:///home/microshift/microshift/deps/github.com/openshift/kubernetes/openshift-kube-apiserver/enablement/intialization.go#L27-L58)
+
+1. Reads temp config file
+2. Deserializes YAML into `KubeAPIServerConfig`
+3. Resolves file paths (relative to config file location)
+4. Applies recommended defaults
+
+## Example: Feature Gates Flow
+
+Let's trace the `featureGates` field specifically:
+
+### 1. User Configuration (`/etc/microshift/config.yaml`)
+```yaml
+apiServer:
+ featureGates:
+ featureSet: "CustomNoUpgrade"
+ customNoUpgrade:
+ enabled: ["UserNamespacesSupport"]
+ disabled: ["SomeOtherGate"]
+```
+
+### 2. Parsed into Config Struct (`pkg/config/apiserver.go`)
+```go
+ApiServer.FeatureGates = FeatureGates{
+ FeatureSet: "CustomNoUpgrade",
+ CustomNoUpgrade: CustomNoUpgrade{
+ Enabled: []string{"UserNamespacesSupport"},
+ Disabled: []string{"SomeOtherGate"},
+ },
+}
+```
+
+### 3. Current State: Hardcoded Feature Gates (`pkg/controllers/kube-apiserver.go` line 224)
+
+**NOTE**: As of the current implementation, feature gates are hardcoded and the `cfg.ApiServer.FeatureGates` config is not yet used. The implementation would need to add logic to read from `cfg.ApiServer.FeatureGates` and construct the feature-gates argument dynamically.
+
+Current hardcoded implementation:
+```go
+APIServerArguments: map[string]Arguments{
+ "feature-gates": {
+ "UserNamespacesSupport=true",
+ "UserNamespacesPodSecurityStandards=true",
+ },
+}
+```
+
+**To implement feature gates from config**, the code would need to:
+1. Read `cfg.ApiServer.FeatureGates.FeatureSet` to determine the base feature set
+2. Parse `cfg.ApiServer.FeatureGates.CustomNoUpgrade.Enabled` and `.Disabled` lists
+3. Construct feature gate strings like `"FeatureName=true"` or `"FeatureName=false"`
+4. Add them to the `"feature-gates"` argument in the `APIServerArguments` map
+
+### 4. Converted to CLI Flags (`deps/.../flags.go`)
+```bash
+--feature-gates=UserNamespacesSupport=true
+--feature-gates=UserNamespacesPodSecurityStandards=true
+```
+
+### 5. Parsed by Kube-APIServer
+The kube-apiserver's flag parser reads these flags and enables the feature gates.
+
+## Summary
+
+The data flow is:
+
+1. **YAML file** → `ActiveConfig()` → **Config struct**
+2. **Config struct** → `KubeAPIServer.configure()` → **KubeAPIServerConfig struct**
+3. **KubeAPIServerConfig** → Marshaled to **temp YAML file**
+4. **Temp YAML file** → `GetOpenshiftConfig()` → **KubeAPIServerConfig struct** (in kube-apiserver process)
+5. **KubeAPIServerConfig** → `ConfigToFlags()` → **CLI flags array**
+6. **CLI flags** → `cmd.ParseFlags()` → **ServerRunOptions** (internal kube-apiserver state)
+7. **ServerRunOptions** → Used to configure and start the actual API server
+
+The key transformation points are:
+- **Config YAML → Config struct**: Standard YAML unmarshaling
+- **Config.ApiServer → KubeAPIServerConfig.APIServerArguments**: Manual mapping in `configure()`
+- **KubeAPIServerConfig → CLI flags**: `ConfigToFlags()` conversion
+- **CLI flags → Runtime config**: Cobra flag parsing in kube-apiserver
+
+All `apiServer` fields from the MicroShift config eventually become command-line flags that are parsed by the standard Kubernetes kube-apiserver flag parser.
+
diff --git a/docs/user/howto_config.md b/docs/user/howto_config.md
index a858de3c2a..79c611f335 100644
--- a/docs/user/howto_config.md
+++ b/docs/user/howto_config.md
@@ -14,6 +14,11 @@ apiServer:
maxFileSize: 0
maxFiles: 0
profile: ""
+ featureGates:
+ customNoUpgrade:
+ disabled: []
+ enabled: []
+ featureSet: ""
namedCertificates:
- certPath: ""
keyPath: ""
@@ -155,6 +160,11 @@ apiServer:
maxFileSize: 200
maxFiles: 10
profile: Default
+ featureGates:
+ customNoUpgrade:
+ disabled: []
+ enabled: []
+ featureSet: ""
namedCertificates:
- certPath: ""
keyPath: ""
diff --git a/packaging/microshift/config.yaml b/packaging/microshift/config.yaml
index 0cbcda48cb..783435e687 100644
--- a/packaging/microshift/config.yaml
+++ b/packaging/microshift/config.yaml
@@ -30,6 +30,11 @@ apiServer:
# to serve from the API server. Allowed values: VersionTLS12, VersionTLS13.
# Defaults to VersionTLS12.
minVersion: VersionTLS12
+ featureGates:
+ featureSet: ""
+ customNoUpgrade:
+ enabled: []
+ disabled: []
debugging:
# Valid values are: "Normal", "Debug", "Trace", "TraceAll".
# Defaults to "Normal".
diff --git a/pkg/config/apiserver.go b/pkg/config/apiserver.go
index 621950d391..baddb09721 100644
--- a/pkg/config/apiserver.go
+++ b/pkg/config/apiserver.go
@@ -2,9 +2,12 @@ package config
import (
"fmt"
+ "reflect"
"slices"
+ "strings"
configv1 "github.com/openshift/api/config/v1"
+ featuresUtils "github.com/openshift/api/features"
"github.com/openshift/library-go/pkg/crypto"
)
@@ -27,6 +30,8 @@ type ApiServer struct {
TLS TLSConfig `json:"tls"`
+ FeatureGates FeatureGates `json:"featureGates"`
+
// The URL and Port of the API server cannot be changed by the user.
URL string `json:"-"`
Port int `json:"-"`
@@ -127,3 +132,76 @@ func (t *TLSConfig) Validate() error {
func getIANACipherSuites(suites []string) []string {
return crypto.OpenSSLToIANACipherSuites(suites)
}
+
+const (
+ FeatureSetCustomNoUpgrade = "CustomNoUpgrade"
+ FeatureSetTechPreviewNoUpgrade = "TechPreviewNoUpgrade"
+ FeatureSetDevPreviewNoUpgrade = "DevPreviewNoUpgrade"
+)
+
+type CustomNoUpgrade struct {
+ Enabled []string `json:"enabled"`
+ Disabled []string `json:"disabled"`
+}
+
+type FeatureGates struct {
+ FeatureSet string `json:"featureSet"`
+ CustomNoUpgrade CustomNoUpgrade `json:"customNoUpgrade"`
+}
+
+func (fg FeatureGates) ConvertToCLIFlags() ([]string, error) {
+ ret := []string{}
+
+ switch fg.FeatureSet {
+ case FeatureSetCustomNoUpgrade:
+ for _, feature := range fg.CustomNoUpgrade.Enabled {
+ ret = append(ret, fmt.Sprintf("%s=true", feature))
+ }
+ for _, feature := range fg.CustomNoUpgrade.Disabled {
+ ret = append(ret, fmt.Sprintf("%s=false", feature))
+ }
+ case FeatureSetDevPreviewNoUpgrade, FeatureSetTechPreviewNoUpgrade:
+ fgEnabledDisabled, err := featuresUtils.FeatureSets(featuresUtils.SelfManaged, configv1.FeatureSet(fg.FeatureSet))
+ if err != nil {
+ return nil, fmt.Errorf("failed to get feature set gates: %w", err)
+ }
+ for _, f := range fgEnabledDisabled.Enabled {
+ ret = append(ret, fmt.Sprintf("%s=true", f.FeatureGateAttributes.Name))
+ }
+ for _, f := range fgEnabledDisabled.Disabled {
+ ret = append(ret, fmt.Sprintf("%s=false", f.FeatureGateAttributes.Name))
+ }
+ }
+ return ret, nil
+}
+
+// Implement the GoStringer interface for better %#v printing
+func (fg FeatureGates) GoString() string {
+ return fmt.Sprintf("FeatureGates{FeatureSet: %q, CustomNoUpgrade: %#v}", fg.FeatureSet, fg.CustomNoUpgrade)
+}
+
+func (fg *FeatureGates) validateFeatureGates() error {
+ // FG is unset
+ if fg == nil || reflect.DeepEqual(*fg, FeatureGates{}) {
+ return nil
+ }
+ // Must use a recognized feature set, or else empty
+ if fg.FeatureSet != "" && fg.FeatureSet != FeatureSetCustomNoUpgrade && fg.FeatureSet != FeatureSetTechPreviewNoUpgrade && fg.FeatureSet != FeatureSetDevPreviewNoUpgrade {
+ return fmt.Errorf("invalid feature set: %s", fg.FeatureSet)
+ }
+ // Must set FeatureSet to CustomNoUpgrade to use custom feature gates
+ if fg.FeatureSet != FeatureSetCustomNoUpgrade && (len(fg.CustomNoUpgrade.Enabled) > 0 || len(fg.CustomNoUpgrade.Disabled) > 0) {
+ return fmt.Errorf("CustomNoUpgrade must be empty when FeatureSet is empty")
+ }
+ // Must not have any feature gates that are enabled and disabled at the same time
+ var illegalFeatures []string
+ for _, enabledFeature := range fg.CustomNoUpgrade.Enabled {
+ if slices.Contains(fg.CustomNoUpgrade.Disabled, enabledFeature) {
+ illegalFeatures = append(illegalFeatures, enabledFeature)
+ }
+ }
+ if len(illegalFeatures) > 0 {
+ return fmt.Errorf("featuregates cannot be enabled and disabled at the same time: %s", strings.Join(illegalFeatures, ", "))
+ }
+ return nil
+}
diff --git a/pkg/config/config.go b/pkg/config/config.go
index 8e9c645262..37451aa780 100644
--- a/pkg/config/config.go
+++ b/pkg/config/config.go
@@ -56,7 +56,6 @@ type Config struct {
Ingress IngressConfig `json:"ingress"`
Storage Storage `json:"storage"`
Telemetry Telemetry `json:"telemetry"`
-
// Settings specified in this section are transferred as-is into the Kubelet config.
// +kubebuilder:validation:Schemaless
Kubelet map[string]any `json:"kubelet"`
@@ -418,6 +417,15 @@ func (c *Config) incorporateUserSettings(u *Config) {
if u.Ingress.AccessLogging.HttpCaptureCookies != nil {
c.Ingress.AccessLogging.HttpCaptureCookies = u.Ingress.AccessLogging.HttpCaptureCookies
}
+ if u.ApiServer.FeatureGates.FeatureSet != "" {
+ c.ApiServer.FeatureGates.FeatureSet = u.ApiServer.FeatureGates.FeatureSet
+ }
+ if len(u.ApiServer.FeatureGates.CustomNoUpgrade.Enabled) > 0 {
+ c.ApiServer.FeatureGates.CustomNoUpgrade.Enabled = u.ApiServer.FeatureGates.CustomNoUpgrade.Enabled
+ }
+ if len(u.ApiServer.FeatureGates.CustomNoUpgrade.Disabled) > 0 {
+ c.ApiServer.FeatureGates.CustomNoUpgrade.Disabled = u.ApiServer.FeatureGates.CustomNoUpgrade.Disabled
+ }
}
// updateComputedValues examins the existing settings and converts any
@@ -647,6 +655,10 @@ func (c *Config) validate() error {
return fmt.Errorf("error validating Generic Device Plugin configuration: %v", err)
}
+ if err := c.ApiServer.FeatureGates.validateFeatureGates(); err != nil {
+ return fmt.Errorf("error validating feature gates: %v", err)
+ }
+
return nil
}
diff --git a/pkg/config/config_test.go b/pkg/config/config_test.go
index e484a04576..abbbd1f6f7 100644
--- a/pkg/config/config_test.go
+++ b/pkg/config/config_test.go
@@ -800,6 +800,56 @@ func TestValidate(t *testing.T) {
}(),
expectErr: true,
},
+ {
+ name: "feature-gates-unset",
+ config: func() *Config {
+ c := mkDefaultConfig()
+ c.ApiServer.FeatureGates = FeatureGates{}
+ return c
+ }(),
+ expectErr: false,
+ },
+ {
+ name: "feature-gates-invalid-feature-set",
+ config: func() *Config {
+ c := mkDefaultConfig()
+ c.ApiServer.FeatureGates.FeatureSet = "invalid"
+ return c
+ }(),
+ expectErr: true,
+ },
+ {
+ name: "feature-gates-custom-no-upgrade-with-feature-set",
+ config: func() *Config {
+ c := mkDefaultConfig()
+ c.ApiServer.FeatureGates.FeatureSet = "CustomNoUpgrade"
+ c.ApiServer.FeatureGates.CustomNoUpgrade.Enabled = []string{"feature1"}
+ c.ApiServer.FeatureGates.CustomNoUpgrade.Disabled = []string{"feature2"}
+ return c
+ }(),
+ },
+ {
+ name: "feature-gates-custom-no-upgrade-with-feature-set-empty",
+ config: func() *Config {
+ c := mkDefaultConfig()
+ c.ApiServer.FeatureGates.FeatureSet = ""
+ c.ApiServer.FeatureGates.CustomNoUpgrade.Enabled = []string{"feature1"}
+ c.ApiServer.FeatureGates.CustomNoUpgrade.Disabled = []string{"feature2"}
+ return c
+ }(),
+ expectErr: true,
+ },
+ {
+ name: "feature-gates-custom-no-upgrade-enabled-and-disabled-have-same-feature-gate",
+ config: func() *Config {
+ c := mkDefaultConfig()
+ c.ApiServer.FeatureGates.FeatureSet = "CustomNoUpgrade"
+ c.ApiServer.FeatureGates.CustomNoUpgrade.Enabled = []string{"feature1"}
+ c.ApiServer.FeatureGates.CustomNoUpgrade.Disabled = []string{"feature1"}
+ return c
+ }(),
+ expectErr: true,
+ },
}
for _, tt := range ttests {
t.Run(tt.name, func(t *testing.T) {
diff --git a/pkg/controllers/kube-apiserver.go b/pkg/controllers/kube-apiserver.go
index 41b4ac3976..4c8eb77471 100644
--- a/pkg/controllers/kube-apiserver.go
+++ b/pkg/controllers/kube-apiserver.go
@@ -63,7 +63,6 @@ var (
embedded.MustAsset("controllers/kube-apiserver/config-overrides.yaml"),
}
)
-
var fixedTLSProfile *configv1.TLSProfileSpec
func init() {
@@ -169,6 +168,15 @@ func (s *KubeAPIServer) configure(ctx context.Context, cfg *config.Config) error
return fmt.Errorf("failed to discover etcd servers: %w", err)
}
+ featureGateArgs, err := cfg.ApiServer.FeatureGates.ConvertToCLIFlags()
+ if err != nil {
+ return fmt.Errorf("failed to convert feature gates to CLI flags: %w", err)
+ }
+
+ // Build the final feature gates list
+ hardcodedFeatureGates := []string{"UserNamespacesSupport=true", "UserNamespacesPodSecurityStandards=true"}
+ featureGateArgs = append(featureGateArgs, hardcodedFeatureGates...)
+
overrides := &kubecontrolplanev1.KubeAPIServerConfig{
APIServerArguments: map[string]kubecontrolplanev1.Arguments{
"advertise-address": {s.advertiseAddress},
@@ -221,7 +229,7 @@ func (s *KubeAPIServer) configure(ctx context.Context, cfg *config.Config) error
"enable-admission-plugins": {},
"send-retry-after-while-not-ready-once": {"true"},
"shutdown-delay-duration": {"5s"},
- "feature-gates": {"UserNamespacesSupport=true", "UserNamespacesPodSecurityStandards=true"},
+ "feature-gates": featureGateArgs,
},
GenericAPIServerConfig: configv1.GenericAPIServerConfig{
AdmissionConfig: configv1.AdmissionConfig{
diff --git a/sequenceDiagram.mmd b/sequenceDiagram.mmd
new file mode 100644
index 0000000000..1744c54139
--- /dev/null
+++ b/sequenceDiagram.mmd
@@ -0,0 +1,73 @@
+sequenceDiagram
+ participant ConfigFile as config.yaml
+ participant ActiveConfig as config.ActiveConfig()
+ participant FillDefaults as config.fillDefaults()
+ participant IncorporateSettings as config.incorporateUserSettings()
+ participant UpdateComputed as config.updateComputedValues()
+ participant KASRun as KubeAPIServer.Run()
+ participant KASConfigure as KubeAPIServer.configure()
+ participant Merge as resourcemerge.MergePrunedProcessConfig()
+ participant TempFile as Create Temp File
+ participant NewCmd as kubeapiserver.NewAPIServerCommand()
+ participant Execute as cmd.ExecuteContext()
+ participant GetOpenshiftConfig as enablement.GetOpenshiftConfig()
+ participant ConfigToFlags as openshiftkubeapiserver.ConfigToFlags()
+ participant ParseFlags as cmd.ParseFlags()
+ participant KASMain as Run() [kube-apiserver]
+
+ Note over ConfigFile: User configuration in
/etc/microshift/config.yaml
apiServer:
advertiseAddress: "10.43.0.2"
port: 6443
auditLog:
profile: "Default"
tls:
minVersion: "VersionTLS12"
cipherSuites: [...]
featureGates:
featureSet: "CustomNoUpgrade"
customNoUpgrade:
enabled: ["UserNamespacesSupport"]
+
+ ConfigFile->>+ActiveConfig: Read config.yaml
[files.go:120]
+
+ ActiveConfig->>+FillDefaults: Set default values
[config.go:92]
+ Note right of FillDefaults: Sets default ApiServer values:
- Port: 6443
- URL: "https://localhost:6443"
- AuditLog defaults
- SubjectAltNames
+ FillDefaults-->>-ActiveConfig: Config with defaults
+
+ ActiveConfig->>+IncorporateSettings: Merge user settings
[config.go:195]
+ Note right of IncorporateSettings: Merges user-provided values:
- ApiServer.AdvertiseAddress
- ApiServer.AuditLog.*
- ApiServer.TLS.*
- ApiServer.NamedCertificates
- ApiServer.FeatureGates.*
+ IncorporateSettings-->>-ActiveConfig: Config with user settings
+
+ ActiveConfig->>+UpdateComputed: Compute derived values
[config.go:434]
+ Note right of UpdateComputed: Computes:
- AdvertiseAddresses (from ServiceNetwork)
- TLS.UpdateValues() - cipher suites
- SkipInterface flag
+ UpdateComputed-->>-ActiveConfig: Final Config
+
+ ActiveConfig-->>-ConfigFile: *config.Config
+
+ Note over KASRun: MicroShift creates and starts
kube-apiserver controller
+
+ KASRun->>+KASConfigure: configure(ctx, cfg)
[kube-apiserver.go:97]
+
+ Note right of KASConfigure: Uses cfg.ApiServer fields to build
KubeAPIServerConfig overrides:
- advertise-address
- audit-policy-file
- audit-log-*
- tls-cert-file
- tls-private-key-file
- tls-min-version
- tls-cipher-suites
- feature-gates
- service-cluster-ip-range
- disable-admission-plugins
- enable-admission-plugins
+
+ KASConfigure->>+Merge: Merge config layers
[kube-apiserver.go:267]
+ Note right of Merge: Merges 3 layers:
1. defaultconfig.yaml (embedded)
2. config-overrides.yaml (embedded)
3. Runtime overrides from cfg.ApiServer
Creates KubeAPIServerConfig struct
with all settings merged
+ Merge-->>-KASConfigure: kasConfigBytes (marshaled YAML)
+
+ KASConfigure-->>-KASRun: Configuration complete
+
+ KASRun->>TempFile: Create temp config file
[kube-apiserver.go:363]
+ Note right of TempFile: Write kasConfigBytes to:
/tmp/kube-apiserver-config-*.yaml
+
+ KASRun->>+NewCmd: NewAPIServerCommand()
[kube-apiserver.go:391]
+ Note right of NewCmd: Initialize kube-apiserver command
from kubernetes/cmd/kube-apiserver
+ NewCmd-->>-KASRun: cmd *cobra.Command
+
+ KASRun->>+Execute: cmd.ExecuteContext(ctx)
[kube-apiserver.go:404]
+ Note right of Execute: Args:
--openshift-config /tmp/kube-apiserver-config-*.yaml
-v [verbosity]
+
+ Execute->>+GetOpenshiftConfig: Read config file
[intialization.go:27]
+ Note right of GetOpenshiftConfig: Reads temp config file and
deserializes into
KubeAPIServerConfig struct
Resolves file paths
Sets recommended defaults
+ GetOpenshiftConfig-->>-Execute: *KubeAPIServerConfig
+
+ Execute->>+ConfigToFlags: Convert to CLI flags
[flags.go:18]
+ Note right of ConfigToFlags: Converts KubeAPIServerConfig.APIServerArguments
to command-line flags:
Map[string][]string -> []string
Examples:
"advertise-address": ["10.43.0.2"]
-> --advertise-address=10.43.0.2
"feature-gates": ["UserNamespacesSupport=true"]
-> --feature-gates=UserNamespacesSupport=true
Also converts:
- ServingInfo -> tls-* flags
- AuditConfig -> audit-* flags
- AdmissionConfig -> admission-* flags
+ ConfigToFlags-->>-Execute: []string (CLI flags)
+
+ Execute->>+ParseFlags: Parse merged flags
[server.go:122]
+ Note right of ParseFlags: Re-parses command flags with
values from OpenShift config
This updates the ServerRunOptions
with all the ApiServer settings
+ ParseFlags-->>-Execute: Updated options
+
+ Execute->>+KASMain: Run(ctx, completedOptions)
[server.go:153]
+ Note right of KASMain: Kube-apiserver starts with
all configuration applied:
- TLS settings
- Audit logging
- Feature gates
- Admission plugins
- Serving configuration
+
+ Note over KASMain: Kube-apiserver running with
all config.ApiServer values
applied via command-line flags
\ No newline at end of file