Three plugins still leaked the built-in default through `<plugin>-global-<property>` after the broader raw/computed split in #8640. External tooling reading `:report --format json` for drift detection could not distinguish "set to default" from "never set" - after a `:set --global` followed by an unset, the JSON still reported the default value. The `<plugin>-global-<property>` keys now hold the raw stored value (empty when nothing has been set) and `<plugin>-computed-<property>` keys hold the effective value with per-app, global, and built-in default fallback. Every global property now has a computed sibling for shape consistency, including those whose default is empty. Internal callers in `scheduler-k3s` that previously read the default-applied global helpers now consume new `getComputed*` helpers, keeping deploy-time behavior unchanged.
Node profiles can be used to specify repetitive options for clusters, making it easier to add specific types of instances quickly.
This PR additionally fixes kubelet-args specification - the same key can be specified multiple times, such as when specifying custom sysctl rules.
Previously, it was possible to have a kubernetes version mismatch on new worker nodes if a newer version of kubernetes was released prior to upgrading the master nodes to that version. This can sometimes cause cluster instability.
This change fetches the lowest node version associated with a master node and then uses that to install k3s on the worker node.
Dokku ships with default charts and customizes their helm values. In certain cases, it may be useful to further customize those values to better support specific installations. With this change, it is now possible to set custom values by providing the key in the format `chart.$CHART_NAME.$PROPERTY`. For instance, to increase timeouts for the keda http addon:
```shell
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tcpConnectTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.keepAlive 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.responseHeaderTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.idleConnTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tlsHandshakeTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.expectContinueTimeout 120s
```
To reset a value to its default, simply set the value to an empty string.
```shell
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tcpConnectTimeout
```
Once the values are set, users will need to then run `scheduler-k3s:ensure-charts` with either the `--force` flag or specify the `--chart` flag with a list of charts to force apply.
```shell
dokku scheduler-k3s:ensure-charts --force
dokku scheduler-k3s:ensure-charts --chart keda-add-ons-http
```
Note that the existing dokku-provided values cannot be removed, but can be overridden.
This will provide the possibility for users to talk to existing kubernetes clusters, thereby removing one of the biggest reasons for having the old scheduler-kubernetes plugin around.
This is almost certainly the correct default for Dokku. While it's a BC break and might cause an increase in memory usage, the api is mostly internal and therefore this is safe to use.
While this will now more or less require a local docker login - which can be overridden by setting the image-pull-secrets property - this ensures every deploy has a valid context.
One annoying thing is that the secret is per-app vs global, meaning that if the secret must be rotated, all apps need to be redeployed. A future change can fix that by deploying a global secret, or if folks really desire, this can be done by setting the image-pull-secrets property on the app/globally.
The previous mechanism of constructing the helm chart was pretty obnoxious due to needing to have some things in the object and some in template files that were appended afterwards. This change simplifies that by using template files per type with minimal replacements.
The idea here is that only k3s nodes can/should handle certificate requests. While users may wish to force these onto dedicated lb nodes in the future, at least for now, separating app workloads with routing seems like an okay trade-off.
Different options get applied to each depending on what is needed for the role.
This also updates how we initialize the cluster to better support how metrics exposure and cross-server networking.
The big feature missing from deployments today is healthchecks, which require extra parsing of the app.json format since the healthchecks there aren't immediately serializable to kubernetes deployment healthchecks.
Other smaller missing functionality from the existing OSS kubernetes scheduler are:
- The ability to set custom pod annotations
- The ability to set custom deployment annotations
- The ability to mount volumes
Exposing these pieces of functionality will come at a later date, if at all.
