Dokku ships with default charts and customizes their helm values. In certain cases, it may be useful to further customize those values to better support specific installations. With this change, it is now possible to set custom values by providing the key in the format `chart.$CHART_NAME.$PROPERTY`. For instance, to increase timeouts for the keda http addon:
```shell
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tcpConnectTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.keepAlive 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.responseHeaderTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.idleConnTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tlsHandshakeTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.expectContinueTimeout 120s
```
To reset a value to its default, simply set the value to an empty string.
```shell
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tcpConnectTimeout
```
Once the values are set, users will need to then run `scheduler-k3s:ensure-charts` with either the `--force` flag or specify the `--chart` flag with a list of charts to force apply.
```shell
dokku scheduler-k3s:ensure-charts --force
dokku scheduler-k3s:ensure-charts --chart keda-add-ons-http
```
Note that the existing dokku-provided values cannot be removed, but can be overridden.
This will provide the possibility for users to talk to existing kubernetes clusters, thereby removing one of the biggest reasons for having the old scheduler-kubernetes plugin around.
This is almost certainly the correct default for Dokku. While it's a BC break and might cause an increase in memory usage, the api is mostly internal and therefore this is safe to use.
While this will now more or less require a local docker login - which can be overridden by setting the image-pull-secrets property - this ensures every deploy has a valid context.
One annoying thing is that the secret is per-app vs global, meaning that if the secret must be rotated, all apps need to be redeployed. A future change can fix that by deploying a global secret, or if folks really desire, this can be done by setting the image-pull-secrets property on the app/globally.
The previous mechanism of constructing the helm chart was pretty obnoxious due to needing to have some things in the object and some in template files that were appended afterwards. This change simplifies that by using template files per type with minimal replacements.
The idea here is that only k3s nodes can/should handle certificate requests. While users may wish to force these onto dedicated lb nodes in the future, at least for now, separating app workloads with routing seems like an okay trade-off.
Different options get applied to each depending on what is needed for the role.
This also updates how we initialize the cluster to better support how metrics exposure and cross-server networking.
The big feature missing from deployments today is healthchecks, which require extra parsing of the app.json format since the healthchecks there aren't immediately serializable to kubernetes deployment healthchecks.
Other smaller missing functionality from the existing OSS kubernetes scheduler are:
- The ability to set custom pod annotations
- The ability to set custom deployment annotations
- The ability to mount volumes
Exposing these pieces of functionality will come at a later date, if at all.
