The `cert-issuer-name` and `cert-issuer-kind` properties point an app's generated `Certificate` at a cert-manager issuer created outside of Dokku, allowing certificates to be issued through solvers the built-in letsencrypt integration cannot use, such as `dns01` for wildcard certificates. Setting an issuer enables https on its own, as a manually managed issuer has no email for Dokku to configure. An imported certificate still takes precedence, and `letsencrypt-server false` remains the single off switch. Dokku warns before a build starts when the referenced issuer is absent from the cluster, without blocking the deploy. Wildcard domains no longer collide with their apex domain when generating ingress names, and `letsencrypt-server` values are now validated when set rather than at deploy time.
Sysctls the kernel does not namespace, such as `vm.max_map_count`, cannot be set from a pod spec and previously had no answer beyond editing `/etc/sysctl.d` on each host by hand. `scheduler-k3s:node-sysctls:set` now applies them through a privileged daemonset, which reaches nodes joined later and reapplies after a reboot. Sysctls may be scoped to a node profile, with a profile scope inheriting the global values and overriding them on conflict so that every node is covered by exactly one daemonset. Clearing a sysctl stops dokku managing it but does not restore the previous value, which persists until the node reboots.
Node profiles controlled how a node joined the cluster but left no trace on the node afterwards, so a profile could not be selected against with `kubectl`, a `nodeSelector`, or a node affinity rule. Nodes joined without a profile are left unlabeled, and the server node created by `scheduler-k3s:initialize` never carries the label since it does not pass through `scheduler-k3s:cluster:add`.
The server node created by `scheduler-k3s:initialize` had no way to receive kubelet arguments, unlike nodes joined through `scheduler-k3s:cluster:add` or configured via `scheduler-k3s:profiles:add`. This meant settings such as `allowed-unsafe-sysctls` were unreachable on a single-node install.
Align autoscaling-auth:report with annotations and labels reporting so export tools can recover configured trigger auth via flat JSON keys and info flags, while stdout stays secret-safe unless --include-metadata is used.
Closes#8800
Kubernetes 1.20 deprecated the node-role.kubernetes.io/master label and replaced it with node-role.kubernetes.io/control-plane, and 1.24 dropped the legacy label entirely. The scheduler-k3s:cluster:add join path still filtered control-plane nodes with master=true, so on any k3s release built from Kubernetes 1.24+ the lookup matched zero nodes and aborted with "no nodes found in the cluster" before the worker could join.
Annotation and label values containing a newline previously round-tripped
as two adjacent `key: value` lines, and the second line then failed the
`SplitN(line, ": ", 2)` parse on read. Move the per-`(processType,
resourceType)` files to `PropertyMap*` storage so `\n` in values and `/`
in keys are preserved verbatim. An idempotent `TriggerInstall` migration
walks `--global` and every app, converting any legacy line-formatted file
in place by probing `PropertyMapGet` first and rewriting via
`PropertyListGet` only when the probe fails. Property names are
unchanged, so the annotations/labels report scanner and the
`reservedAnnotationPrefixes` filter keep working without modification.
Closes#8719.
Chart property names containing `/` failed because per-key flat-file storage interpreted the slash as a filesystem path separator, and any move to line-based storage would silently truncate multi-line values. A new `PropertyMap*` helper family in `common` persists each map as a single JSON file so both `/` in keys and `\n` in values round-trip losslessly. Chart overrides move to `chart-overrides.<chart>` and an idempotent `TriggerInstall` migration rewrites any legacy `chart.<chart>.<key>` files into the new layout. The deprecated `scheduler-k3s:set --global chart.*` form is rerouted through the same map storage so it does not silently orphan writes. Closes#8717.
The scheduler-k3s plugin exposes `annotations:set` and `labels:set` but no matching report subcommands, so scripts could not inspect the configured state for idempotent management without reading the property files directly. The new `scheduler-k3s:annotations:report` and `scheduler-k3s:labels:report` surface the configured entries with stdout, JSON, single-flag query, and `--process-type` / `--resource-type` filtering, mirroring the recently added `scheduler-k3s:charts:report`. The literal `GlobalProcessType` value `--global` is rendered as `global` in report keys to avoid leading dashes. The `scheduler-k3s:autoscaling-auth:report` command is updated alongside to loop over every app when no app and no `--global` flag is provided, matching the convention used by `scheduler-k3s:report`.
Helm chart overrides on the scheduler-k3s plugin were previously set
through the generic `scheduler-k3s:set --global chart.<chart>.<property>`
interface, which mixed chart-level configuration into the same command
that manages scheduler properties and offered no focused way to inspect
which overrides were configured. A dedicated `scheduler-k3s:charts:set`
sets and clears chart-specific helm values, and a complementary
`scheduler-k3s:charts:report` surfaces configured overrides per chart
with optional JSON output and single-field flag queries. The legacy form
on `scheduler-k3s:set` continues to work but now emits a deprecation
warning pointing users at the new subcommand.
Three plugins still leaked the built-in default through `<plugin>-global-<property>` after the broader raw/computed split in #8640. External tooling reading `:report --format json` for drift detection could not distinguish "set to default" from "never set" - after a `:set --global` followed by an unset, the JSON still reported the default value. The `<plugin>-global-<property>` keys now hold the raw stored value (empty when nothing has been set) and `<plugin>-computed-<property>` keys hold the effective value with per-app, global, and built-in default fallback. Every global property now has a computed sibling for shape consistency, including those whose default is empty. Internal callers in `scheduler-k3s` that previously read the default-applied global helpers now consume new `getComputed*` helpers, keeping deploy-time behavior unchanged.
Node profiles can be used to specify repetitive options for clusters, making it easier to add specific types of instances quickly.
This PR additionally fixes kubelet-args specification - the same key can be specified multiple times, such as when specifying custom sysctl rules.
Previously, it was possible to have a kubernetes version mismatch on new worker nodes if a newer version of kubernetes was released prior to upgrading the master nodes to that version. This can sometimes cause cluster instability.
This change fetches the lowest node version associated with a master node and then uses that to install k3s on the worker node.
Dokku ships with default charts and customizes their helm values. In certain cases, it may be useful to further customize those values to better support specific installations. With this change, it is now possible to set custom values by providing the key in the format `chart.$CHART_NAME.$PROPERTY`. For instance, to increase timeouts for the keda http addon:
```shell
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tcpConnectTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.keepAlive 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.responseHeaderTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.idleConnTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tlsHandshakeTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.expectContinueTimeout 120s
```
To reset a value to its default, simply set the value to an empty string.
```shell
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tcpConnectTimeout
```
Once the values are set, users will need to then run `scheduler-k3s:ensure-charts` with either the `--force` flag or specify the `--chart` flag with a list of charts to force apply.
```shell
dokku scheduler-k3s:ensure-charts --force
dokku scheduler-k3s:ensure-charts --chart keda-add-ons-http
```
Note that the existing dokku-provided values cannot be removed, but can be overridden.
This will provide the possibility for users to talk to existing kubernetes clusters, thereby removing one of the biggest reasons for having the old scheduler-kubernetes plugin around.
This is almost certainly the correct default for Dokku. While it's a BC break and might cause an increase in memory usage, the api is mostly internal and therefore this is safe to use.
While this will now more or less require a local docker login - which can be overridden by setting the image-pull-secrets property - this ensures every deploy has a valid context.
One annoying thing is that the secret is per-app vs global, meaning that if the secret must be rotated, all apps need to be redeployed. A future change can fix that by deploying a global secret, or if folks really desire, this can be done by setting the image-pull-secrets property on the app/globally.
The previous mechanism of constructing the helm chart was pretty obnoxious due to needing to have some things in the object and some in template files that were appended afterwards. This change simplifies that by using template files per type with minimal replacements.
The idea here is that only k3s nodes can/should handle certificate requests. While users may wish to force these onto dedicated lb nodes in the future, at least for now, separating app workloads with routing seems like an okay trade-off.
