Scheduled cron task output previously reached only the `dokku` user's cron mail, and could not be redirected because `app.json` rejects bare shell operators in a cron `command`. Setting `vector-cron-sink` on an app or globally routes that output to a dedicated sink instead, on both the `docker-local` and `k3s` schedulers, which keeps log destinations under operator control rather than in a deployed repository. Cron events carry `dokku_app` and `dokku_cron_id` fields so a sink can give each task its own destination. This also fixes a `k3s` bug where configuring a global `vector-sink` silently removed the vector prometheus exporter sink.
The `cert-issuer-name` and `cert-issuer-kind` properties point an app's generated `Certificate` at a cert-manager issuer created outside of Dokku, allowing certificates to be issued through solvers the built-in letsencrypt integration cannot use, such as `dns01` for wildcard certificates. Setting an issuer enables https on its own, as a manually managed issuer has no email for Dokku to configure. An imported certificate still takes precedence, and `letsencrypt-server false` remains the single off switch. Dokku warns before a build starts when the referenced issuer is absent from the cluster, without blocking the deploy. Wildcard domains no longer collide with their apex domain when generating ingress names, and `letsencrypt-server` values are now validated when set rather than at deploy time.
Sysctls the kernel does not namespace, such as `vm.max_map_count`, cannot be set from a pod spec and previously had no answer beyond editing `/etc/sysctl.d` on each host by hand. `scheduler-k3s:node-sysctls:set` now applies them through a privileged daemonset, which reaches nodes joined later and reapplies after a reboot. Sysctls may be scoped to a node profile, with a profile scope inheriting the global values and overriding them on conflict so that every node is covered by exactly one daemonset. Clearing a sysctl stops dokku managing it but does not restore the previous value, which persists until the node reboots.
The `docker-local` scheduler supports `--sysctl` for free because docker options are passed verbatim to `docker run`, but the k3s scheduler silently dropped it. Namespaced sysctls now render into the pod's `securityContext.sysctls` for deployments, cron jobs, and one-off runs. A sysctl the kernel does not namespace fails the deploy instead of being dropped, since it cannot take effect within a pod regardless of what was requested.
Node profiles controlled how a node joined the cluster but left no trace on the node afterwards, so a profile could not be selected against with `kubectl`, a `nodeSelector`, or a node affinity rule. Nodes joined without a profile are left unlabeled, and the server node created by `scheduler-k3s:initialize` never carries the label since it does not pass through `scheduler-k3s:cluster:add`.
The server node created by `scheduler-k3s:initialize` had no way to receive kubelet arguments, unlike nodes joined through `scheduler-k3s:cluster:add` or configured via `scheduler-k3s:profiles:add`. This meant settings such as `allowed-unsafe-sysctls` were unreachable on a single-node install.
The `letsencrypt-email-prod` and `letsencrypt-email-stag` properties can now be set per app in addition to globally, resolving app-level before the global value for the app's selected `letsencrypt-server`. An app that sets its own email renders a namespaced cert-manager `Issuer` using that email, while apps without an override continue to use the shared `ClusterIssuer` with the global email.
Annotation and label values containing a newline previously round-tripped
as two adjacent `key: value` lines, and the second line then failed the
`SplitN(line, ": ", 2)` parse on read. Move the per-`(processType,
resourceType)` files to `PropertyMap*` storage so `\n` in values and `/`
in keys are preserved verbatim. An idempotent `TriggerInstall` migration
walks `--global` and every app, converting any legacy line-formatted file
in place by probing `PropertyMapGet` first and rewriting via
`PropertyListGet` only when the probe fails. Property names are
unchanged, so the annotations/labels report scanner and the
`reservedAnnotationPrefixes` filter keep working without modification.
Closes#8719.
Chart property names containing `/` failed because per-key flat-file storage interpreted the slash as a filesystem path separator, and any move to line-based storage would silently truncate multi-line values. A new `PropertyMap*` helper family in `common` persists each map as a single JSON file so both `/` in keys and `\n` in values round-trip losslessly. Chart overrides move to `chart-overrides.<chart>` and an idempotent `TriggerInstall` migration rewrites any legacy `chart.<chart>.<key>` files into the new layout. The deprecated `scheduler-k3s:set --global chart.*` form is rerouted through the same map storage so it does not silently orphan writes. Closes#8717.
Three plugins still leaked the built-in default through `<plugin>-global-<property>` after the broader raw/computed split in #8640. External tooling reading `:report --format json` for drift detection could not distinguish "set to default" from "never set" - after a `:set --global` followed by an unset, the JSON still reported the default value. The `<plugin>-global-<property>` keys now hold the raw stored value (empty when nothing has been set) and `<plugin>-computed-<property>` keys hold the effective value with per-app, global, and built-in default fallback. Every global property now has a computed sibling for shape consistency, including those whose default is empty. Internal callers in `scheduler-k3s` that previously read the default-applied global helpers now consume new `getComputed*` helpers, keeping deploy-time behavior unchanged.
The `dokku.com/cron-id` label could exceed Kubernetes' 63-byte cap because the cron ID is `base36(appName === command === schedule)`, which expands roughly 1.5x per byte. The label is now keyed `dokku.com/cron-hash` and holds the `sha1` hex digest of the cron-id, a fixed 40-character value that always fits the cap. The same hex digest is mirrored into the `dokku.com/cron-hash` annotation, and the original base36 cron-id stays in the `dokku.com/cron-id` annotation that `cron:list` reads when surfacing user-facing IDs. Per-task lookups stay server-side via label selectors, so `cron:set --maintenance` and the forbid/replace concurrency checks on `dokku run --cron-id` keep working without any in-memory filtering.
The cron-id label could exceed Kubernetes' 63-byte cap when commands or
schedules were long, and an all-digit job-suffix or cron-id rendered as
an unquoted YAML scalar caused the API server to reject manifests. Run
pods built from dockerfiles also occasionally hit the 10s startup wait
on a cold image pull, even though the pod was scheduled correctly.
The cron-id is now stored as an annotation and a shorter hash is used as
the selector label. Every interpolated annotation and label value in the
cron-job and deployment templates is now quoted to prevent numeric
coercion, and the run-pod wait timeout is raised to 30 seconds.
After streaming logs from a run pod on scheduler-k3s, the apiserver may still report `PodRunning` for a short window due to kubelet status propagation lag, causing `dokku run` to fail with `Unable to attach as the pod is in an unknown state: Running`. Wait briefly for the pod to reach a terminal phase before classifying the outcome.
Bundling these Secrets in the app helm chart caused two bugs in the scheduler-k3s plugin: a chart rollback could delete Secrets that older ReplicaSets still referenced by exact timestamped name (`env-{app}.{ts}` and `ims-{app}.{ts}`), hard-crashing pods until manual intervention; and the strategic-merge `patchMergeKey` on `imagePullSecrets` let stale entries leak into the live Deployment until the list pointed at many nonexistent Secrets. Each Secret now lives in its own helm release with a stable name (`config-{app}` and `pull-secret-{app}`), installed before the app chart on every deploy. The deployment trigger also prunes any leaked `imagePullSecrets` entries from the live Deployment so the next deploy lands on a clean list, and the rename and destroy paths uninstall the new releases (and the previously-leaked TLS release on rename) under the old app name.
Some helm chart upgrades require side-effects that a plain `helm upgrade` cannot perform, such as deleting deployments whose immutable selectors changed. Charts may now register pre and post upgrade hooks against a target version; applicable hooks fire in ascending semver order, each bracketed around an upgrade to its version, with a final upgrade to the chart's configured version when needed. The new `keda-add-ons-http` 0.12.2 hook deletes the chart-managed deployments before the upgrade so the new selectors take effect cleanly.
These shadow existing values managed by k3s itself, causing issues where the same header is injected with multiple values, which some frameworks do not support as they use the header directly without parsing to construct urls.
If a global vector-sink property on the logs plugin is set, we will use that to configure a log sink (defaulting to console output otherwise). Note that the actual configuration still needs to be a valid DSN representing a Vector log sink.
While this does not support pinning log sinks to specific apps, it opens up integration for those who use a single k3s cluster in shared tenancy.
There is a bug in the Keda project that caused Keda to incorrectly handle empty trigger lists. If self-managing the ScaledObject when using an HTTPScaledObject, an external-push trigger must be created.
This change additionally updates the chart to fix any potential bugs in usage of keda.
Dokku ships with default charts and customizes their helm values. In certain cases, it may be useful to further customize those values to better support specific installations. With this change, it is now possible to set custom values by providing the key in the format `chart.$CHART_NAME.$PROPERTY`. For instance, to increase timeouts for the keda http addon:
```shell
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tcpConnectTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.keepAlive 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.responseHeaderTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.idleConnTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tlsHandshakeTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.expectContinueTimeout 120s
```
To reset a value to its default, simply set the value to an empty string.
```shell
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tcpConnectTimeout
```
Once the values are set, users will need to then run `scheduler-k3s:ensure-charts` with either the `--force` flag or specify the `--chart` flag with a list of charts to force apply.
```shell
dokku scheduler-k3s:ensure-charts --force
dokku scheduler-k3s:ensure-charts --chart keda-add-ons-http
```
Note that the existing dokku-provided values cannot be removed, but can be overridden.
Previously, a container that exited immediately would just show an error message, which is probably not what we want.
If the pod has exited, we will display the logs on stdout for the user and then exit appropriately.
This change also refactors how log streaming works so that we respect exiting when the stream has no content (such as when the container has exited). Previously, the container would just hang around forever due to not being properly canceled. Due to this refactor, the prefix is now split out into a separate writer.
If the scheduler is not docker-local, we will assume that a separate plugin trigger will pull access/error logs on our behalf. For k3s, the naive implementation is to pull logs from a single container (multiple ingress-nginx pods will be supported in a future release).
Closes#7268
If keda is not installed, warn the user that it isn't installed and ignore the rest.
Ignoring the error here is fine as it gives the user context but doesn't block a deploy.
