Kubernetes pulls the app image itself, so a k3s host is free to reap its local copy while the workload keeps running, which the `registry` plugin already does on its own. Deploys, restarts, `dokku run`, and in-cluster cron no longer assert that the image is present locally, falling back to the metadata recorded in the app's current Helm release. A `ps:restart` naming a single process type now rolls only that process type's pods rather than silently redeploying every one. Apps with an `app.json` postdeploy task still require the image locally, as that task runs on the Dokku host.
`storage:set` now takes `<name> <property> [<value>]` like every other `:set` command, where omitting the value unsets the property. Previously it took flags and could not distinguish an empty value from an omitted one, so nothing it set could ever be cleared. The flag form keeps working and emits a deprecation warning.
Annotations and labels move to `storage:annotations:set`, `storage:annotations:report`, `storage:labels:set`, and `storage:labels:report`, matching the `scheduler-k3s` equivalents. These operate on a single key, so clearing one leaves the rest in place rather than replacing the whole map as the `--annotation` and `--label` flags do.
Scheduled cron task output previously reached only the `dokku` user's cron mail, and could not be redirected because `app.json` rejects bare shell operators in a cron `command`. Setting `vector-cron-sink` on an app or globally routes that output to a dedicated sink instead, on both the `docker-local` and `k3s` schedulers, which keeps log destinations under operator control rather than in a deployed repository. Cron events carry `dokku_app` and `dokku_cron_id` fields so a sink can give each task its own destination. This also fixes a `k3s` bug where configuring a global `vector-sink` silently removed the vector prometheus exporter sink.
Traefik matches hosts exactly, so an app serving a wildcard domain under the `traefik` ingress class had a valid certificate but silently 404d on every request. Wildcard domains now render as a `HostRegexp` rule that matches a single label, the same semantics as a Kubernetes wildcard host, so both ingress classes behave the same. Those routes carry an explicit low priority so an exact domain on any app still wins over another app's wildcard, mirroring ingress-nginx.
The `cert-issuer-name` and `cert-issuer-kind` properties point an app's generated `Certificate` at a cert-manager issuer created outside of Dokku, allowing certificates to be issued through solvers the built-in letsencrypt integration cannot use, such as `dns01` for wildcard certificates. Setting an issuer enables https on its own, as a manually managed issuer has no email for Dokku to configure. An imported certificate still takes precedence, and `letsencrypt-server false` remains the single off switch. Dokku warns before a build starts when the referenced issuer is absent from the cluster, without blocking the deploy. Wildcard domains no longer collide with their apex domain when generating ingress names, and `letsencrypt-server` values are now validated when set rather than at deploy time.
The usage strings for `scheduler-k3s:report` and `scheduler-k3s:set` omitted the `--global` option, which is the only way to reach the scheduler-wide report since a bare `scheduler-k3s:report` iterates every app, and `:report` also omitted `--format stdout|json`. The command listing in the k3s documentation is resynced with the help output, which additionally restores flags that had been dropped from `scheduler-k3s:cluster:add`, `scheduler-k3s:cluster:list`, and `scheduler-k3s:initialize`.
Sysctls the kernel does not namespace, such as `vm.max_map_count`, cannot be set from a pod spec and previously had no answer beyond editing `/etc/sysctl.d` on each host by hand. `scheduler-k3s:node-sysctls:set` now applies them through a privileged daemonset, which reaches nodes joined later and reapplies after a reboot. Sysctls may be scoped to a node profile, with a profile scope inheriting the global values and overriding them on conflict so that every node is covered by exactly one daemonset. Clearing a sysctl stops dokku managing it but does not restore the previous value, which persists until the node reboots.
The `docker-local` scheduler supports `--sysctl` for free because docker options are passed verbatim to `docker run`, but the k3s scheduler silently dropped it. Namespaced sysctls now render into the pod's `securityContext.sysctls` for deployments, cron jobs, and one-off runs. A sysctl the kernel does not namespace fails the deploy instead of being dropped, since it cannot take effect within a pod regardless of what was requested.
Node profiles controlled how a node joined the cluster but left no trace on the node afterwards, so a profile could not be selected against with `kubectl`, a `nodeSelector`, or a node affinity rule. Nodes joined without a profile are left unlabeled, and the server node created by `scheduler-k3s:initialize` never carries the label since it does not pass through `scheduler-k3s:cluster:add`.
The server node created by `scheduler-k3s:initialize` had no way to receive kubelet arguments, unlike nodes joined through `scheduler-k3s:cluster:add` or configured via `scheduler-k3s:profiles:add`. This meant settings such as `allowed-unsafe-sysctls` were unreachable on a single-node install.
Host-crontab generation for `app.json` cron tasks now lives in the `cron` plugin, gated by a new `scheduler-uses-host-cron` trigger that the `docker-local` scheduler answers true while self-managed schedulers such as `k3s` answer false. This lets any host-cron scheduler participate in normal `app.json` cron without coupling to `scheduler-docker-local` or duplicating the crontab writer, while the `k3s` scheduler continues to manage its own in-cluster cron jobs.
Closes#8862.
The `letsencrypt-email-prod` and `letsencrypt-email-stag` properties can now be set per app in addition to globally, resolving app-level before the global value for the app's selected `letsencrypt-server`. An app that sets its own email renders a namespaced cert-manager `Issuer` using that email, while apps without an override continue to use the shared `ClusterIssuer` with the global email.
Align autoscaling-auth:report with annotations and labels reporting so export tools can recover configured trigger auth via flat JSON keys and info flags, while stdout stays secret-safe unless --include-metadata is used.
Closes#8800
Users can override the default 3-line surrounding context per change. Pass
`--context -1` to render the full resource around every change, matching
upstream `helm diff upgrade` behavior.
Adds a `scheduler-k3s:preview <app>` command that renders a unified diff
between the manifests currently stored in the live Helm release for an app
and the manifests that the next deploy would roll out. The diff is produced
entirely via the Helm Go SDK that scheduler-k3s already uses, with no
external `helm` binary or `helm-diff` plugin install required. The
underlying diff and parse logic is vendored from `databus23/helm-diff`
(Apache 2.0) into a new internal package, trimmed of three-way-merge,
release-ownership tracking, and the non-default output formatters.
The chart-construction half of `TriggerSchedulerDeploy` has been extracted
into a reusable `BuildAppChart` helper so deploy and preview share the
same chart materialization without duplicating template logic. Secret
values are redacted by default in preview output; `--show-secrets` and
`--show-secrets-decoded` flags match the upstream `helm diff` UX. Only the
main app helm release is compared in this iteration; auxiliary releases
for config vars, image-pull secrets, and TLS certificates are out of scope
and documented as such.
Annotation and label values containing a newline previously round-tripped
as two adjacent `key: value` lines, and the second line then failed the
`SplitN(line, ": ", 2)` parse on read. Move the per-`(processType,
resourceType)` files to `PropertyMap*` storage so `\n` in values and `/`
in keys are preserved verbatim. An idempotent `TriggerInstall` migration
walks `--global` and every app, converting any legacy line-formatted file
in place by probing `PropertyMapGet` first and rewriting via
`PropertyListGet` only when the probe fails. Property names are
unchanged, so the annotations/labels report scanner and the
`reservedAnnotationPrefixes` filter keep working without modification.
Closes#8719.
Chart property names containing `/` failed because per-key flat-file storage interpreted the slash as a filesystem path separator, and any move to line-based storage would silently truncate multi-line values. A new `PropertyMap*` helper family in `common` persists each map as a single JSON file so both `/` in keys and `\n` in values round-trip losslessly. Chart overrides move to `chart-overrides.<chart>` and an idempotent `TriggerInstall` migration rewrites any legacy `chart.<chart>.<key>` files into the new layout. The deprecated `scheduler-k3s:set --global chart.*` form is rerouted through the same map storage so it does not silently orphan writes. Closes#8717.
The scheduler-k3s plugin exposes `annotations:set` and `labels:set` but no matching report subcommands, so scripts could not inspect the configured state for idempotent management without reading the property files directly. The new `scheduler-k3s:annotations:report` and `scheduler-k3s:labels:report` surface the configured entries with stdout, JSON, single-flag query, and `--process-type` / `--resource-type` filtering, mirroring the recently added `scheduler-k3s:charts:report`. The literal `GlobalProcessType` value `--global` is rendered as `global` in report keys to avoid leading dashes. The `scheduler-k3s:autoscaling-auth:report` command is updated alongside to loop over every app when no app and no `--global` flag is provided, matching the convention used by `scheduler-k3s:report`.
Helm chart overrides on the scheduler-k3s plugin were previously set
through the generic `scheduler-k3s:set --global chart.<chart>.<property>`
interface, which mixed chart-level configuration into the same command
that manages scheduler properties and offered no focused way to inspect
which overrides were configured. A dedicated `scheduler-k3s:charts:set`
sets and clears chart-specific helm values, and a complementary
`scheduler-k3s:charts:report` surfaces configured overrides per chart
with optional JSON output and single-field flag queries. The legacy form
on `scheduler-k3s:set` continues to work but now emits a deprecation
warning pointing users at the new subcommand.
Every Go-implemented plugin's `:report --format json` now emits keys without the redundant `<plugin>-` head segment, matching the shape bash plugins have always emitted. The CLI flag names and `:set` semantics are unchanged. For backwards compatibility during the 0.38.x patch series, the legacy `<plugin>-<property>` keys are emitted side-by-side with the new keys, and a future major release will drop the legacy keys. `common.ReportSingleApp` is refactored to accept a `ReportSingleAppInput` struct with a `Validate()` method so the input is checked before any work runs, which also catches the latent `"docker options"` reportType bug at the API boundary.
Adds the missing raw/global/computed report keys for ten settable properties across the ps, builder, scheduler, proxy, openresty, nginx, and scheduler-k3s plugins so external tooling can verify drift through `:report --format json`. The scheduler-k3s `token` is masked as `*******` in default stdout output and only unmasked when the report is requested via `--format json` or when the flag is queried explicitly by name; the same explicit-query unmasking rule is extended to the existing traefik `dns-provider-<env_var>` keys.
Three plugins still leaked the built-in default through `<plugin>-global-<property>` after the broader raw/computed split in #8640. External tooling reading `:report --format json` for drift detection could not distinguish "set to default" from "never set" - after a `:set --global` followed by an unset, the JSON still reported the default value. The `<plugin>-global-<property>` keys now hold the raw stored value (empty when nothing has been set) and `<plugin>-computed-<property>` keys hold the effective value with per-app, global, and built-in default fallback. Every global property now has a computed sibling for shape consistency, including those whose default is empty. Internal callers in `scheduler-k3s` that previously read the default-applied global helpers now consume new `getComputed*` helpers, keeping deploy-time behavior unchanged.
Renames the existing `## Internal properties` umbrella section to `## Properties` across all plugin docs and adds a new `### Settable properties` subsection enumerating every property accepted by each plugin's `:set` command. Each row lists scope, default, the actual `:report` flag shapes that surface the value, and a one-line description. The buildpacks plugin is included even though it uses `:set-property`, with a note in the subsection clarifying the legacy command name. Property lists, scopes, and report flags were verified against plugin source and the live dokku binary.
The `<plugin>-global-<property>` keys in `:report` output returned the resolved value with the built-in default substituted in, so external tooling could not distinguish a property that had been set globally to the default from one that had never been set. The bare keys for caddy, haproxy, and traefik global-only properties had the same shape. The `global-<property>` keys now hold the raw stored value and are empty when nothing has been set, and a new `computed-<property>` key holds the effective value used at runtime, falling back through the per-app value (where one exists), the global value, and the built-in default. The bare global-only keys for the three proxy plugins are removed in favor of the raw/computed pair. Closes#8631.
The bare `init-process` and `parallel-schedule-count` keys previously returned the computed value, so external tooling could not tell whether a property had been set on the app or was merely defaulting. Both properties are now also configurable with `--global`, the report exposes `computed-*` and `global-*` keys alongside the bare raw keys, and the deploy path honors the global value before falling back to the linuxserver.io vendor heuristic.
Per-plugin management docs now describe the properties introduced by the env-var-to-property migration in PR #8498, and stale prose and command-output examples that still referenced the old `DOKKU_*` names have been refreshed. The deprecated env vars table moves out of `environment-variables.md` and into the 0.38.0 migration guide, where it functions as a one-time pointer for upgrading users rather than ongoing reference material.
Updates persistent-storage.md to lead with the named storage entry workflow while keeping the legacy colon-form documentation intact, adds a Persistent storage section to the k3s scheduler doc, documents the storage-app-mounts, storage-create, storage-destroy, and storage-status triggers in plugin-triggers, and adds an entry to the 0.38.0 migration guide explaining the install-time migration of legacy mounts and the new DNS-1123 name validation. Bats coverage in tests/unit/storage.bats now exercises storage:create / list-entries / destroy, name validation rejections, multi-entry attachment, the destroy-while-mounted error, and the ensure-directory deprecation warning.
Keda 2.17+ rejects ScaledObjects whose spec.fallback is set unless at least one trigger is not a cpu or memory scaler, so unconditionally emitting fallback broke deploys for apps autoscaled on cpu or memory alone. The chart now skips the fallback block when every configured trigger is cpu or memory and keeps the existing behavior otherwise.
Some helm chart upgrades require side-effects that a plain `helm upgrade` cannot perform, such as deleting deployments whose immutable selectors changed. Charts may now register pre and post upgrade hooks against a target version; applicable hooks fire in ascending semver order, each bracketed around an upgrade to its version, with a final upgrade to the chart's configured version when needed. The new `keda-add-ons-http` 0.12.2 hook deletes the chart-managed deployments before the upgrade so the new selectors take effect cleanly.
Node profiles can be used to specify repetitive options for clusters, making it easier to add specific types of instances quickly.
This PR additionally fixes kubelet-args specification - the same key can be specified multiple times, such as when specifying custom sysctl rules.
If a global vector-sink property on the logs plugin is set, we will use that to configure a log sink (defaulting to console output otherwise). Note that the actual configuration still needs to be a valid DSN representing a Vector log sink.
While this does not support pinning log sinks to specific apps, it opens up integration for those who use a single k3s cluster in shared tenancy.
Dokku ships with default charts and customizes their helm values. In certain cases, it may be useful to further customize those values to better support specific installations. With this change, it is now possible to set custom values by providing the key in the format `chart.$CHART_NAME.$PROPERTY`. For instance, to increase timeouts for the keda http addon:
```shell
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tcpConnectTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.keepAlive 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.responseHeaderTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.idleConnTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tlsHandshakeTimeout 120s
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.expectContinueTimeout 120s
```
To reset a value to its default, simply set the value to an empty string.
```shell
dokku scheduler-k3s:set --global chart.keda-add-ons-http.interceptor.tcpConnectTimeout
```
Once the values are set, users will need to then run `scheduler-k3s:ensure-charts` with either the `--force` flag or specify the `--chart` flag with a list of charts to force apply.
```shell
dokku scheduler-k3s:ensure-charts --force
dokku scheduler-k3s:ensure-charts --chart keda-add-ons-http
```
Note that the existing dokku-provided values cannot be removed, but can be overridden.
