Merge pull request #8538 from dokku/6814-k3s-plugin-add-pv-option
Add scheduler-aware named storage entries
This commit is contained in:
Jose Diaz-Gonzalez
@@ -1,24 +1,37 @@
|
||||
# Persistent Storage
|
||||
|
||||
> [!IMPORTANT]
|
||||
> New as of 0.5.0
|
||||
> New as of 0.5.0. Named storage entries new as of 0.38.0.
|
||||
|
||||
The preferred method to mount external containers to a Dokku managed container, is to use the Dokku storage plugin.
|
||||
The preferred method to attach persistent storage to a Dokku-managed container is the Dokku storage plugin.
|
||||
|
||||
```
|
||||
storage:ensure-directory [--chown option] <directory> # Creates a persistent storage directory in the recommended storage path
|
||||
storage:list <app> [--format text|json] # List bind mounts for app's container(s) (host:container)
|
||||
storage:mount <app> <host-dir:container-dir> # Create a new bind mount
|
||||
storage:report [<app>] [<flag>] # Displays a checks report for one or more apps
|
||||
storage:unmount <app> <host-dir:container-dir> # Remove an existing bind mount
|
||||
storage:create <name> [<path>] [flags] # Register a named storage entry
|
||||
storage:destroy <name> # Remove a named storage entry (must be unmounted from every app first)
|
||||
storage:ensure-directory [--chown option] <directory> # [DEPRECATED] use storage:create instead
|
||||
storage:exec <name> [-- <cmd>...] # Run a command (or shell) in a temporary container that mounts the entry
|
||||
storage:info <name> [--format text|json] # Show details for one storage entry
|
||||
storage:list <app> [--format text|json] # List bind mounts for an app's container(s) (legacy host:container view)
|
||||
storage:list-entries [--scheduler s] [--format text|json] # List registered storage entries
|
||||
storage:mount <app> <name> --container-dir <path> [flags] # Mount a named entry into an app
|
||||
storage:mount <app> <host-dir:container-dir> # [LEGACY] colon-form mount, docker-local only
|
||||
storage:report [<app>] [<flag>] # Display a storage report for one or more apps
|
||||
storage:report --global # Display a cluster-wide entry inventory
|
||||
storage:set <name> [flags] # Update a storage entry in place
|
||||
storage:unmount <app> <name> [--container-dir <path>] # Remove an attachment
|
||||
storage:wait <name> # Block until a k3s entry's PVC is bound
|
||||
```
|
||||
|
||||
> The storage plugin is compatible with storage mounts created with the docker-options. The storage plugin will only list mounts from the deploy/run phase.
|
||||
A storage entry is the source of truth for the underlying volume - a host directory on docker-local, or a PersistentVolumeClaim on k3s. Multiple apps can mount the same entry, and an attachment carries the per-app details (container path, phases, subpath, readonly, process type). Names are globally unique across the install and must be DNS-1123 labels (lowercase letters, digits, dashes) of 45 characters or less so they can be used verbatim as Helm release and PVC names.
|
||||
|
||||
The legacy `storage:mount <app> <host>:<container>` form continues to work on docker-local. On a k3s app it is rejected; create a named entry with `storage:create --scheduler k3s` and mount it instead. Existing colon-form mounts are migrated automatically the first time the new code runs - they show up as `legacy-<hash>` entries in `storage:list-entries`.
|
||||
|
||||
The storage plugin supports the following mount points:
|
||||
|
||||
- explicit paths that exist on the host
|
||||
- docker volumes
|
||||
- explicit paths that exist on the host (docker-local)
|
||||
- docker volumes (docker-local)
|
||||
- PersistentVolumeClaims provisioned via a StorageClass (k3s)
|
||||
- hostPath-backed PVs (k3s)
|
||||
|
||||
## Usage
|
||||
|
||||
|
||||
@@ -22,6 +22,7 @@
|
||||
- The `docker-local` scheduler no longer queues an image for retirement when another running container of the same app still uses it. This fixes the case where a `ps:rebuild` against an image-based deploy (`git:from-image`) produced an identical-SHA image and the `dokku-retire` cron timer would log `Image ... has running containers, skipping rm` on every run. Stuck entries from prior versions are pruned automatically on the next `ps:retire` run.
|
||||
- All `:report` subcommands now accept the `--global` flag, which scopes the report to globally-configured properties. The flag composes with `--format json`, so a JSON report of global properties can be obtained via, for example, `dokku scheduler:report --global --format json`. Previously, combining `--global` with `--format json` was rejected with an "info flag" error, and `--global` on its own was treated as an unknown flag.
|
||||
- The `scheduler-k3s` plugin now manages env config and the dokku-generated image pull Secret as their own helm releases with stable names (`config-{app}` and `pull-secret-{app}`) rather than bundling them into the app helm chart with a per-deploy timestamp suffix (`env-{app}.{ts}` / `ims-{app}.{ts}`). This fixes two bugs: a helm rollback of the app chart no longer deletes Secrets that older ReplicaSets still reference, and the Deployment's `imagePullSecrets` list no longer accumulates references to nonexistent Secrets across deploys. The next deploy of an app switches the Deployment's `envFrom` and `imagePullSecrets` references to the stable names and prunes any leaked entries; existing live Deployments do not need to be patched manually. App rename now also uninstalls the old `tls-{app}`, `config-{app}`, and `pull-secret-{app}` releases under the previous app name; the new name's releases are recreated on the next deploy or certs sync.
|
||||
- The storage plugin now treats persistent volumes as named, scheduler-aware first-class resources via `storage:create`, `storage:mount`, `storage:set`, and `storage:destroy`. The legacy `storage:mount <app> <host>:<container>` colon form continues to work on docker-local apps but is deprecated; on k3s apps it is rejected. Existing colon-form mounts are migrated automatically the first time the new storage plugin runs (during the install trigger) - they appear as `legacy-<hash>` entries in `storage:list-entries`. The migration is idempotent and tied to a per-app flag file at `$DOKKU_LIB_ROOT/config/storage/.migrated/<app>`; deleting that file forces a re-scan on the next install. The `storage:ensure-directory` command keeps working but now emits a deprecation warning - prefer `storage:create <name> [<path>]` (the path defaults to the same `$DOKKU_LIB_ROOT/data/storage/<name>` location). Storage entry names must now be DNS-1123 labels of 45 characters or less so they can be used verbatim as Helm release and Kubernetes resource names; underscores and uppercase characters that the older `ensure-directory` validator accepted are rejected for new names. The migration synthesizer always uses lowercase hex hashes so existing data is never locked out.
|
||||
|
||||
### TLS handshake behavior change
|
||||
|
||||
|
||||
@@ -685,6 +685,27 @@ Alternatively, a comma separated list of chart names can be specified to only fo
|
||||
dokku scheduler-k3s:ensure-charts --charts cert-manager
|
||||
```
|
||||
|
||||
### Persistent storage
|
||||
|
||||
K3s apps mount persistent volumes through the Dokku storage plugin. A `storage:create --scheduler k3s` call provisions a PersistentVolumeClaim through a small per-entry helm release; `storage:mount <app> <name> --container-dir <path>` then attaches it to an app. Because storage entries are global, the same PVC can be mounted into multiple apps (for `ReadWriteMany` workloads) or into the same app at multiple subpaths.
|
||||
|
||||
```shell
|
||||
# dynamically-provisioned PVC backed by Longhorn
|
||||
dokku storage:create demo-data \
|
||||
--scheduler k3s \
|
||||
--size 2Gi \
|
||||
--access-mode ReadWriteOnce \
|
||||
--storage-class-name longhorn
|
||||
|
||||
dokku storage:mount demo demo-data --container-dir /data
|
||||
dokku storage:wait demo-data
|
||||
git push dokku master
|
||||
```
|
||||
|
||||
For a hostPath-backed PV (no StorageClass), pass `<path>` as the second positional argument and omit `--storage-class-name`. The plugin renders both the PV and the PVC into the entry's helm release. The `--reclaim-policy` flag (`Retain` or `Delete`) controls whether the underlying PV survives `storage:destroy`. Annotations and labels on `storage:create` / `storage:set` propagate to both the PVC and the PV so backup tools (Velero, Longhorn snapshots) can find them.
|
||||
|
||||
The legacy `storage:mount <app> <host>:<container>` colon form is rejected on k3s apps; create a named entry instead. See [Persistent Storage](/docs/advanced-usage/persistent-storage.md) for the full command reference.
|
||||
|
||||
### Chart upgrade callbacks
|
||||
|
||||
Some chart version transitions require side-effects that a plain `helm upgrade` cannot perform. For example, the upgrade of `keda-add-ons-http` to `0.12.2` introduces breaking changes to deployment selectors, which are immutable in Kubernetes; the chart-managed deployments must be deleted before the upgrade can proceed.
|
||||
|
||||
@@ -2904,21 +2904,52 @@ DOKKU_SCHEDULER="$1"; APP="$2"; REMOVE_CONTAINERS="$3";
|
||||
# TODO
|
||||
```
|
||||
|
||||
### `storage-list`
|
||||
### `storage-list` (deprecated)
|
||||
|
||||
- Description: Returns a list of storage mounts
|
||||
- Invoked by: `dokku storage:list` and `dokku deploy`
|
||||
> [!WARNING]
|
||||
> Deprecated as of 0.38.0. The trigger handler still functions for back-compat
|
||||
> with external plugins but emits a deprecation warning on every invocation.
|
||||
> In-process callers should use the `storage-app-mounts` trigger instead, or
|
||||
> the `storage` Go package directly.
|
||||
|
||||
- Description: Returns a list of storage mounts for an app
|
||||
- Invoked by: external plugins (the core `storage:list` subcommand no longer uses it)
|
||||
- Arguments: `$APP $PHASE $FORMAT`
|
||||
- Example:
|
||||
|
||||
```shell
|
||||
#!/usr/bin/env bash
|
||||
### `storage-app-mounts`
|
||||
|
||||
set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
|
||||
APP="$1"
|
||||
- Description: Returns the (entry, attachment) pairs an app has for the given phase, in JSON. Schedulers consume this to render volume mounts.
|
||||
- Invoked by: scheduler plugins during deploy
|
||||
- Arguments: `$APP $PHASE`
|
||||
- Output: a JSON array of objects shaped `{"entry": Entry, "attachment": Attachment}`
|
||||
|
||||
# TODO
|
||||
```
|
||||
### `storage-create`
|
||||
|
||||
- Description: Provisions or updates the underlying volume for a k3s storage entry. Called when `storage:create` or `storage:set` runs against a k3s entry. Receives the JSON-encoded entry on stdin.
|
||||
- Invoked by: `dokku storage:create`, `dokku storage:set`
|
||||
- Arguments: `$ENTRY_NAME`
|
||||
- Stdin: JSON-encoded `Entry` payload
|
||||
|
||||
### `storage-destroy`
|
||||
|
||||
- Description: Releases the underlying volume for a k3s storage entry. Called when `storage:destroy` runs against a k3s entry. Receives the JSON-encoded entry on stdin.
|
||||
- Invoked by: `dokku storage:destroy`
|
||||
- Arguments: `$ENTRY_NAME`
|
||||
- Stdin: JSON-encoded `Entry` payload
|
||||
|
||||
### `storage-status`
|
||||
|
||||
- Description: Returns the current status of a storage entry's underlying volume. Used by `storage:wait` to poll the cluster. The handler should print one line: `Bound`, `Pending`, or `Lost` for k3s entries.
|
||||
- Invoked by: `dokku storage:wait`
|
||||
- Arguments: `$ENTRY_NAME`
|
||||
- Stdin: JSON-encoded `Entry` payload
|
||||
|
||||
### `scheduler-storage-exec`
|
||||
|
||||
- Description: Runs an interactive or non-interactive command against a storage entry. The storage plugin fires this with `<scheduler>` as the first arg; each scheduler plugin's handler matches against its own scheduler name and either handles or no-ops, mirroring `scheduler-deploy` / `scheduler-app-status`. Plugn forwards stdin/stdout/stderr to the handler subprocess so an interactive shell streams cleanly. The handler exits with the underlying tool's status code so `dokku storage:exec` propagates exit codes verbatim.
|
||||
- Invoked by: `dokku storage:exec`
|
||||
- Arguments: `$SCHEDULER $ENTRY_NAME $IMAGE [-- $cmd...]`
|
||||
- Flags: `--interactive` (stdin is open), `--tty` (stdin is a terminal), `--as-user <uid>` (override `entry.Chown`).
|
||||
|
||||
### `traefik-template-source`
|
||||
|
||||
|
||||
Reference in New Issue
Block a user