Merge pull request #4809 from dokku/4805-ensure-directory

Implement storage:ensure-directory command
This commit is contained in:
Jose Diaz-Gonzalez authored and GitHub committed 2021-09-30 01:25:33 -04:00
commit 5502a61452
7 files changed
+266 -36

No files matched your search

+62 -34
View File
@@ -5,10 +5,11 @@
The preferred method to mount external containers to a Dokku managed container, is to use the Dokku storage plugin. The preferred method to mount external containers to a Dokku managed container, is to use the Dokku storage plugin.
``` ```
storage:list <app> # List bind mounts for app's container(s) (host:container) storage:ensure-directory [--chown option] <directory> # Creates a persistent storage directory in the recommended storage path
storage:mount <app> <host-dir:container-dir> # Create a new bind mount storage:list <app> # List bind mounts for app's container(s) (host:container)
storage:report [<app>] [<flag>] # Displays a checks report for one or more apps storage:mount <app> <host-dir:container-dir> # Create a new bind mount
storage:unmount <app> <host-dir:container-dir> # Remove an existing bind mount storage:report [<app>] [<flag>] # Displays a checks report for one or more apps
storage:unmount <app> <host-dir:container-dir> # Remove an existing bind mount
``` ```
> The storage plugin is compatible with storage mounts created with the docker-options. The storage plugin will only list mounts from the deploy/run phase. > The storage plugin is compatible with storage mounts created with the docker-options. The storage plugin will only list mounts from the deploy/run phase.
@@ -20,50 +21,77 @@ The storage plugin supports the following mount points:
## Usage ## Usage
This example demonstrates how to mount the recommended directory to `/storage` inside an application called `node-js-app`. For simplicity, the Dokku project recommends using the directory `/var/lib/dokku/data/storage` directory as the root host path for mounts. This directory is created on Dokku installation. ### Creating storage directories
> New as of 0.25.5
A storage directory can be created with the `storage:ensure-directory` command. This command will create a subdirectory in the recommended `/var/lib/doku/data/storage` path - created during Dokku installation - and prepare it for use with an app.
```shell ```shell
# we use a subdirectory inside of the host directory to scope it to just the app dokku storage:ensure-directory lollipop
dokku storage:mount node-js-app /var/lib/dokku/data/storage/node-js-app:/storage
``` ```
Dokku will then mount the shared contents of `/var/lib/dokku/data/storage/node-js-app` to `/storage` inside the container. Mounts are only available for containers crated via `run` and by the deploy process, and not during the build process. In addition, the host path is never auto-created by either Dokku or Docker, and should be an explicit path, not one relative to the current working directory. ```
-----> Ensuring /var/lib/dokku/data/storage/lollipop exists
Setting directory ownership to 32767:32767
Directory ready for mounting
```
> If the `/storage` path within the container had pre-existing content, the container files will be overrwritten. This may be an issue for users that create assets at build time but then mount a directory at the same place during runtime. Files are not merged. By default, permissions are set for usage with Herokuish buildpacks. These permissions can be changed via the `--chown` option according to the following table:
Once you have mounted persistent storage, you will also need to restart the application. See the - `--chown herokuish` (default): Use `32767:32767` as the folder permissions.
[process scaling documentation](/docs/processes/process-management.md) for more information. - This is used for apps deployed with Buildpacks via Herokuish.
- `--chown heroku`: Use `1000:1000` as the folder permissions.
- This is used for apps deployed with Cloud Native Buildpacks using the `heroku/buildpacks` builder.
- `--chown packeto`: Use `2000:2000` as the folder permissions.
- This is used for apps deployed with Cloud Native Buildpacks using the `cloudfoundry/cnb` or `packeto` builders.
- `--chown false`: Skips the `chown` call.
Users deploying via Dockerfile will want to specify `--chown false` and manually `chown` the created directory if the user and/or group id of the runnning process in the deployed container do not correspond to any of the above options.
> Warning: Failing to set the correct directory ownership may result in issues in persisting files written to the mounted storage directory.
### Mounting storage into apps
Dokku supports mounting both explicit host paths as well as docker volumes via the `storage:mount` command. This takes two arguments, an app name and a `host-path:container-path` or `docker-volume:container-path` combination.
```shell
# mount the directory into your container's /app/storage directory, relative to the container root (/)
# explicit host paths _must_ exist prior to usage.
dokku storage:mount node-js-app /var/lib/dokku/data/storage/node-js-app:/app/storage
# mount the docker volume into your container's /app/storage directory, relative to the container root (/)
# docker volumes _must_ exist prior to usage.
dokku storage:mount node-js-app some-docker-volume:/app/storage
```
In the first example, Dokku will then mount the shared contents of `/var/lib/dokku/data/storage/node-js-app` to `/app/storage` inside the container. The mount point is *not* relative to your app's working directory, and is instead relative to the root (`/`) of the container. Mounts are only available for containers created via `run` and by the deploy process, and not during the build process. In addition, the host path is never auto-created by either Dokku or Docker, and should be an explicit path, not one relative to the current working directory.
> If the `/storage` path within the container had pre-existing content, the container files will be over-written. This may be an issue for users that create assets at build time but then mount a directory at the same place during runtime. Files are not merged.
Once persistent storage is mounted, the app requires a restart. See the [process scaling documentation](/docs/processes/process-management.md) for more information.
```shell ```shell
dokku ps:restart app-name dokku ps:restart app-name
``` ```
A more complete workflow may require making a custom directory for your application and mounting it within your `/app/storage` directory instead. The mount point is *not* relative to your application's working directory, and is instead relative to the root (`/`) of the container. ### Unmounting storage
If an app no longer requires a mounted volume or directory, the `storage:unmount` command can be called. This takes the same arguments as the `storage:mount` command, an app name and a `host-path:container-path` or `docker-volume:container-path` combination.
```shell ```shell
# creating storage for the app 'node-js-app' # unmount the directory from your container's /app/storage directory, relative to the container root (/)
mkdir -p /var/lib/dokku/data/storage/node-js-app dokku storage:unmount node-js-app /var/lib/dokku/data/storage/node-js-app:/app/storage
# set the directory ownership. Below is an example for herokuish # unmount the docker volume from your container's /app/storage directory, relative to the container root (/)
# but see the `Directory Permissions` section for more details dokku storage:unmount node-js-app some-docker-volume:/app/storage
chown -R 32767:32767 /var/lib/dokku/data/storage/node-js-app
# mount the directory into your container's /app/storage directory, relative to root
dokku storage:mount app-name /var/lib/dokku/data/storage/node-js-app:/app/storage
``` ```
You can mount one or more directories as desired by following the above pattern. Once persistent storage is unmounted, the app requires a restart. See the [process scaling documentation](/docs/processes/process-management.md) for more information.
### Directory Permissions ```shell
dokku ps:restart app-name
The host directory should always be owned by the container user and group id. If this is not the case, files may not persist when written to mounted storage. ```
- Buildpacks via Herokuish: Use `32767:32767` as the file permissions
- For Cloud Native Buildpacks: This will depend on the builder in question, but can be retrieved via the `CNB_USER_ID` and `CNB_GROUP_ID` environment variables on the builder image. Common builders are as follows:
- heroku/buildpacks: `1000:1000`
- cloudfoundry/cnb: `2000:2000`
- packeto: `2000:2000`
- Dockerfile and Docker Image: Use the user and group id which corresponds to the one running the process within the container.
### Displaying storage reports for an app ### Displaying storage reports for an app
@@ -113,7 +141,7 @@ dokku storage:report node-js-app --storage-deploy-mounts
### Sharing storage across deploys ### Sharing storage across deploys
Dokku is powered by Docker containers, which recommends in their [best practices](https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#containers-should-be-ephemeral) that containers be treated as ephemeral. In order to manage persistent storage for web applications, like user uploads or large binary assets like images, a directory outside the container should be mounted. Dokku is powered by Docker containers, which recommends in their [best practices](https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#containers-should-be-ephemeral) that containers be treated as ephemeral. In order to manage persistent storage for web apps, like user uploads or large binary assets like images, a directory outside the container should be mounted.
### Shared storage between containers ### Shared storage between containers
@@ -139,11 +167,11 @@ You cannot use mounted volumes during the build phase of a Dockerfile deploy. Th
> Note: **This can cause data loss** if you bind a mount under `/app` in buildpack apps as herokuish will attempt to remove the original app path during the build phase. > Note: **This can cause data loss** if you bind a mount under `/app` in buildpack apps as herokuish will attempt to remove the original app path during the build phase.
## Application User and Persistent Storage file ownership (buildpack apps only) ## App User and Persistent Storage file ownership (buildpack apps only)
> New as of 0.7.1 > New as of 0.7.1
By default, Dokku will execute your buildpack application processes as the `herokuishuser` user. You may override this by setting the `DOKKU_APP_USER` config variable. By default, Dokku will execute your buildpack app processes as the `herokuishuser` user. You may override this by setting the `DOKKU_APP_USER` config variable.
> NOTE: this user must exist in your herokuish image. > NOTE: this user must exist in your herokuish image.
+27
View File
@@ -0,0 +1,27 @@
#!/usr/bin/env bash
set -eo pipefail
[[ $DOKKU_TRACE ]] && set -x
main() {
declare desc="chowns a storage directory"
declare DIRECTORY="$1" CHOWN_ID="$2"
if [[ -z "$DIRECTORY" ]]; then
echo " ! Please specify a directory to create" 1>&2
exit 1
fi
if [[ ! "$DIRECTORY" =~ ^[A-Za-z0-9\\_-]+$ ]]; then
echo " ! Directory can only contain the following set of characters: [A-Za-z0-9_-]" 1>&2
exit 1
fi
if [[ "$CHOWN_ID" != "32767" ]] && [[ "$CHOWN_ID" != "1000" ]] && [[ "$CHOWN_ID" != "2000" ]]; then
echo " ! Unsupported chown permissions. Supported values: 32767, 1000, 2000"
exit 1
fi
chown -R "$CHOWN_ID:$CHOWN_ID" "${DOKKU_LIB_ROOT}/data/storage/$DIRECTORY"
}
main "$@"
+1
View File
@@ -27,6 +27,7 @@ help_desc
fn-help-content() { fn-help-content() {
declare desc="return help content" declare desc="return help content"
cat <<help_content cat <<help_content
storage:ensure-directory [--chown option] <directory>, Creates a persistent storage directory in the recommended storage path
storage:list <app>, List bind mounts for app's container(s) (host:container) storage:list <app>, List bind mounts for app's container(s) (host:container)
storage:mount <app> <host-dir:container-dir>, Create a new bind mount storage:mount <app> <host-dir:container-dir>, Create a new bind mount
storage:report [<app>] [<flag>], Displays a checks report for one or more apps storage:report [<app>] [<flag>], Displays a checks report for one or more apps
+33 -2
View File
@@ -5,9 +5,40 @@ set -eo pipefail
trigger-storage-install() { trigger-storage-install() {
declare desc="storage install trigger" declare desc="storage install trigger"
declare trigger="install" declare trigger="install"
local storage_directory="${DOKKU_LIB_ROOT}/data/storage"
mkdir -p "${storage_directory}"
chown dokku:dokku "${storage_directory}"
mkdir -p "${DOKKU_LIB_ROOT}/data/storage" STORAGE_SUDOERS_FILE="/etc/sudoers.d/dokku-storage"
chown dokku:dokku "${DOKKU_LIB_ROOT}/data/storage"
case "$DOKKU_DISTRO" in
debian)
echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE"
echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE"
;;
ubuntu)
echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE"
echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE"
;;
opensuse)
echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE"
echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE"
;;
arch)
echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE"
echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE"
;;
centos | fedora | rhel)
echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE"
echo "Defaults:dokku !requiretty" >>"$STORAGE_SUDOERS_FILE"
echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE"
;;
esac
} }
trigger-storage-install "$@" trigger-storage-install "$@"
+56
View File
@@ -5,6 +5,62 @@ source "$PLUGIN_AVAILABLE_PATH/storage/functions"
set -eo pipefail set -eo pipefail
[[ $DOKKU_TRACE ]] && set -x [[ $DOKKU_TRACE ]] && set -x
cmd-storage-ensure-directory() {
declare desc="creates a persistent storage directory in the recommended storage path"
declare cmd="storage:ensure-directory"
[[ "$1" == "$cmd" ]] && shift 1
declare DIRECTORY CHOWN_FLAG
CHOWN_FLAG=herokuish
skip=false
for arg in "$@"; do
if [[ "$arg" == "--chown" ]]; then
skip=true
continue
fi
if [[ "$skip" == "true" ]]; then
CHOWN_FLAG="$arg"
skip=false
continue
fi
if [[ -z "$DIRECTORY" ]]; then
DIRECTORY="$arg"
fi
done
if [[ -z "$DIRECTORY" ]]; then
dokku_log_fail "Please specify a directory to create"
fi
if [[ ! "$DIRECTORY" =~ ^[A-Za-z0-9\\_-]+$ ]]; then
dokku_log_fail "Directory can only contain the following set of characters: [A-Za-z0-9_-]"
fi
if [[ "$CHOWN_FLAG" == "herokuish" ]]; then
CHOWN_FLAG="32767"
elif [[ "$CHOWN_FLAG" == "heroku" ]]; then
CHOWN_FLAG="1000"
elif [[ "$CHOWN_FLAG" == "packeto" ]]; then
CHOWN_FLAG="2000"
elif [[ "$CHOWN_FLAG" == "false" ]]; then
CHOWN_FLAG="false"
else
dokku_log_fail "Unsupported chown permissions"
fi
local storage_directory="${DOKKU_LIB_ROOT}/data/storage/$DIRECTORY"
dokku_log_info1 "Ensuring ${storage_directory} exists"
mkdir -p "${storage_directory}"
if [[ "$CHOWN_FLAG" != "false" ]]; then
dokku_log_verbose_quiet "Setting directory ownership to $CHOWN_FLAG:$CHOWN_FLAG"
sudo "$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir" "$DIRECTORY" "$CHOWN_FLAG"
fi
dokku_log_verbose_quiet "Directory ready for mounting"
}
cmd-storage-report() { cmd-storage-report() {
declare desc="displays a storage report for one or more apps" declare desc="displays a storage report for one or more apps"
declare cmd="storage:report" declare cmd="storage:report"
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
set -eo pipefail
[[ $DOKKU_TRACE ]] && set -x
source "$PLUGIN_AVAILABLE_PATH/storage/internal-functions"
cmd-storage-ensure-directory "$@"
+81
View File
@@ -5,6 +5,7 @@ load test_helper
setup() { setup() {
global_setup global_setup
create_app create_app
rm -rf "$DOKKU_LIB_ROOT/data/storage/rdmtestapp*"
} }
teardown() { teardown() {
@@ -26,6 +27,86 @@ teardown() {
assert_output "$help_output" assert_output "$help_output"
} }
@test "(storage) storage:ensure-directory" {
run /bin/bash -c "test -d $DOKKU_LIB_ROOT/data/storage/$TEST_APP"
echo "output: $output"
echo "status: $status"
assert_failure
run /bin/bash -c "dokku storage:ensure-directory @"
echo "output: $output"
echo "status: $status"
assert_failure
run /bin/bash -c "dokku storage:ensure-directory $TEST_APP/"
echo "output: $output"
echo "status: $status"
assert_failure
run /bin/bash -c "dokku storage:ensure-directory $TEST_APP/$TEST_APP"
echo "output: $output"
echo "status: $status"
assert_failure
run /bin/bash -c "dokku storage:ensure-directory $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Setting directory ownership to 1000:1000" 0
assert_output_contains "Setting directory ownership to 2000:2000" 0
assert_output_contains "Setting directory ownership to 32767:32767" 1
run /bin/bash -c "test -d $DOKKU_LIB_ROOT/data/storage/$TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "dokku storage:ensure-directory $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "dokku storage:ensure-directory --chown false $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Setting directory ownership to 1000:1000" 0
assert_output_contains "Setting directory ownership to 2000:2000" 0
assert_output_contains "Setting directory ownership to 32767:32767" 0
run /bin/bash -c "dokku storage:ensure-directory $TEST_APP --chown false"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Setting directory ownership to 1000:1000" 0
assert_output_contains "Setting directory ownership to 2000:2000" 0
assert_output_contains "Setting directory ownership to 32767:32767" 0
run /bin/bash -c "dokku storage:ensure-directory --chown heroku $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Setting directory ownership to 1000:1000" 1
assert_output_contains "Setting directory ownership to 2000:2000" 0
assert_output_contains "Setting directory ownership to 32767:32767" 0
run /bin/bash -c "dokku storage:ensure-directory --chown packeto $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Setting directory ownership to 1000:1000" 0
assert_output_contains "Setting directory ownership to 2000:2000" 1
assert_output_contains "Setting directory ownership to 32767:32767" 0
run /bin/bash -c "dokku storage:ensure-directory --chown herokuish $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Setting directory ownership to 1000:1000" 0
assert_output_contains "Setting directory ownership to 2000:2000" 0
assert_output_contains "Setting directory ownership to 32767:32767" 1
}
@test "(storage) storage:mount, storage:list, storage:umount" { @test "(storage) storage:mount, storage:list, storage:umount" {
run /bin/bash -c "dokku storage:mount $TEST_APP /tmp/mount:/mount" run /bin/bash -c "dokku storage:mount $TEST_APP /tmp/mount:/mount"
echo "output: $output" echo "output: $output"