github/dokku
0
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Merge pull request #4809 from dokku/4805-ensure-directory

Browse Source 
Implement storage:ensure-directory command
This commit is contained in:
Jose Diaz-Gonzalez
2021-09-30 01:25:33 -04:00
committed by GitHub
parent fbff08a580 d9eea74f18
commit 5502a61452
7 changed files with 266 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

96
docs/advanced-usage/persistent-storage.md
View File

@@ -5,10 +5,11 @@
The preferred method to mount external containers to a Dokku managed container, is to use the Dokku storage plugin.
```
storage:list <app> # List bind mounts for app's container(s) (host:container)
storage:mount <app> <host-dir:container-dir> # Create a new bind mount
storage:report [<app>] [<flag>] # Displays a checks report for one or more apps
storage:unmount <app> <host-dir:container-dir> # Remove an existing bind mount
storage:ensure-directory [--chown option] <directory> # Creates a persistent storage directory in the recommended storage path
storage:list <app> # List bind mounts for app's container(s) (host:container)
storage:mount <app> <host-dir:container-dir> # Create a new bind mount
storage:report [<app>] [<flag>] # Displays a checks report for one or more apps
storage:unmount <app> <host-dir:container-dir> # Remove an existing bind mount
```
> The storage plugin is compatible with storage mounts created with the docker-options. The storage plugin will only list mounts from the deploy/run phase.
@@ -20,50 +21,77 @@ The storage plugin supports the following mount points:
## Usage
This example demonstrates how to mount the recommended directory to `/storage` inside an application called `node-js-app`. For simplicity, the Dokku project recommends using the directory `/var/lib/dokku/data/storage` directory as the root host path for mounts. This directory is created on Dokku installation.
### Creating storage directories
> New as of 0.25.5
A storage directory can be created with the `storage:ensure-directory` command. This command will create a subdirectory in the recommended `/var/lib/doku/data/storage` path - created during Dokku installation - and prepare it for use with an app.
```shell
# we use a subdirectory inside of the host directory to scope it to just the app
dokku storage:mount node-js-app /var/lib/dokku/data/storage/node-js-app:/storage
dokku storage:ensure-directory lollipop
```
Dokku will then mount the shared contents of `/var/lib/dokku/data/storage/node-js-app` to `/storage` inside the container. Mounts are only available for containers crated via `run` and by the deploy process, and not during the build process. In addition, the host path is never auto-created by either Dokku or Docker, and should be an explicit path, not one relative to the current working directory.
```
-----> Ensuring /var/lib/dokku/data/storage/lollipop exists
Setting directory ownership to 32767:32767
Directory ready for mounting
```
> If the `/storage` path within the container had pre-existing content, the container files will be overrwritten. This may be an issue for users that create assets at build time but then mount a directory at the same place during runtime. Files are not merged.
By default, permissions are set for usage with Herokuish buildpacks. These permissions can be changed via the `--chown` option according to the following table:
Once you have mounted persistent storage, you will also need to restart the application. See the
[process scaling documentation](/docs/processes/process-management.md) for more information.
- `--chown herokuish` (default): Use `32767:32767` as the folder permissions.
- This is used for apps deployed with Buildpacks via Herokuish.
- `--chown heroku`: Use `1000:1000` as the folder permissions.
- This is used for apps deployed with Cloud Native Buildpacks using the `heroku/buildpacks` builder.
- `--chown packeto`: Use `2000:2000` as the folder permissions.
- This is used for apps deployed with Cloud Native Buildpacks using the `cloudfoundry/cnb` or `packeto` builders.
- `--chown false`: Skips the `chown` call.
Users deploying via Dockerfile will want to specify `--chown false` and manually `chown` the created directory if the user and/or group id of the runnning process in the deployed container do not correspond to any of the above options.
> Warning: Failing to set the correct directory ownership may result in issues in persisting files written to the mounted storage directory.
### Mounting storage into apps
Dokku supports mounting both explicit host paths as well as docker volumes via the `storage:mount` command. This takes two arguments, an app name and a `host-path:container-path` or `docker-volume:container-path` combination.
```shell
# mount the directory into your container's /app/storage directory, relative to the container root (/)
# explicit host paths _must_ exist prior to usage.
dokku storage:mount node-js-app /var/lib/dokku/data/storage/node-js-app:/app/storage
# mount the docker volume into your container's /app/storage directory, relative to the container root (/)
# docker volumes _must_ exist prior to usage.
dokku storage:mount node-js-app some-docker-volume:/app/storage
```
In the first example, Dokku will then mount the shared contents of `/var/lib/dokku/data/storage/node-js-app` to `/app/storage` inside the container. The mount point is *not* relative to your app's working directory, and is instead relative to the root (`/`) of the container. Mounts are only available for containers created via `run` and by the deploy process, and not during the build process. In addition, the host path is never auto-created by either Dokku or Docker, and should be an explicit path, not one relative to the current working directory.
> If the `/storage` path within the container had pre-existing content, the container files will be over-written. This may be an issue for users that create assets at build time but then mount a directory at the same place during runtime. Files are not merged.
Once persistent storage is mounted, the app requires a restart. See the [process scaling documentation](/docs/processes/process-management.md) for more information.
```shell
dokku ps:restart app-name
```
A more complete workflow may require making a custom directory for your application and mounting it within your `/app/storage` directory instead. The mount point is *not* relative to your application's working directory, and is instead relative to the root (`/`) of the container.
### Unmounting storage
If an app no longer requires a mounted volume or directory, the `storage:unmount` command can be called. This takes the same arguments as the `storage:mount` command, an app name and a `host-path:container-path` or `docker-volume:container-path` combination.
```shell
# creating storage for the app 'node-js-app'
mkdir -p /var/lib/dokku/data/storage/node-js-app
# unmount the directory from your container's /app/storage directory, relative to the container root (/)
dokku storage:unmount node-js-app /var/lib/dokku/data/storage/node-js-app:/app/storage
# set the directory ownership. Below is an example for herokuish
# but see the `Directory Permissions` section for more details
chown -R 32767:32767 /var/lib/dokku/data/storage/node-js-app
# mount the directory into your container's /app/storage directory, relative to root
dokku storage:mount app-name /var/lib/dokku/data/storage/node-js-app:/app/storage
# unmount the docker volume from your container's /app/storage directory, relative to the container root (/)
dokku storage:unmount node-js-app some-docker-volume:/app/storage
```
You can mount one or more directories as desired by following the above pattern.
Once persistent storage is unmounted, the app requires a restart. See the [process scaling documentation](/docs/processes/process-management.md) for more information.
### Directory Permissions
The host directory should always be owned by the container user and group id. If this is not the case, files may not persist when written to mounted storage.
- Buildpacks via Herokuish: Use `32767:32767` as the file permissions
- For Cloud Native Buildpacks: This will depend on the builder in question, but can be retrieved via the `CNB_USER_ID` and `CNB_GROUP_ID` environment variables on the builder image. Common builders are as follows:
- heroku/buildpacks: `1000:1000`
- cloudfoundry/cnb: `2000:2000`
- packeto: `2000:2000`
- Dockerfile and Docker Image: Use the user and group id which corresponds to the one running the process within the container.
```shell
dokku ps:restart app-name
```
### Displaying storage reports for an app
@@ -113,7 +141,7 @@ dokku storage:report node-js-app --storage-deploy-mounts
### Sharing storage across deploys
Dokku is powered by Docker containers, which recommends in their [best practices](https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#containers-should-be-ephemeral) that containers be treated as ephemeral. In order to manage persistent storage for web applications, like user uploads or large binary assets like images, a directory outside the container should be mounted.
Dokku is powered by Docker containers, which recommends in their [best practices](https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#containers-should-be-ephemeral) that containers be treated as ephemeral. In order to manage persistent storage for web apps, like user uploads or large binary assets like images, a directory outside the container should be mounted.
### Shared storage between containers
@@ -139,11 +167,11 @@ You cannot use mounted volumes during the build phase of a Dockerfile deploy. Th
> Note: **This can cause data loss** if you bind a mount under `/app` in buildpack apps as herokuish will attempt to remove the original app path during the build phase.
## Application User and Persistent Storage file ownership (buildpack apps only)
## App User and Persistent Storage file ownership (buildpack apps only)
> New as of 0.7.1
By default, Dokku will execute your buildpack application processes as the `herokuishuser` user. You may override this by setting the `DOKKU_APP_USER` config variable.
By default, Dokku will execute your buildpack app processes as the `herokuishuser` user. You may override this by setting the `DOKKU_APP_USER` config variable.
> NOTE: this user must exist in your herokuish image.

27
plugins/storage/bin/chown-storage-dir Executable file
View File

@@ -0,0 +1,27 @@
#!/usr/bin/env bash
set -eo pipefail
[[ $DOKKU_TRACE ]] && set -x
main() {
declare desc="chowns a storage directory"
declare DIRECTORY="$1" CHOWN_ID="$2"
if [[ -z "$DIRECTORY" ]]; then
echo " ! Please specify a directory to create" 1>&2
exit 1
fi
if [[ ! "$DIRECTORY" =~ ^[A-Za-z0-9\\_-]+$ ]]; then
echo " ! Directory can only contain the following set of characters: [A-Za-z0-9_-]" 1>&2
exit 1
fi
if [[ "$CHOWN_ID" != "32767" ]] && [[ "$CHOWN_ID" != "1000" ]] && [[ "$CHOWN_ID" != "2000" ]]; then
echo " ! Unsupported chown permissions. Supported values: 32767, 1000, 2000"
exit 1
fi
chown -R "$CHOWN_ID:$CHOWN_ID" "${DOKKU_LIB_ROOT}/data/storage/$DIRECTORY"
}
main "$@"

1
plugins/storage/help-functions
View File

@@ -27,6 +27,7 @@ help_desc
fn-help-content() {
declare desc="return help content"
cat <<help_content
storage:ensure-directory [--chown option] <directory>, Creates a persistent storage directory in the recommended storage path
storage:list <app>, List bind mounts for app's container(s) (host:container)
storage:mount <app> <host-dir:container-dir>, Create a new bind mount
storage:report [<app>] [<flag>], Displays a checks report for one or more apps

35
plugins/storage/install
View File

@@ -5,9 +5,40 @@ set -eo pipefail
trigger-storage-install() {
declare desc="storage install trigger"
declare trigger="install"
local storage_directory="${DOKKU_LIB_ROOT}/data/storage"
mkdir -p "${storage_directory}"
chown dokku:dokku "${storage_directory}"
mkdir -p "${DOKKU_LIB_ROOT}/data/storage"
chown dokku:dokku "${DOKKU_LIB_ROOT}/data/storage"
STORAGE_SUDOERS_FILE="/etc/sudoers.d/dokku-storage"
case "$DOKKU_DISTRO" in
debian)
echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE"
echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE"
;;
ubuntu)
echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE"
echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE"
;;
opensuse)
echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE"
echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE"
;;
arch)
echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE"
echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE"
;;
centos | fedora | rhel)
echo "%dokku ALL=(ALL) NOPASSWD:$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir *" >"$STORAGE_SUDOERS_FILE"
echo "Defaults:dokku !requiretty" >>"$STORAGE_SUDOERS_FILE"
echo "Defaults env_keep += \"DOKKU_LIB_ROOT\"" >>"$STORAGE_SUDOERS_FILE"
;;
esac
}
trigger-storage-install "$@"

56
plugins/storage/internal-functions
View File

@@ -5,6 +5,62 @@ source "$PLUGIN_AVAILABLE_PATH/storage/functions"
set -eo pipefail
[[ $DOKKU_TRACE ]] && set -x
cmd-storage-ensure-directory() {
declare desc="creates a persistent storage directory in the recommended storage path"
declare cmd="storage:ensure-directory"
[[ "$1" == "$cmd" ]] && shift 1
declare DIRECTORY CHOWN_FLAG
CHOWN_FLAG=herokuish
skip=false
for arg in "$@"; do
if [[ "$arg" == "--chown" ]]; then
skip=true
continue
fi
if [[ "$skip" == "true" ]]; then
CHOWN_FLAG="$arg"
skip=false
continue
fi
if [[ -z "$DIRECTORY" ]]; then
DIRECTORY="$arg"
fi
done
if [[ -z "$DIRECTORY" ]]; then
dokku_log_fail "Please specify a directory to create"
fi
if [[ ! "$DIRECTORY" =~ ^[A-Za-z0-9\\_-]+$ ]]; then
dokku_log_fail "Directory can only contain the following set of characters: [A-Za-z0-9_-]"
fi
if [[ "$CHOWN_FLAG" == "herokuish" ]]; then
CHOWN_FLAG="32767"
elif [[ "$CHOWN_FLAG" == "heroku" ]]; then
CHOWN_FLAG="1000"
elif [[ "$CHOWN_FLAG" == "packeto" ]]; then
CHOWN_FLAG="2000"
elif [[ "$CHOWN_FLAG" == "false" ]]; then
CHOWN_FLAG="false"
else
dokku_log_fail "Unsupported chown permissions"
fi
local storage_directory="${DOKKU_LIB_ROOT}/data/storage/$DIRECTORY"
dokku_log_info1 "Ensuring ${storage_directory} exists"
mkdir -p "${storage_directory}"
if [[ "$CHOWN_FLAG" != "false" ]]; then
dokku_log_verbose_quiet "Setting directory ownership to $CHOWN_FLAG:$CHOWN_FLAG"
sudo "$PLUGIN_AVAILABLE_PATH/storage/bin/chown-storage-dir" "$DIRECTORY" "$CHOWN_FLAG"
fi
dokku_log_verbose_quiet "Directory ready for mounting"
}
cmd-storage-report() {
declare desc="displays a storage report for one or more apps"
declare cmd="storage:report"

6
plugins/storage/subcommands/ensure-directory Executable file
View File

@@ -0,0 +1,6 @@
#!/usr/bin/env bash
set -eo pipefail
[[ $DOKKU_TRACE ]] && set -x
source "$PLUGIN_AVAILABLE_PATH/storage/internal-functions"
cmd-storage-ensure-directory "$@"

81
tests/unit/storage.bats
View File

@@ -5,6 +5,7 @@ load test_helper
setup() {
global_setup
create_app
rm -rf "$DOKKU_LIB_ROOT/data/storage/rdmtestapp*"
}
teardown() {
@@ -26,6 +27,86 @@ teardown() {
assert_output "$help_output"
}
@test "(storage) storage:ensure-directory" {
run /bin/bash -c "test -d $DOKKU_LIB_ROOT/data/storage/$TEST_APP"
echo "output: $output"
echo "status: $status"
assert_failure
run /bin/bash -c "dokku storage:ensure-directory @"
echo "output: $output"
echo "status: $status"
assert_failure
run /bin/bash -c "dokku storage:ensure-directory $TEST_APP/"
echo "output: $output"
echo "status: $status"
assert_failure
run /bin/bash -c "dokku storage:ensure-directory $TEST_APP/$TEST_APP"
echo "output: $output"
echo "status: $status"
assert_failure
run /bin/bash -c "dokku storage:ensure-directory $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Setting directory ownership to 1000:1000" 0
assert_output_contains "Setting directory ownership to 2000:2000" 0
assert_output_contains "Setting directory ownership to 32767:32767" 1
run /bin/bash -c "test -d $DOKKU_LIB_ROOT/data/storage/$TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "dokku storage:ensure-directory $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "dokku storage:ensure-directory --chown false $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Setting directory ownership to 1000:1000" 0
assert_output_contains "Setting directory ownership to 2000:2000" 0
assert_output_contains "Setting directory ownership to 32767:32767" 0
run /bin/bash -c "dokku storage:ensure-directory $TEST_APP --chown false"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Setting directory ownership to 1000:1000" 0
assert_output_contains "Setting directory ownership to 2000:2000" 0
assert_output_contains "Setting directory ownership to 32767:32767" 0
run /bin/bash -c "dokku storage:ensure-directory --chown heroku $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Setting directory ownership to 1000:1000" 1
assert_output_contains "Setting directory ownership to 2000:2000" 0
assert_output_contains "Setting directory ownership to 32767:32767" 0
run /bin/bash -c "dokku storage:ensure-directory --chown packeto $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Setting directory ownership to 1000:1000" 0
assert_output_contains "Setting directory ownership to 2000:2000" 1
assert_output_contains "Setting directory ownership to 32767:32767" 0
run /bin/bash -c "dokku storage:ensure-directory --chown herokuish $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Setting directory ownership to 1000:1000" 0
assert_output_contains "Setting directory ownership to 2000:2000" 0
assert_output_contains "Setting directory ownership to 32767:32767" 1
}
@test "(storage) storage:mount, storage:list, storage:umount" {
run /bin/bash -c "dokku storage:mount $TEST_APP /tmp/mount:/mount"
echo "output: $output"