fix: keep track of the source image when building an app via git:from-image

This change fixes issues with ps:rebuild not working as expected - essentially ignoring files in the repo - as well as adds tests.

Closes #5518
This commit is contained in:
Jose Diaz-Gonzalez committed 2022-12-18 17:57:24 -05:00
1 parent 654939afa7
commit c3c6e2174a
9 files changed
+171 -10

No files matched your search

+3 -1
View File
@@ -5,13 +5,15 @@
- The output of `run:detached` now uses the container name - eg. `node-js-app.run.1` - vs the container id. - The output of `run:detached` now uses the container name - eg. `node-js-app.run.1` - vs the container id.
- The ID of `cron` tasks is now base36-encoded instead of base64-encoded. - The ID of `cron` tasks is now base36-encoded instead of base64-encoded.
- The `nginx.conf.sigil` is now extracted when source code is extracted for a build and not from the built image. Users can specify alternative paths via the `nginx-conf-sigil-path` property of the `nginx` plugin. See the [nginx documentation](/docs/networking/proxies/nginx.md#customizing-the-nginx-configuration) for more information on how to configure the `nginx.conf.sigil` path for your application. - The `nginx.conf.sigil` is now extracted when source code is extracted for a build and not from the built image. Users can specify alternative paths via the `nginx-conf-sigil-path` property of the `nginx` plugin. See the [nginx documentation](/docs/networking/proxies/nginx.md#customizing-the-nginx-configuration) for more information on how to configure the `nginx.conf.sigil` path for your application.
- For deploys via `git:from-image`, the `nginx.conf.sigil` file will be extracted from the source image, respecting the value of `nginx-conf-sigil-path`.
- The `Procfile` is now extracted when source code is extracted for a build and not from the built image. Users can specify alternative paths via the `procfile-path` property of the `ps` plugin. See the [process management documentation](/docs/processes/process-management.md#changing-the-procfile-location) for more information on how to configure the `Procfile` path for your application.
- For deploys via `git:from-image`, the `Procfile` file will be extracted from the source image, respecting the value of `procfile-path`.
- The existing `pre-restore` hook has been renamed to `scheduler-pre-restore`. There is a new `pre-restore` hook that is triggered within the `ps:restore` command prior to restoring any apps. - The existing `pre-restore` hook has been renamed to `scheduler-pre-restore`. There is a new `pre-restore` hook that is triggered within the `ps:restore` command prior to restoring any apps.
- Nginx init commands are now performed via systemctl on Ubuntu systems when `/usr/bin/systemctl` is available. - Nginx init commands are now performed via systemctl on Ubuntu systems when `/usr/bin/systemctl` is available.
## Removals ## Removals
- The `DOKKU_WAIT_TO_RETIRE` environment variable has been migrated to a `checks` property named `wait-to-retire` and will be ignored if set as an environment variable. - The `DOKKU_WAIT_TO_RETIRE` environment variable has been migrated to a `checks` property named `wait-to-retire` and will be ignored if set as an environment variable.
- The `Procfile` is now extracted when source code is extracted for a build and not from the built image. Users can specify alternative paths via the `procfile-path` property of the `ps` plugin. See the [process management documentation](/docs/processes/process-management.md#changing-the-procfile-location) for more information on how to configure the `Procfile` path for your application.
- The `domains-setup` trigger has been removed. Initial app domains will now be automatically setup during app creation. - The `domains-setup` trigger has been removed. Initial app domains will now be automatically setup during app creation.
- The `URLS` file containing generated urls for an app is no longer generated or referenced. Users should retrieve app urls via the new `domains-urls` plugin trigger. - The `URLS` file containing generated urls for an app is no longer generated or referenced. Users should retrieve app urls via the new `domains-urls` plugin trigger.
- The common function `get_app_urls` has been removed. Users should retrieve app urls via the new `domains-urls` plugin trigger. - The common function `get_app_urls` has been removed. Users should retrieve app urls via the new `domains-urls` plugin trigger.
+12
View File
@@ -159,6 +159,18 @@ dokku git:from-image node-js-app dokku/node-js-getting-started:latest "Camila" "
If the image is a private image that requires a docker login to access, the `registry:login` command should be used to log into the registry. See the [registry documentation](/docs/advanced-usage/registry-management.md#logging-into-a-registry) for more details on this process. If the image is a private image that requires a docker login to access, the `registry:login` command should be used to log into the registry. See the [registry documentation](/docs/advanced-usage/registry-management.md#logging-into-a-registry) for more details on this process.
Building an app from an image will result in the following files being extracted from the source image (with all custom paths specified for each file being respected):
- nginx.conf.sigil
- Procfile
In the case where the repository is later modified to manually add any of the above files and deployed via `git push`, the files will still be extracted from the initial source image. To avoid this, please clear the `source-image` git property. It will be set back to the original source image on any subsequent `git:from-image` calls.
```shell
# sets an empty value
dokku git:set node-js-app source-image
```
Finally, certain images may require a custom build context in order for `ONBUILD ADD` and `ONBUILD COPY` statements to succeed. A custom build context can be specified via the `--build-dir` flag. All files in the specified `build-dir` will be copied into the repository for use within the `docker build` process. The build context _must_ be specified on each deploy, and is not otherwise persisted between builds. Finally, certain images may require a custom build context in order for `ONBUILD ADD` and `ONBUILD COPY` statements to succeed. A custom build context can be specified via the `--build-dir` flag. All files in the specified `build-dir` will be copied into the repository for use within the `docker build` process. The build context _must_ be specified on each deploy, and is not otherwise persisted between builds.
```shell ```shell
+2 -1
View File
@@ -1,5 +1,6 @@
#!/usr/bin/env bash #!/usr/bin/env bash
source "$PLUGIN_CORE_AVAILABLE_PATH/common/functions" source "$PLUGIN_CORE_AVAILABLE_PATH/common/functions"
source "$PLUGIN_CORE_AVAILABLE_PATH/common/property-functions"
set -eo pipefail set -eo pipefail
[[ $DOKKU_TRACE ]] && set -x [[ $DOKKU_TRACE ]] && set -x
@@ -33,7 +34,7 @@ trigger-git-git-from-image() {
dokku_log_info1 "Image exists on host, skipping pull" dokku_log_info1 "Image exists on host, skipping pull"
fi fi
export DOKKU_BUILDING_FROM_IMAGE="$DOCKER_IMAGE" fn-plugin-property-write "git" "$APP" "source-image" "$DOCKER_IMAGE"
plugn trigger git-from-directory "$APP" "$TMP_WORK_DIR" "$USER_NAME" "$USER_EMAIL" plugn trigger git-from-directory "$APP" "$TMP_WORK_DIR" "$USER_NAME" "$USER_EMAIL"
} }
+8
View File
@@ -256,6 +256,7 @@ cmd-git-report-single() {
"--git-keep-git-dir: $(fn-plugin-property-get "git" "$APP" "keep-git-dir" "false")" "--git-keep-git-dir: $(fn-plugin-property-get "git" "$APP" "keep-git-dir" "false")"
"--git-rev-env-var: $(fn-plugin-property-get "git" "$APP" "rev-env-var" "GIT_REV")" "--git-rev-env-var: $(fn-plugin-property-get "git" "$APP" "rev-env-var" "GIT_REV")"
"--git-sha: $(fn-git-cmd "$APP_ROOT" rev-parse --short HEAD 2>/dev/null || false)" "--git-sha: $(fn-git-cmd "$APP_ROOT" rev-parse --short HEAD 2>/dev/null || false)"
"--git-source-image: $(fn-git-source-image "$APP")"
"--git-last-updated-at: $(fn-git-last-updated-at "$APP")" "--git-last-updated-at: $(fn-git-last-updated-at "$APP")"
) )
@@ -544,6 +545,13 @@ fn-git-setup-build-dir-submodules() {
fi fi
} }
fn-git-source-image() {
declare desc="retrieve the source-image for a given application"
declare APP="$1"
fn-plugin-property-get "git" "$APP" "source-image" ""
}
fn-git-use-worktree() { fn-git-use-worktree() {
declare desc="detects whether to use git worktree" declare desc="detects whether to use git worktree"
local GIT_VERSION MAJOR_VERSION MINOR_VERSION local GIT_VERSION MAJOR_VERSION MINOR_VERSION
+2 -2
View File
@@ -9,13 +9,13 @@ cmd-git-set() {
declare cmd="git:set" declare cmd="git:set"
[[ "$1" == "$cmd" ]] && shift 1 [[ "$1" == "$cmd" ]] && shift 1
declare APP="$1" KEY="$2" VALUE="$3" declare APP="$1" KEY="$2" VALUE="$3"
local VALID_KEYS=("deploy-branch" "keep-git-dir" "rev-env-var") local VALID_KEYS=("deploy-branch" "keep-git-dir" "rev-env-var" "source-image")
[[ "$APP" == "--global" ]] || verify_app_name "$APP" [[ "$APP" == "--global" ]] || verify_app_name "$APP"
[[ -z "$KEY" ]] && dokku_log_fail "No key specified" [[ -z "$KEY" ]] && dokku_log_fail "No key specified"
if ! fn-in-array "$KEY" "${VALID_KEYS[@]}"; then if ! fn-in-array "$KEY" "${VALID_KEYS[@]}"; then
dokku_log_fail "Invalid key specified, valid keys include: deploy-branch, keep-git-dir, rev-env-var" dokku_log_fail "Invalid key specified, valid keys include: deploy-branch, keep-git-dir, rev-env-var, source-image"
fi fi
if [[ -n "$VALUE" ]]; then if [[ -n "$VALUE" ]]; then
+4 -2
View File
@@ -41,9 +41,11 @@ trigger-nginx-vhosts-core-post-extract() {
declare trigger="post-extract" declare trigger="post-extract"
declare APP="$1" SOURCECODE_WORK_DIR="$2" declare APP="$1" SOURCECODE_WORK_DIR="$2"
local CONF_SIGIL_PATH="$(fn-nginx-computed-nginx-conf-sigil-path "$APP")" local CONF_SIGIL_PATH="$(fn-nginx-computed-nginx-conf-sigil-path "$APP")"
local app_source_image
if [[ -n "$DOKKU_BUILDING_FROM_IMAGE" ]]; then app_source_image="$(plugn trigger git-get-property "$APP" "source-image")"
fn-nginx-vhosts-copy-from-image "$APP" "$DOKKU_BUILDING_FROM_IMAGE" "$CONF_SIGIL_PATH" if [[ -n "$app_source_image" ]]; then
fn-nginx-vhosts-copy-from-image "$APP" "$app_source_image" "$CONF_SIGIL_PATH"
else else
fn-nginx-vhosts-copy-from-directory "$APP" "$SOURCECODE_WORK_DIR" "$CONF_SIGIL_PATH" fn-nginx-vhosts-copy-from-directory "$APP" "$SOURCECODE_WORK_DIR" "$CONF_SIGIL_PATH"
fi fi
+6 -4
View File
@@ -74,7 +74,10 @@ func TriggerCorePostExtract(appName string, sourceWorkDir string) error {
} }
processSpecificProcfile := fmt.Sprintf("%s.%s", existingProcfile, os.Getenv("DOKKU_PID")) processSpecificProcfile := fmt.Sprintf("%s.%s", existingProcfile, os.Getenv("DOKKU_PID"))
if os.Getenv("DOKKU_BUILDING_FROM_IMAGE") == "" { b, _ := common.PlugnTriggerOutput("git-get-property", []string{appName, "source-image"}...)
appSourceImage := strings.TrimSpace(string(b[:]))
if appSourceImage == "" {
repoProcfilePath := path.Join(sourceWorkDir, procfilePath) repoProcfilePath := path.Join(sourceWorkDir, procfilePath)
if !common.FileExists(repoProcfilePath) { if !common.FileExists(repoProcfilePath) {
return common.TouchFile(fmt.Sprintf("%s.missing", processSpecificProcfile)) return common.TouchFile(fmt.Sprintf("%s.missing", processSpecificProcfile))
@@ -84,13 +87,12 @@ func TriggerCorePostExtract(appName string, sourceWorkDir string) error {
return fmt.Errorf("Unable to extract Procfile: %v", err.Error()) return fmt.Errorf("Unable to extract Procfile: %v", err.Error())
} }
} else { } else {
if err := common.CopyFromImage(appName, os.Getenv("DOKKU_BUILDING_FROM_IMAGE"), procfilePath, processSpecificProcfile); err != nil { if err := common.CopyFromImage(appName, appSourceImage, procfilePath, processSpecificProcfile); err != nil {
return common.TouchFile(fmt.Sprintf("%s.missing", processSpecificProcfile)) return common.TouchFile(fmt.Sprintf("%s.missing", processSpecificProcfile))
} }
} }
b, err := sh.Command("procfile-util", "check", "-P", processSpecificProcfile).CombinedOutput() if b, err := sh.Command("procfile-util", "check", "-P", processSpecificProcfile).CombinedOutput(); err != nil {
if err != nil {
return fmt.Errorf(strings.TrimSpace(string(b[:]))) return fmt.Errorf(strings.TrimSpace(string(b[:])))
} }
return nil return nil
+5
View File
@@ -0,0 +1,5 @@
FROM python:3.11.0-buster
WORKDIR /app
COPY . /app
+129
View File
@@ -0,0 +1,129 @@
#!/usr/bin/env bats
load test_helper
setup() {
global_setup
[[ -f "$DOKKU_ROOT/VHOST" ]] && cp -fp "$DOKKU_ROOT/VHOST" "$DOKKU_ROOT/VHOST.bak"
create_app
}
teardown() {
destroy_app 0 $TEST_APP
[[ -f "$DOKKU_ROOT/VHOST.bak" ]] && mv "$DOKKU_ROOT/VHOST.bak" "$DOKKU_ROOT/VHOST" && chown dokku:dokku "$DOKKU_ROOT/VHOST"
global_teardown
}
@test "(nginx-vhosts) git:from-image nginx.conf.sigil" {
local CUSTOM_TMP=$(mktemp -d "/tmp/dokku.me.XXXXX")
trap 'popd &>/dev/null || true; rm -rf "$CUSTOM_TMP"' INT TERM
rmdir "$CUSTOM_TMP" && cp -r "${BATS_TEST_DIRNAME}/../../tests/apps/python" "$CUSTOM_TMP"
pushd $CUSTOM_TMP
custom_nginx_template "$TEST_APP" "$CUSTOM_TMP"
run /bin/bash -c "dokku nginx:set --global nginx-conf-sigil-path"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "git init"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "git add nginx.conf.sigil"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "git commit -m 'Add custom nginx.conf.sigil'"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "docker image build -t dokku-test/$TEST_APP:latest -f $CUSTOM_TMP/alt.Dockerfile $CUSTOM_TMP"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "dokku git:from-image $TEST_APP dokku-test/$TEST_APP:latest"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Overriding default nginx.conf with detected nginx.conf.sigil"
run /bin/bash -c "dokku nginx:set --global nginx-conf-sigil-path dokku/nginx.conf.sigil"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "dokku proxy:report $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "dokku ps:rebuild $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Overriding default nginx.conf with detected nginx.conf.sigil" 0
run /bin/bash -c "mkdir -p $CUSTOM_TMP/dokku"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "mv $CUSTOM_TMP/nginx.conf.sigil $CUSTOM_TMP/dokku/nginx.conf.sigil"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "git add ."
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "git commit -m 'Move the nginx.conf.sigil'"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "docker image build -t dokku-test/$TEST_APP:v2 -f $CUSTOM_TMP/alt.Dockerfile $CUSTOM_TMP"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "dokku git:from-image $TEST_APP dokku-test/$TEST_APP:v2"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Overriding default nginx.conf with detected nginx.conf.sigil"
run /bin/bash -c "dokku nginx:set --global nginx-conf-sigil-path"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "dokku ps:rebuild $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Overriding default nginx.conf with detected nginx.conf.sigil" 0
run /bin/bash -c "dokku nginx:set $TEST_APP nginx-conf-sigil-path dokku/nginx.conf.sigil"
echo "output: $output"
echo "status: $status"
assert_success
run /bin/bash -c "dokku ps:rebuild $TEST_APP"
echo "output: $output"
echo "status: $status"
assert_success
assert_output_contains "Overriding default nginx.conf with detected nginx.conf.sigil"
run /bin/bash -c "docker image rm dokku-test/$TEST_APP:latest dokku-test/$TEST_APP:v2"
echo "output: $output"
echo "status: $status"
assert_success
}