diff --git a/docs/appendices/0.38.0-migration-guide.md b/docs/appendices/0.38.0-migration-guide.md index e547325ec..e5e465ee3 100644 --- a/docs/appendices/0.38.0-migration-guide.md +++ b/docs/appendices/0.38.0-migration-guide.md @@ -4,6 +4,7 @@ - Dokku now generates a minimal nginx configuration for apps without running `web` processes (undeployed apps, apps with no `web` process type, or apps with stopped web processes). This configuration returns `502 Bad Gateway` responses, ensuring the app's domain resolves and monitoring tools can detect non-200 status codes. The configuration is automatically replaced with the full proxy configuration once the app is deployed with running `web` processes. See the [nginx documentation](/docs/networking/proxies/nginx.md#nginx-configuration-for-undeployed-apps) for more details. - Users with custom `nginx.conf.sigil` templates that reference `DOKKU_APP_WEB_LISTENERS` should be aware that this variable may now be empty when the template is rendered for apps without running web processes. Custom templates should handle this case gracefully, for example by using a conditional to serve an error page instead of proxying: + ``` location / { {{ if $.DOKKU_APP_WEB_LISTENERS }} @@ -13,3 +14,5 @@ {{ end }} } ``` + +- The path on disk to both the global `ENV` file and app `ENV` files have been moved. Users should reference environment variables via the provided plugin triggers rather than directly sourcing the ENV files. Existing ENV files are left untouched and will be removed on the subsequent Dokku install. diff --git a/docs/configuration/environment-variables.md b/docs/configuration/environment-variables.md index 9c29dcc38..d862ba640 100644 --- a/docs/configuration/environment-variables.md +++ b/docs/configuration/environment-variables.md @@ -22,12 +22,12 @@ Environment variables are available both at run time and during the application For buildpack deploys, Dokku will create a `/app/.env` file that can be used for legacy buildpacks. Note that this is _not_ updated when `config:set` or `config:unset` is called, and is only written during a `deploy` or `ps:rebuild`. Developers are encouraged to instead read from the application environment directly, as the proper values will be available then. > [!NOTE] -> Global `ENV` files are sourced before app-specific `ENV` files. This means that app-specific variables will take precedence over global variables. Configuring your global `ENV` file is manual, and should be considered potentially dangerous as configuration applies to all applications. +> Global environment variables are sourced before app-specific environment variables. This means that app-specific variables will take precedence over global variables. Configuring global environment variables should be considered potentially dangerous as configuration applies to all applications. You can set multiple environment variables at once: ```shell -dokku config:set node-js-app ENV=prod COMPILE_ASSETS=1 +dokku config:set node-js-app APP_ENV=prod COMPILE_ASSETS=1 ``` Whitespace and special characters get tricky. If you are using dokku locally you don't need to do any special escaping. If you are using dokku over ssh you will need to backslash-escape spaces: @@ -45,7 +45,7 @@ dokku config:set --encoded node-js-app KEY="$(base64 -w 0 ~/.ssh/id_rsa)" When setting or unsetting environment variables, you may wish to avoid an application restart. This is useful when developing plugins or when setting multiple environment variables in a scripted manner. To do so, use the `--no-restart` flag: ```shell -dokku config:set --no-restart node-js-app ENV=prod +dokku config:set --no-restart node-js-app APP_ENV=prod ``` If you wish to have the variables output in an `eval`-compatible form, you can use the `config:export` command @@ -54,7 +54,7 @@ If you wish to have the variables output in an `eval`-compatible form, you can u dokku config:export node-js-app # outputs variables in the form: # -# export ENV='prod' +# export APP_ENV='prod' # export COMPILE_ASSETS='1' # source in all the node-js-app app environment variables @@ -68,7 +68,7 @@ dokku config:export --format shell node-js-app # outputs variables in the form: # -# ENV='prod' COMPILE_ASSETS='1' +# APP_ENV='prod' COMPILE_ASSETS='1' ``` ## Setting Environment Variables via app.json diff --git a/docs/getting-started/upgrading/index.md b/docs/getting-started/upgrading/index.md index 7c3ae989d..1fe40c8bd 100644 --- a/docs/getting-started/upgrading/index.md +++ b/docs/getting-started/upgrading/index.md @@ -18,6 +18,7 @@ Docker releases updates periodically to their engine. We recommend reading their Before upgrading, check the migration guides to get comfortable with new features and prepare your deployment to be upgraded. +- [Upgrading to 0.38](/docs/appendices/0.38.0-migration-guide.md) - [Upgrading to 0.37](/docs/appendices/0.37.0-migration-guide.md) - [Upgrading to 0.36](/docs/appendices/0.36.0-migration-guide.md) - [Upgrading to 0.35](/docs/appendices/0.35.0-migration-guide.md) @@ -152,4 +153,3 @@ new buildpacks that were released: ```shell dokku ps:rebuild --all ``` - diff --git a/plugins/common/properties.go b/plugins/common/properties.go index 7653e168b..a91f8d5e7 100644 --- a/plugins/common/properties.go +++ b/plugins/common/properties.go @@ -541,6 +541,10 @@ func PropertySetup(pluginName string) error { } func PropertySetupApp(pluginName string, appName string) error { + if err := PropertySetup(pluginName); err != nil { + return err + } + if err := makePluginAppPropertyPath(pluginName, appName); err != nil { return fmt.Errorf("Unable to create %s config directory for %s: %s", pluginName, appName, err.Error()) } diff --git a/plugins/config/.gitignore b/plugins/config/.gitignore index 3b7e2f512..63f3adbe8 100644 --- a/plugins/config/.gitignore +++ b/plugins/config/.gitignore @@ -5,3 +5,4 @@ /config-* /config_sub /post-* +/install diff --git a/plugins/config/Makefile b/plugins/config/Makefile index d4cb33428..c1f6c067f 100644 --- a/plugins/config/Makefile +++ b/plugins/config/Makefile @@ -1,6 +1,6 @@ GOARCH ?= amd64 SUBCOMMANDS = subcommands/bundle subcommands/clear subcommands/export subcommands/get subcommands/import subcommands/keys subcommands/show subcommands/set subcommands/unset -TRIGGERS = triggers/config-export triggers/config-get triggers/config-get-global triggers/config-set triggers/config-unset triggers/post-app-clone-setup triggers/post-app-rename-setup +TRIGGERS = triggers/config-export triggers/config-get triggers/config-get-global triggers/install triggers/config-set triggers/config-unset triggers/post-app-clone-setup triggers/post-app-rename-setup triggers/post-create triggers/post-delete BUILD = commands config_sub subcommands triggers PLUGIN_NAME = config diff --git a/plugins/config/config_test.go b/plugins/config/config_test.go index 9eb0d0358..e512fc42c 100644 --- a/plugins/config/config_test.go +++ b/plugins/config/config_test.go @@ -2,6 +2,7 @@ package config import ( "os" + "path/filepath" "strings" "testing" @@ -13,8 +14,10 @@ import ( var ( testAppName = "test-app-1" dokkuRoot = common.MustGetEnv("DOKKU_ROOT") - testAppDir = strings.Join([]string{dokkuRoot, testAppName}, "/") - globalConfigFile = strings.Join([]string{dokkuRoot, "ENV"}, "/") + dokkuLibRoot = common.MustGetEnv("DOKKU_LIB_ROOT") + testAppRoot = filepath.Join(dokkuRoot, testAppName) + testAppDir = filepath.Join(dokkuLibRoot, "config", testAppName) + globalConfigFile = filepath.Join(dokkuLibRoot, "config", "--global", "ENV") ) func setupTests() (err error) { @@ -26,12 +29,14 @@ func setupTests() (err error) { } func setupTestApp() (err error) { + Expect(os.MkdirAll(testAppRoot, 0766)).To(Succeed()) Expect(os.MkdirAll(testAppDir, 0766)).To(Succeed()) b := []byte("export testKey=TESTING\n") if err = os.WriteFile(strings.Join([]string{testAppDir, "/ENV"}, ""), b, 0644); err != nil { return } + Expect(os.MkdirAll(filepath.Dir(globalConfigFile), 0766)).To(Succeed()) b = []byte("export testKey=GLOBAL_TESTING\nexport globalKey=GLOBAL_VALUE") if err = os.WriteFile(globalConfigFile, b, 0644); err != nil { return @@ -40,6 +45,7 @@ func setupTestApp() (err error) { } func teardownTestApp() { + os.RemoveAll(testAppRoot) os.RemoveAll(testAppDir) } diff --git a/plugins/config/environment.go b/plugins/config/environment.go index 6e341dd62..3e9d5313d 100644 --- a/plugins/config/environment.go +++ b/plugins/config/environment.go @@ -392,9 +392,9 @@ func loadFromFileJSON(name string, filename string) (env *Env, err error) { }, nil } func getAppFile(appName string) (string, error) { - return filepath.Join(common.MustGetEnv("DOKKU_ROOT"), appName, "ENV"), nil + return filepath.Join(common.MustGetEnv("DOKKU_LIB_ROOT"), "config", appName, "ENV"), nil } func getGlobalFile() string { - return filepath.Join(common.MustGetEnv("DOKKU_ROOT"), "ENV") + return filepath.Join(common.MustGetEnv("DOKKU_LIB_ROOT"), "config", "--global", "ENV") } diff --git a/plugins/config/src/triggers/triggers.go b/plugins/config/src/triggers/triggers.go index 4b0783a0a..dd5f1da77 100644 --- a/plugins/config/src/triggers/triggers.go +++ b/plugins/config/src/triggers/triggers.go @@ -51,6 +51,8 @@ func main() { restart = flag.Arg(1) } err = config.TriggerConfigUnset(appName, key, common.ToBool(restart)) + case "install": + err = config.TriggerInstall() case "post-app-clone-setup": oldAppName := flag.Arg(0) newAppName := flag.Arg(1) @@ -59,6 +61,12 @@ func main() { oldAppName := flag.Arg(0) newAppName := flag.Arg(1) err = config.TriggerPostAppRenameSetup(oldAppName, newAppName) + case "post-create": + appName := flag.Arg(0) + err = config.TriggerPostCreate(appName) + case "post-delete": + appName := flag.Arg(0) + err = config.TriggerPostDelete(appName) default: err = fmt.Errorf("Invalid plugin trigger call: %s", trigger) } diff --git a/plugins/config/triggers.go b/plugins/config/triggers.go index 72bbc6719..724aa348d 100644 --- a/plugins/config/triggers.go +++ b/plugins/config/triggers.go @@ -2,7 +2,11 @@ package config import ( "fmt" + "os" + "path/filepath" "strconv" + + "github.com/dokku/dokku/plugins/common" ) // TriggerConfigExport returns a global config value by key @@ -56,6 +60,130 @@ func TriggerConfigUnset(appName string, key string, restart bool) error { return nil } +func setupAppConfigDir(appName string) error { + appFile, err := getAppFile(appName) + if err != nil { + return err + } + + appConfigDir := filepath.Dir(appFile) + if err := os.MkdirAll(appConfigDir, 0755); err != nil { + return err + } + + return common.SetPermissions(common.SetPermissionInput{ + Filename: appConfigDir, + Mode: os.FileMode(0755), + }) +} + +func migrateGlobalEnv() error { + if err := common.PropertySetup("--global"); err != nil { + return fmt.Errorf("Unable to setup global environment: %s", err.Error()) + } + + oldGlobalEnvFile := filepath.Join(common.MustGetEnv("DOKKU_ROOT"), "ENV") + isGlobalMigrated := common.PropertyGetDefault("config", "--global", "env-migrated", "") + if isGlobalMigrated == "true" { + return nil + } + + oldGlobalEnv, err := loadFromFile("--global", oldGlobalEnvFile) + if err != nil { + return fmt.Errorf("Unable to load old global environment: %s", err.Error()) + } + + globalEnv, err := LoadGlobalEnv() + if err != nil { + return fmt.Errorf("Unable to load global environment: %s", err.Error()) + } + + globalEnv.Merge(oldGlobalEnv) + if err := globalEnv.Write(); err != nil { + return fmt.Errorf("Unable to write global environment: %s", err.Error()) + } + + if err := common.SetPermissions(common.SetPermissionInput{ + Filename: globalEnv.Filename(), + Mode: os.FileMode(0600), + }); err != nil { + return fmt.Errorf("Unable to set permissions on global environment: %s", err.Error()) + } + + if err := common.PropertyWrite("config", "--global", "env-migrated", "true"); err != nil { + return fmt.Errorf("Unable to set env-migrated property: %s", err.Error()) + } + + return nil +} + +// TriggerInstall runs the install step for the config plugin +func TriggerInstall() error { + if err := common.PropertySetup("config"); err != nil { + return fmt.Errorf("Unable to install the config plugin: %s", err.Error()) + } + + if err := migrateGlobalEnv(); err != nil { + return fmt.Errorf("Unable to migrate global environment: %s", err.Error()) + } + + apps, err := common.UnfilteredDokkuApps() + if err != nil { + return nil + } + + // migrate all app ENV files to config path + for _, appName := range apps { + if err := common.PropertySetupApp("config", appName); err != nil { + return fmt.Errorf("Unable to setup app environment: %s", err.Error()) + } + + if err := setupAppConfigDir(appName); err != nil { + return fmt.Errorf("Unable to setup app config directory: %s", err.Error()) + } + + oldEnvFile := filepath.Join(common.AppRoot(appName), "ENV") + isMigrated := common.PropertyGetDefault("config", appName, "env-migrated", "") + // delete the old file on the next install + if isMigrated == "true" { + if err := os.RemoveAll(oldEnvFile); err != nil { + return fmt.Errorf("Unable to remove old ENV file: %s", err.Error()) + } + continue + } + + // skip if the file doesn't exist + if _, err := os.Stat(oldEnvFile); err != nil { + if err := common.PropertyWrite("config", appName, "env-migrated", "true"); err != nil { + return fmt.Errorf("Unable to set env-migrated property: %s", err.Error()) + } + continue + } + + // merge in the old env into the new env + oldEnv, err := loadFromFile(appName, oldEnvFile) + if err != nil { + return fmt.Errorf("Unable to load old environment: %s", err.Error()) + } + + env, err := LoadAppEnv(appName) + if err != nil { + return fmt.Errorf("Unable to load app environment: %s", err.Error()) + } + + env.Merge(oldEnv) + if err := env.Write(); err != nil { + return fmt.Errorf("Unable to write app environment: %s", err.Error()) + } + + if err := common.PropertyWrite("config", appName, "env-migrated", "true"); err != nil { + return fmt.Errorf("Unable to set env-migrated property: %s", err.Error()) + } + } + + return nil +} + // TriggerPostAppCloneSetup creates new buildpacks files func TriggerPostAppCloneSetup(oldAppName string, newAppName string) error { oldEnv, err := LoadAppEnv(oldAppName) @@ -95,3 +223,22 @@ func TriggerPostAppRenameSetup(oldAppName string, newAppName string) error { return nil } + +// TriggerPostCreate ensures apps have the correct config structure +func TriggerPostCreate(appName string) error { + if err := common.PropertySetupApp("config", appName); err != nil { + return err + } + + return setupAppConfigDir(appName) +} + +// TriggerPostDelete destroys the config data for a given app container +func TriggerPostDelete(appName string) error { + appFile, err := getAppFile(appName) + if err == nil { + os.RemoveAll(filepath.Dir(appFile)) + } + + return common.PropertyDestroy("config", appName) +} diff --git a/tests.mk b/tests.mk index d13240a98..6178cae64 100644 --- a/tests.mk +++ b/tests.mk @@ -155,6 +155,7 @@ ci-go-coverage-plugin: mkdir -p test-results/coverage docker run --rm \ -e DOKKU_ROOT=/home/dokku \ + -e DOKKU_LIB_ROOT=/var/lib/dokku \ -e CODACY_TOKEN=$$CODACY_TOKEN \ -e CIRCLE_SHA1=$$CIRCLE_SHA1 \ -e GO111MODULE=on \ diff --git a/tests/unit/config-oddities.bats b/tests/unit/config-oddities.bats index 011d46e16..7ec94b7f1 100644 --- a/tests/unit/config-oddities.bats +++ b/tests/unit/config-oddities.bats @@ -4,16 +4,16 @@ load test_helper setup() { global_setup - [[ -f ${DOKKU_ROOT}/ENV ]] && mv -f ${DOKKU_ROOT}/ENV ${DOKKU_ROOT}/ENV.bak - sudo -H -u dokku /bin/bash -c "echo 'export global_test=true' > ${DOKKU_ROOT}/ENV" + mkdir -p "${DOKKU_LIB_ROOT}/config/--global" + [[ -f ${DOKKU_LIB_ROOT}/config/--global/ENV ]] && mv -f ${DOKKU_LIB_ROOT}/config/--global/ENV ${DOKKU_LIB_ROOT}/config/--global/ENV.bak + sudo -H -u dokku /bin/bash -c "echo 'export global_test=true' > ${DOKKU_LIB_ROOT}/config/--global/ENV" create_app } teardown() { destroy_app - ls -la ${DOKKU_ROOT} - if [[ -f ${DOKKU_ROOT}/ENV.bak ]]; then - mv -f ${DOKKU_ROOT}/ENV.bak ${DOKKU_ROOT}/ENV + if [[ -f ${DOKKU_LIB_ROOT}/config/--global/ENV.bak ]]; then + mv -f ${DOKKU_LIB_ROOT}/config/--global/ENV.bak ${DOKKU_LIB_ROOT}/config/--global/ENV fi global_teardown } diff --git a/tests/unit/config.bats b/tests/unit/config.bats index e9bfb1fce..d29605468 100644 --- a/tests/unit/config.bats +++ b/tests/unit/config.bats @@ -4,16 +4,16 @@ load test_helper setup() { global_setup - [[ -f ${DOKKU_ROOT}/ENV ]] && mv -f ${DOKKU_ROOT}/ENV ${DOKKU_ROOT}/ENV.bak - sudo -H -u dokku /bin/bash -c "echo 'export global_test=true' > ${DOKKU_ROOT}/ENV" + mkdir -p "${DOKKU_LIB_ROOT}/config/--global" + [[ -f ${DOKKU_LIB_ROOT}/config/--global/ENV ]] && mv -f ${DOKKU_LIB_ROOT}/config/--global/ENV ${DOKKU_LIB_ROOT}/config/--global/ENV.bak + sudo -H -u dokku /bin/bash -c "echo 'export global_test=true' > ${DOKKU_LIB_ROOT}/config/--global/ENV" create_app } teardown() { destroy_app - ls -la ${DOKKU_ROOT} - if [[ -f ${DOKKU_ROOT}/ENV.bak ]]; then - mv -f ${DOKKU_ROOT}/ENV.bak ${DOKKU_ROOT}/ENV + if [[ -f ${DOKKU_LIB_ROOT}/config/--global/ENV.bak ]]; then + mv -f ${DOKKU_LIB_ROOT}/config/--global/ENV.bak ${DOKKU_LIB_ROOT}/config/--global/ENV fi global_teardown }