feat: add initial support for cloud native buildpacks

This change adds a new builder - builder-cnb - via the 'DOKKU_CNB_EXPERIMENTAL' environment variable. While support for customizing pack builds is not currently implemented and there may be other missing functionality, this initial implementation allows folks to start playing with CNB.

Future releases of Dokku will provide tighter CNB integration and eventually switch to CNB for default building, as well as allow users to better select builders needed for their applications.

All new code for CNB support should be considered experimental and subject to change as the support evolves over time.

docs/deployment/application-deployment.md

@@ -102,7 +102,7 @@ Once the deploy is complete, the application's web URL will be generated as abov
 
 Dokku supports deploying applications in a few ways:
 
-- [Heroku buildpacks](https://devcenter.heroku.com/articles/buildpacks) via [Herokuish](https://github.com/gliderlabs/herokuish#buildpacks): See the [buildpacks documentation](/docs/deployment/methods/buildpacks.md) to learn about the different ways to specify a buildpack.
+- [Heroku buildpacks](https://devcenter.heroku.com/articles/buildpacks) via [Herokuish](https://github.com/gliderlabs/herokuish#buildpacks): See the [herokuish buildpacks documentation](/docs/deployment/methods/herokuish-buildpacks.md) to learn about the different ways to specify a buildpack.
 - [Dockerfile](https://docs.docker.com/reference/builder/): See the [dockerfile documentation](/docs/deployment/methods/dockerfiles.md) to learn about the different ways to configure Dockerfile-based deploys.
 - [Docker Image](https://docs.docker.com/get-started/overview/#docker-objects): See the [docker image documentation](/docs/deployment/methods/images.md) to learn about how to deploy a Docker Image.
 
@@ -211,7 +211,7 @@ See the [image tagging documentation](/docs/deployment/methods/images.md) for mo
 
 ## Specifying a custom buildpack
 
-See the [buildpack documentation](/docs/deployment/methods/buildpacks.md) for more information on how to specify a set of custom buildpacks for your application.
+See the [herokuish buildpack documentation](/docs/deployment/methods/herokuish-buildpacks.md) for more information on how to specify a set of custom buildpacks for your application.
 
 ## Removing a deployed app
 

docs/deployment/methods/cloud-native-buildpacks.md

@@ -0,0 +1,35 @@
+# Cloud Native Buildpacks (Experimental)
+
+> New as of 0.22.0
+
+Cloud Native Buildpacks are an evolution over the Buildpacks technology provided by the Herokuish builder. See the [herokuish buildpacks documentation](/docs/deployment/methods/herokuish.md) for more information on how to clear buildpack build cache for an application.
+
+> Warning: This functionality uses the `pack` cli from the [Cloud Native Buildpacks](https://buildpacks.io) project to build apps. As the integration is experimental in Dokku, it is likely to change over time.
+
+## Usage
+
+To use this builder instead of either `Dockerfile` or `herokuish`, you must set the `DOKKU_CNB_EXPERIMENTAL` environment variable for your app to `1`.
+
+```shell
+dokku config:set --no-restart node-js-app DOKKU_CNB_EXPERIMENTAL=1
+```
+
+The `pack` cli tool is not included by default with Dokku or as a dependency. It must also be installed as shown on [this page](https://buildpacks.io/docs/tools/pack/).
+
+Builds will proceed with the `pack` cli for the app from then on.
+
+## Caveats
+
+As this functionality is highly experimental, there are a number of caveats. Please note that not all issuesare listed below.
+
+- The `heroku/buildpacks:latest` CNB builder is currently enforced. Specifying specific builders is not currently possible.
+  - A future release will allow modifying the chosen builder.
+- Specifying specific buildpacks is not currently possible.
+  - A future release will add support for the `buildpacks` plugin.
+- There is currently no way to specify extra arguments for `pack` cli invocations.
+  - A future release will add support for injecting extra arguments during the build process.
+- The default process type is `web`.
+- Build cache is stored in Docker volumes instead of on disk. As such, `repo:purge-cache` currently has no effect.
+  - A future version will add integration with the `repo` plugin.
+- `pack` is not currently included with Dokku, nor is it added as a package dependency.
+  - A future version will include it as a package dependency.

docs/deployment/methods/git.md

@@ -8,7 +8,11 @@ git:report [<app>] [<flag>]              # Displays a git report for one or more
 git:set <app> <key> (<value>)            # Set or clear a git property for an app
 ```
 
-Git-based deployment has been the traditional method of deploying applications in Dokku. As of v0.12.0, Dokku introduces a few ways to customize the experience of deploying via `git push`. A Git-based deployment currently supports building applications via both [Buildpack](/docs/deployment/methods/buildpacks.md) and [Dockerfile](/docs/deployment/methods/dockerfiles.md). 
+Git-based deployment has been the traditional method of deploying applications in Dokku. As of v0.12.0, Dokku introduces a few ways to customize the experience of deploying via `git push`. A Git-based deployment currently supports building applications via:
+
+- [Cloud Native Buildpacks](/docs/deployment/methods/cloud-native-buildpacks.md)
+- [Herokuish Buildpack](/docs/deployment/methods/herokuish-buildpacks.md)
+- [Dockerfiles](/docs/deployment/methods/dockerfiles.md)
 
 ## Usage
 

docs/deployment/methods/herokuish-buildpacks.md

@@ -1,4 +1,4 @@
-# Buildpack Deployment
+# Herokuish Buildpack Deployment
 
 > Subcommands new as of 0.15.0
 

docs/development/plugin-creation.md

@@ -97,12 +97,12 @@ If a plugin depends on a specific command-line tool, check whether that tool exi
 ```shell
 # `command -v` example
 if ! command -v "nginx" &>/dev/null; then
-  log-fail "Missing nginx, install it"
+  dokku_log_fail "Missing nginx, install it"
 fi
 
 # `which` example
 if ! which nginx >/dev/null 2>&1; then
-  log-fail "Missing nginx, install it"
+  dokku_log_fail "Missing nginx, install it"
 fi
 ```
 

docs/development/plugin-triggers.md

@@ -1015,6 +1015,24 @@ set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
 # TODO
 ```
 
+### `post-build-cnb`
+
+> Warning: The cnb plugin trigger apis are under development and may change
+> between minor releases until the 1.0 release.
+
+- Description: Allows you to run commands after the build image is create for a given app. Only applies to apps using cnb.
+- Invoked by: `internal function dokku_build() (build phase)`
+- Arguments: `$APP` `$SOURCECODE_WORK_DIR`
+- Example:
+
+```shell
+#!/usr/bin/env bash
+
+set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
+
+# TODO
+```
+
 ### `post-build-dockerfile`
 
 - Description: Allows you to run commands after the build image is create for a given app. Only applies to apps using a dockerfile.
@@ -1231,6 +1249,29 @@ verify_app_name "$APP"
 # TODO
 ```
 
+### `post-release-cnb`
+
+> Warning: The cnb plugin trigger apis are under development and may change
+> between minor releases until the 1.0 release.
+
+> Warning: Image mutation in this trigger may result in an invalid run state, and is heavily discouraged.
+
+- Description: Allows you to run commands after environment variables are set for the release step of the deploy. Only applies to apps using cnb.
+- Invoked by: `internal function dokku_release() (release phase)`
+- Arguments: `$APP $IMAGE_TAG`
+- Example:
+
+```shell
+#!/usr/bin/env bash
+
+set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
+source "$PLUGIN_CORE_AVAILABLE_PATH/common/functions"
+APP="$1"; IMAGE_TAG="$2"; IMAGE=$(get_app_image_name $APP $IMAGE_TAG)
+verify_app_name "$APP"
+
+# TODO
+```
+
 ### `post-release-dockerfile`
 
 > Warning: Image mutation in this trigger may result in an invalid run state, and is heavily discouraged.
@@ -1284,6 +1325,24 @@ set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
 # TODO
 ```
 
+### `pre-build-cnb`
+
+> Warning: The cnb plugin trigger apis are under development and may change
+> between minor releases until the 1.0 release.
+
+- Description: Allows you to run commands before the build image is created for a given app. For instance, this can be useful to add env vars to your container. Only applies to apps using cnb.
+- Invoked by: `internal function dokku_build() (build phase)`
+- Arguments: `$APP` `$SOURCECODE_WORK_DIR`
+- Example:
+
+```shell
+#!/usr/bin/env bash
+
+set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
+
+# TODO
+```
+
 ### `pre-build-dockerfile`
 
 - Description: Allows you to run commands before the build image is created for a given app. For instance, this can be useful to add env vars to your container. Only applies to apps using a dockerfile.
@@ -1425,6 +1484,27 @@ DOCKER_COMMIT_LABEL_ARGS=("--change" "LABEL org.label-schema.schema-version=1.0"
 docker commit "${DOCKER_COMMIT_LABEL_ARGS[@]}" $CID $IMAGE >/dev/null
 ```
 
+
+### `pre-release-cnb`
+
+> Warning: The cnb plugin trigger apis are under development and may change
+> between minor releases until the 1.0 release.
+
+- Description: Allows you to run commands before environment variables are set for the release step of the deploy. Only applies to apps using cnb.
+- Invoked by: `internal function dokku_release() (release phase)`
+- Arguments: `$APP $IMAGE_TAG`
+- Example:
+
+```shell
+#!/usr/bin/env bash
+
+set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
+source "$PLUGIN_CORE_AVAILABLE_PATH/common/functions"
+APP="$1"; IMAGE_TAG="$2";
+
+# TODO
+```
+
 ### `pre-release-dockerfile`
 
 - Description: Allows you to run commands before environment variables are set for the release step of the deploy. Only applies to apps using a dockerfile.

docs/template.html

@@ -128,7 +128,8 @@
 
             <a href="#" class="list-group-item disabled">Deployment Methods</a>
 
-            <a href="/{{NAME}}/deployment/methods/buildpacks/" class="list-group-item">Buildpack Deployment</a>
+            <a href="/{{NAME}}/deployment/methods/cloud-native-buildpacks/" class="list-group-item">Cloud Native Buildpacks</a>
+            <a href="/{{NAME}}/deployment/methods/herokuish-buildpacks/" class="list-group-item">Herokuish Buildpacks</a>
             <a href="/{{NAME}}/deployment/methods/dockerfiles/" class="list-group-item">Dockerfile Deployment</a>
             <a href="/{{NAME}}/deployment/methods/images/" class="list-group-item">Docker Image Deployment</a>
             <a href="/{{NAME}}/deployment/methods/git/" class="list-group-item">Git Deployment</a>

docs/viewdocs.json

@@ -10,7 +10,8 @@
     "process-management": "deployment/process-management/",
     "remote-commands": "deployment/remote-commands/",
 
-    "deployment/buildpacks": "deployment/methods/buildpacks/",
+    "deployment/buildpacks": "deployment/methods/herokuish-buildpacks/",
+    "deployment/methods/buildpacks": "deployment/methods/herokuish-buildpacks/",
     "deployment/dockerfiles": "deployment/methods/dockerfiles/",
     "deployment/images": "deployment/methods/images/",
     "configuration-management": "configuration/environment-variables/",

plugins/20_events/post-build-cnb

@@ -0,0 +1 @@
+hook

plugins/20_events/post-release-cnb

@@ -0,0 +1 @@
+hook

plugins/20_events/pre-build-cnb

@@ -0,0 +1 @@
+hook

plugins/20_events/pre-release-cnb

@@ -0,0 +1 @@
+hook

plugins/builder-cnb/builder-build

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+set -eo pipefail
+[[ $DOKKU_TRACE ]] && set -x
+source "$PLUGIN_CORE_AVAILABLE_PATH/common/functions"
+source "$PLUGIN_AVAILABLE_PATH/config/functions"
+
+trigger-builder-cnb-builder-build() {
+  declare desc="builder-cnb builder-build plugin trigger"
+  declare trigger="builder-build"
+  declare BUILDER_TYPE="$1" APP="$2" SOURCECODE_WORK_DIR="$3"
+
+  if [[ "$BUILDER_TYPE" != "cnb" ]]; then
+    return
+  fi
+
+  if ! command -v "pack" &>/dev/null; then
+    dokku_log_fail "Missing pack, install it"
+  fi
+
+  local IMAGE=$(get_app_image_name "$APP")
+
+  pushd "$SOURCECODE_WORK_DIR" &>/dev/null
+
+  local TMP_FILE=$(mktemp "/tmp/dokku-${DOKKU_PID}-${FUNCNAME[0]}.XXXXXX")
+  trap "rm -rf '$TMP_FILE' >/dev/null" RETURN INT TERM
+
+  config_export app "$APP" --format envfile --merged > "$TMP_FILE"
+
+  plugn trigger pre-build-cnb "$APP" "$SOURCECODE_WORK_DIR"
+
+  pack build "$IMAGE" --builder heroku/buildpacks --path "$SOURCECODE_WORK_DIR" --default-process web
+
+  echo "FROM $IMAGE" | suppress_output "$DOCKER_BIN" image build --label=com.dokku.image-stage=build --label=org.label-schema.schema-version=1.0 --label=org.label-schema.vendor=dokku --label=com.dokku.app-name=$APP --label=dokku -t "$IMAGE" -
+
+  plugn trigger post-build-cnb "$APP" "$SOURCECODE_WORK_DIR"
+}
+
+trigger-builder-cnb-builder-build "$@"

plugins/builder-cnb/builder-release

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+set -eo pipefail
+[[ $DOKKU_TRACE ]] && set -x
+source "$PLUGIN_CORE_AVAILABLE_PATH/common/functions"
+
+trigger-builder-cnb-builder-release() {
+  declare desc="builder-cnb builder-release plugin trigger"
+  declare trigger="builder-release"
+  declare BUILDER_TYPE="$1" APP="$2" IMAGE_TAG="$3"
+
+  if [[ "$BUILDER_TYPE" != "cnb" ]]; then
+    return
+  fi
+
+  plugn trigger pre-release-cnb "$APP" "$IMAGE_TAG"
+
+  local IMAGE=$(get_app_image_name "$APP" "$IMAGE_TAG")
+  echo "FROM $IMAGE" | suppress_output "$DOCKER_BIN" image build --label=com.dokku.image-stage=release --label=com.dokku.app-name=$APP --label=org.label-schema.schema-version=1.0 --label=org.label-schema.vendor=dokku --label=dokku -t "$IMAGE" -
+  plugn trigger post-release-cnb "$APP" "$IMAGE_TAG"
+}
+
+trigger-builder-cnb-builder-release "$@"

plugins/builder-cnb/plugin.toml

@@ -0,0 +1,4 @@
+[plugin]
+description = "dokku core builder-cnb plugin"
+version = "0.21.4"
+[plugin.config]

plugins/builder-dockerfile/builder-release

@@ -12,7 +12,6 @@ trigger-builder-dockerfile-builder-release() {
     return
   fi
 
-  # buildstep plugins don't necessarily make sense for dockerfiles. call the new breed!!!
   plugn trigger pre-release-dockerfile "$APP" "$IMAGE_TAG"
 
   local IMAGE=$(get_app_image_name "$APP" "$IMAGE_TAG")

plugins/common/functions

@@ -293,20 +293,41 @@ get_running_image_tag() {
   echo "$RUNNING_IMAGE_TAG"
 }
 
+is_image_cnb_based() {
+  declare desc="returns true if app image is based on cnb"
+  declare IMAGE="$1"
+
+  if [[ -z "$IMAGE" ]]; then
+    return 1
+  fi
+
+  CNB_STACK_ID="$("$DOCKER_BIN" image inspect --format '{{ index .Config.Labels "io.buildpacks.stack.id" }}' "$IMAGE")"
+  if [[ -n "$CNB_STACK_ID" ]]; then
+    return 0
+  fi
+
+  return 1
+}
+
 is_image_herokuish_based() {
-  declare desc="returns true if app image is based on herokuish"
+  declare desc="returns true if app image is based on herokuish or a buildpack-like env"
   declare IMAGE="$1" APP="$2"
   local DOKKU_APP_USER
-  if [[ -n "$APP" ]]; then
-    DOKKU_APP_USER=$(config_get "$APP" DOKKU_APP_USER || true)
-  fi
-  DOKKU_APP_USER=${DOKKU_APP_USER:="herokuishuser"}
   local USER_VALUE
 
   if [[ -z "$IMAGE" ]]; then
     return 1
   fi
 
+  if is_image_cnb_based "$IMAGE"; then
+    return 0
+  fi
+
+  if [[ -n "$APP" ]]; then
+    DOKKU_APP_USER=$(config_get "$APP" DOKKU_APP_USER || true)
+  fi
+  DOKKU_APP_USER=${DOKKU_APP_USER:="herokuishuser"}
+
   # due to how the build process works, all herokuish images have the Environment variable USER=$DOKKU_APP_USER
   USER_VALUE="$("$DOCKER_BIN" image inspect --format '{{range .Config.Env}}{{if eq . "USER='"$DOKKU_APP_USER"'"}}{{println .}}{{end}}{{end}}' "$IMAGE")"
   [[ "$USER_VALUE" == "" ]] && return 1
@@ -566,6 +587,11 @@ dokku_release() {
   declare desc="release phase"
   declare APP="$1" IMAGE_SOURCE_TYPE="$2" IMAGE_TAG="$3"
 
+  local IMAGE=$(get_app_image_name "$APP" "$IMAGE_TAG")
+  if is_image_cnb_based "$IMAGE"; then
+    IMAGE_SOURCE_TYPE="cnb"
+  fi
+
   plugn trigger builder-release "$IMAGE_SOURCE_TYPE" "$APP" "$IMAGE_TAG"
 }
 
@@ -630,9 +656,17 @@ dokku_receive() {
   local IMAGE=$(get_app_image_name "$APP")
   local IMAGE_SOURCE_TYPE="$2"
   local TMP_WORK_DIR="$3"
-  local DOKKU_SKIP_CLEANUP="$(config_get "$APP" DOKKU_SKIP_CLEANUP || true)"
+
   docker_cleanup "$APP"
-  dokku_log_info1 "Building $APP from $IMAGE_SOURCE_TYPE..."
+
+  local DOKKU_CNB_EXPERIMENTAL="$(config_get "$APP" DOKKU_CNB_EXPERIMENTAL || true)"
+  if [[ "$DOKKU_CNB_EXPERIMENTAL" == "1" ]]; then
+    IMAGE_SOURCE_TYPE="cnb"
+    dokku_log_info1 "Building $APP from $IMAGE_SOURCE_TYPE (experimental)..."
+  else
+    dokku_log_info1 "Building $APP from $IMAGE_SOURCE_TYPE..."
+  fi
+
   DOKKU_QUIET_OUTPUT=1 config_set --no-restart "$APP" DOKKU_APP_TYPE="$IMAGE_SOURCE_TYPE"
   dokku_build "$APP" "$IMAGE_SOURCE_TYPE" "$TMP_WORK_DIR"
   plugn trigger release-and-deploy "$APP"

plugins/scheduler-docker-local/scheduler-enter

@@ -72,7 +72,8 @@ trigger-scheduler-docker-local-scheduler-enter() {
   local EXEC_CMD=""
   has_tty && local DOKKU_RUN_OPTS+=" -i -t"
   local IMAGE="$("$DOCKER_BIN" container inspect --format '{{ .Config.Image }}' "$CONTAINER_ID")"
-  is_image_herokuish_based "$IMAGE" "$APP" && local EXEC_CMD="/exec"
+  is_image_herokuish_based "$IMAGE" "$APP" && EXEC_CMD="/exec"
+  is_image_cnb_based "$IMAGE" && EXEC_CMD="" && DOKKU_RUN_OPTS+=" -w /app"
   # shellcheck disable=SC2086
   "$DOCKER_BIN" container exec $DOKKU_RUN_OPTS "$CONTAINER_ID" $EXEC_CMD "${@:-$DOKKU_APP_SHELL}"
 }