Merge pull request #4860 from dokku/parallel-process-deploys

Add ability to increase the max parallelism when deploying a given process type

HISTORY.md

@@ -3370,7 +3370,6 @@ Thanks to all the contributors who helped with this release!
 - #2388: @josegonzalez Add documentation for proxy ports scheme handling
 - #2389: @josegonzalez Add plugin management documentation
 
-
 ## 0.7.0
 
 Another great minor release! There are no known backwards incompatibilities with this release, though the following may be of interest to our users:
@@ -4756,7 +4755,6 @@ This release pegs Dokku to Docker 1.6.2. Docker 1.7.0 introduced changes in `doc
 - #749: @vincentfretin Fix app certificate directory in backup-import/export
 - #750: @th4t Remove unintended phrase repetition in installation.md
 
-
 ## 0.2.0 (2013-11-24)
 
 * Added DOKKU_TRACE variable for verbose trace information

docs/appendices/0.25.0-migration-guide.md

@@ -25,7 +25,7 @@ The [dokku-registry](https://github.com/dokku/dokku-registry) plugin is now buil
 - In previous versions of Dokku, the only way to specify a custom `Dockerfile` was to use the `docker-options` plugin to set the `--file` flag for a docker build. As of 0.25.0, the `builder-dockerfile:set` command should be used instead, as outlined in the [docs here](/docs/deployment/builders/dockerfiles.md#changingthe-dockerfile-location). Usage of the old method should be migrated to the new method.
 - The `--rm` and `--rm-container` flags may be specified but no longer have any effect on `dokku run`.
 - The `--detach` flag is deprecated in favor of the `run:detached` command.
-- The `DOKKU_SCALE` file is deprecated. Please see the [process management documentation](/docs/processes/process-management.md#manually-managing-process-scaling) for more information on it's replacement with the `formations` key of the `app.json` file.
+- The `DOKKU_SCALE` file is deprecated. Please see the [process management documentation](/docs/processes/process-management.md#manually-managing-process-scaling) for more information on it's replacement with the `formation` key of the `app.json` file.
 - The hooks `post-release-buildpack`, `post-release-dockerfile`, and `post-release-pack` are deprecated in favor of `post-release-builder`. See the [plugin triggers documentation](https://dokku.com/docs/development/plugin-triggers/#post-release-builder) for more details.
 
 ## Removals

docs/deployment/schedulers/docker-local.md

@@ -37,9 +37,9 @@ Once set, you may re-enable it by setting a blank value for `disable-chown`:
 dokku scheduler-docker-local:set node-js-app disable-chown
 ```
 
-### Deploying Process Types in parallel
+### Deploying Process Types in Parallel
 
-> New as of 25.5
+> New as of 0.25.5
 
 By default, Dokku deploys an app's processes one-by-one in order, with the `web` process being deployed first. Deployment parallelism may be achieved by setting the `parallel-schedule-count` property, which defaults to `1`. Increasing this number increases the number of process types that may be deployed in parallel (with the web process being the exception).
 
@@ -60,6 +60,37 @@ Container scheduling output is shown in the order it is received, and thus may b
 
 Note that increasing the value of `parallel-schedule-count` may significantly impact CPU utilization on your host as your app containers - and their respective processes - start up. Setting a value higher than the number of available CPUs is discouraged. It is recommended that users carefully set this value so as not to overburden their server.
 
+#### Increasing parallelism within a process deploy
+
+> New as of 0.26.0
+
+By default, Dokku will deploy one instance of a given process type at a time. This can be increased by customizing the `app.json` `formation` key to include a `max_parallel` key for the given process type.
+
+An `app.json` file can be committed to the root of the pushed app repository, and must be within the built image artifact in the image's working directory as shown below.
+
+- Buildpacks: `/app/app.json`
+- Dockerfile: `WORKDIR/app.json` or `/app.json` (if no working directory specified)
+- Docker Image: `WORKDIR/app.json` or `/app.json` (if no working directory specified)
+
+The `formation` key should be specified as follows in the `app.json` file:
+
+```Procfile
+{
+  "formation": {
+    "web": {
+      "max_parallel": 1
+    },
+    "worker": {
+      "max_parallel": 4
+    }
+  }
+}
+```
+
+Omitting or removing the entry will result in parallelism for that process type to return to 1 entry at a time. This can be combined with the  `parallel-schedule-count` property to speed up deployments.
+
+Note that increasing the value of `max_parallel` may significantly impact CPU utilization on your host as your app containers - and their respective processes - start up. Setting a value higher than the number of available CPUs is discouraged. It is recommended that users carefully set this value so as not to overburden their server.
+
 ## Implemented Triggers
 
 This plugin implements various functionality through `plugn` triggers to integrate with Docker for running apps on a single server. The following functionality is supported by the `scheduler-docker-local` plugin.

docs/development/plugin-triggers.md

@@ -74,6 +74,21 @@ set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
 # TODO
 ```
 
+### `app-json-process-deploy-parallelism`
+
+- Description: Decides the parallelism to use when deploying a given process type. The default is 1 process entry at a type.
+- Invoked by: `dokku deploy`
+- Arguments: `$APP $PROCESS_TYPE`
+- Example:
+
+```shell
+#!/usr/bin/env bash
+
+set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
+
+# TODO
+```
+
 ### `app-maybe-create`
 
 - Description: Creates an app (gated by whether this is globally enabled or not)
@@ -962,7 +977,7 @@ set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
 
 - Description: Return the port for a given app container
 - Invoked by: `internally triggered by a deploy`
-- Arguments: `$APP $PROC_TYPE $CONTAINER_ID $IS_HEROKUISH_CONTAINER`
+- Arguments: `$APP $PROCESS_TYPE $CONTAINER_ID $IS_HEROKUISH_CONTAINER`
 - Example:
 
 ```shell

docs/processes/process-management.md

@@ -131,7 +131,7 @@ dokku ps:scale --skip-deploy node-js-app web=1
 
 #### Manually managing process scaling
 
-> Using a `formation` key in an `app.json` file disables the ability to use `ps:scale` for scaling.
+> Using a `formation` key in an `app.json` file with _any_ `quantity` specified disables the ability to use `ps:scale` for scaling.
 
 An `app.json` file can be committed to the root of the pushed app repository, and must be within the built image artifact in the image's working directory as shown below.
 

plugins/20_events/app-json-process-deploy-parallelism

@@ -0,0 +1 @@
+hook

plugins/20_events/scheduler-detect

@@ -0,0 +1 @@
+hook

plugins/app-json/.gitignore

@@ -6,3 +6,4 @@
 /subcommands/*
 /triggers
 /triggers/*
+/app-json-process-deploy-parallelism

plugins/app-json/Makefile

@@ -1,5 +1,5 @@
 SUBCOMMANDS = subcommands/report subcommands/set
-TRIGGERS = triggers/install triggers/post-delete triggers/post-deploy triggers/pre-deploy triggers/report
+TRIGGERS = triggers/app-json-process-deploy-parallelism triggers/install triggers/post-delete triggers/post-deploy triggers/pre-deploy triggers/report
 BUILD = commands subcommands triggers
 PLUGIN_NAME = app-json
 

plugins/app-json/appjson.go

@@ -39,7 +39,8 @@ type CronCommand struct {
 
 // Formation is a struct that represents the scale for a process from an app.json file
 type Formation struct {
-	Quantity int `json:"quantity"`
+	Quantity    *int `json:"quantity"`
+	MaxParallel *int `json:"max_parallel"`
 }
 
 // GetAppjsonDirectory returns the directory containing a given app's extracted app.json file

plugins/app-json/functions.go

@@ -500,7 +500,9 @@ func setScale(appName string, image string) error {
 	clearExisting := false
 	args := []string{appName, strconv.FormatBool(skipDeploy), strconv.FormatBool(clearExisting)}
 	for processType, formation := range appJSON.Formation {
-		args = append(args, fmt.Sprintf("%s=%d", processType, formation.Quantity))
+		if formation.Quantity != nil {
+			args = append(args, fmt.Sprintf("%s=%d", processType, *formation.Quantity))
+		}
 	}
 
 	if len(args) == 3 {
@@ -567,7 +569,7 @@ func injectDokkuScale(appName string, image string) error {
 		}
 
 		appJSON.Formation[processType] = Formation{
-			Quantity: quantity,
+			Quantity: &quantity,
 		}
 	}
 

plugins/app-json/src/triggers/triggers.go

@@ -18,6 +18,10 @@ func main() {
 
 	var err error
 	switch trigger {
+	case "app-json-process-deploy-parallelism":
+		appName := flag.Arg(0)
+		processType := flag.Arg(1)
+		err = appjson.TriggerAppJSONProcessDeployParallelism(appName, processType)
 	case "install":
 		err = appjson.TriggerInstall()
 	case "post-delete":

plugins/app-json/triggers.go

@@ -8,6 +8,32 @@ import (
 	"github.com/dokku/dokku/plugins/common"
 )
 
+// TriggerAppJSONProcessDeployParallelism returns the max number of processes to deploy in parallel
+func TriggerAppJSONProcessDeployParallelism(appName string, processType string) error {
+	appJSON, err := getAppJSON(appName)
+	if err != nil {
+		return err
+	}
+
+	parallelism := 1
+	for procType, formation := range appJSON.Formation {
+		if procType != processType {
+			continue
+		}
+
+		if formation.MaxParallel == nil {
+			continue
+		}
+
+		if *formation.MaxParallel > 0 {
+			parallelism = *formation.MaxParallel
+		}
+	}
+
+	fmt.Println(parallelism)
+	return nil
+}
+
 // TriggerInstall initializes app restart policies
 func TriggerInstall() error {
 	if err := common.PropertySetup("app-json"); err != nil {

plugins/scheduler-docker-local/bin/scheduler-deploy-process

@@ -27,11 +27,17 @@ main() {
     done
   fi
 
+  local PROCESS_TMP_FILE=$(mktemp "/tmp/dokku-${DOKKU_PID}-${FUNCNAME[0]}.XXXXXX")
+  trap "rm -rf '$PROCESS_TMP_FILE' >/dev/null" RETURN INT TERM
+
   while [[ $CONTAINER_INDEX -le $PROC_COUNT ]]; do
-    export DOKKU_CHECKS_DISABLED="$DOKKU_CHECKS_DISABLED"
+    echo "$PLUGIN_AVAILABLE_PATH/scheduler-docker-local/bin/scheduler-deploy-process-container $APP $IMAGE_SOURCE_TYPE $IMAGE $IMAGE_TAG $PROC_TYPE $PROC_COUNT $CONTAINER_INDEX" >>"$PROCESS_TMP_FILE"
-    "$PLUGIN_AVAILABLE_PATH/scheduler-docker-local/bin/scheduler-deploy-process-container" "$APP" "$IMAGE_SOURCE_TYPE" "$IMAGE" "$IMAGE_TAG" "$PROC_TYPE" "$PROC_COUNT" "$CONTAINER_INDEX"
     CONTAINER_INDEX=$((CONTAINER_INDEX + 1))
   done
+
+  PARALLEL_DEPLOY_COUNT="$(plugn trigger "app-json-process-deploy-parallelism" "$APP" "$PROC_TYPE")"
+  DOKKU_CHECKS_DISABLED="$DOKKU_CHECKS_DISABLED" parallel --will-cite --halt soon,fail=1 --jobs "$PARALLEL_DEPLOY_COUNT" --ungroup <"$PROCESS_TMP_FILE"
+
   # cleanup when we scale down
   if [[ "$PROC_COUNT" == 0 ]]; then
     local CONTAINER_IDX_OFFSET=0