Handle crashing containers by using restart=on-failure policy

Previously a crashed container would stay down, regardless of exit status. In some cases, it may be useful to restart the container. For example, an application may not be correctly implementing their error handling, or the crash may be caused by a transient error.

By setting the restart policy to `on-failure:N` - where N is a number of max restarts - we can help developers guard against crashing applications. Note that this is not a replacement for proper error handling, nor does this include notifications to a developer when a container is restarted. Those patterns should be implemented application side, or via a feature request to docker.

The value is configurable at the app-level by setting DOKKU_RESTART_LIMIT to a number. By default, containers will be restarted a max of 10 times. If a container crashes during the check-deploy plugin trigger, then the deploy will be marked as a failure.

Closes #216
Closes #398
Closes #1327

plugins/checks/check-deploy

@@ -97,6 +97,12 @@ if [[ ! -s "${TMPDIR}/CHECKS" ]] || [[ "$DOKKU_APP_CONTAINER_TYPE" != "web" ]];
   sleep $DOKKU_DEFAULT_CHECKS_WAIT
 
   ! (is_container_running $DOKKU_APP_CONTAINER_ID) && dokku_log_fail "App container failed to start!!"
+  container_restarts=$(docker inspect -f "{{ .RestartCount }}" $DOKKU_APP_CONTAINER_ID)
+  if [[ $container_restarts -ne 0 ]]; then
+    docker stop "$DOKKU_APP_CONTAINER_ID" || true
+    dokku_log_fail "App container failed to start!!"
+  fi
+
   dokku_log_info1 "Default container check successful!" && exit 0
 fi