feat: resolve nginx upstreams via coredns-docker

Stops nginx from routing to stale container IPs after host reboots, Docker upgrades without live-restore, or runtime container restarts. The bundled `nginx.conf.sigil` can use an nginx `resolver` directive against the local `coredns-docker` socket and a variable-based `proxy_pass` keyed on `<app>.<process-type>.docker`, so the upstream is resolved at request time honoring DNS TTL. `coredns-docker` becomes a hard `Depends:` of the dokku deb and is bundled in the Docker image under runit. Closes #8313.
This commit is contained in:
Jose Diaz-Gonzalez committed 2026-11-06 20:43:16 -05:00
1 parent bcd563c2db
commit 5241fd4be2
16 files changed
+419 -11

No files matched your search

+16
View File
@@ -7,6 +7,22 @@
- The `--buildpacks-stack`, `--buildpacks-global-stack`, and `--buildpacks-computed-stack` report flags (and the corresponding `stack`, `global-stack`, `computed-stack` JSON keys) have been removed from `buildpacks:report`. Use `builder-herokuish:report` and `builder-pack:report` instead. The `buildpacks:report` command continues to report the buildpacks `list`.
- The internal `buildpack-stack-name` plugin trigger has been removed. Each builder now reads its own `stack` property directly during the build.
- Changing a builder's `stack` now clears that builder's build cache so the next deploy starts fresh. For herokuish this removes the `cache-<app>` Docker volume; for pack the next `pack build` runs with `--clear-cache`.
- The `dokku` Debian package now declares `coredns-docker (>= 0.6.0)` as a hard `Depends:`. `apt install dokku` and `apt upgrade dokku` pull it in automatically. The deb ships a default `/etc/coredns/Corefile` that binds `127.0.0.1:1053`, serves the `docker.` zone, and synthesizes `<app>.<process-type>.docker` and `<app>.docker` `A` records from the `com.dokku.app-name` and `com.dokku.process-type` labels Dokku already stamps on every deployed container. The Docker image of Dokku ships coredns-docker preinstalled and runs it under runit on the same port.
- The bundled `nginx.conf.sigil` now uses an nginx `resolver` directive and a variable-based `proxy_pass` against `<app>.<process-type>.docker` instead of pinning the upstream to a literal IP. This is what addresses the long-standing post-reboot and post-`apt-upgrade-docker` stale-vhost bug ([#8313](https://github.com/dokku/dokku/issues/8313)) - the rendered upstream re-resolves at request time via coredns-docker, so a host reboot or single-container restart no longer leaves nginx routing to a stale IP. See the [nginx documentation](/docs/networking/proxies/nginx.md#upstream-resolution-via-coredns-docker) for details.
- Two new global nginx properties control the resolver path:
- `dns-resolver` (default: `127.0.0.1:1053`) - the address nginx talks to.
- `dns-zone` (default: `docker`) - the suffix appended to `<app>.<process-type>` when constructing the upstream hostname.
Both can be set via `dokku nginx:set --global <property> <value>`.
- The literal string `off` is the sentinel for disabling the resolver path. **Setting `dns-resolver` to the empty string does not disable it** - that deletes the property file and the new default (`127.0.0.1:1053`) re-engages. To revert to the legacy IP-based upstream block on hosts where coredns-docker cannot run (source/git installs, locally held coredns-docker package, postinst failures):
```shell
dokku nginx:set --global dns-resolver off
dokku proxy:build-config --all
```
- Vendored `nginx.conf.sigil` templates are unaffected by the change. The existing `DOKKU_APP_<PROC>_LISTENERS` variable continues to be populated with the IP-based listener list exactly as before, so templates that only reference it continue to render IP-based upstreams regardless of the new property defaults. To opt a vendored template into the resolver path, switch its `proxy_pass` line to use the new `DOKKU_APP_<PROC>_LISTENER_HOST`, `NGINX_DNS_RESOLVER`, and `NGINX_DNS_ZONE` sigil variables - see the [nginx.conf.sigil reference](/docs/appendices/file-formats/nginx-conf-sigil.md#upstream-rendering-ip-based-vs-dns-based) for the exact snippet.
### Stack property migrated to builder plugins
@@ -4,7 +4,10 @@ The `nginx.conf.sigil` file is used to configure the nginx server for an applica
## Validation
A custom `nginx.conf.sigil` is pre-validated at the start of every deploy, immediately after it is extracted from the source tree and before the build phase runs. Pre-validation renders the template via sigil with the same parameters used at deploy time, wraps the rendered config in a minimal `events`/`http` scaffold, and runs `nginx -t` against the result. The deploy is aborted if either the sigil render or the `nginx -t` check fails, so build work is not wasted on a syntactically invalid template. When no app listeners exist yet (typical for first deploys), pre-validation injects a placeholder `127.0.0.1:5000` listener for `DOKKU_APP_WEB_LISTENERS` so that the rendered upstream block has a static server entry and `nginx -t` does not bail out on "host not found in upstream".
A custom `nginx.conf.sigil` is pre-validated at the start of every deploy, immediately after it is extracted from the source tree and before the build phase runs. Pre-validation renders the template via sigil with the same parameters used at deploy time, wraps the rendered config in a minimal `events`/`http` scaffold, and runs `nginx -t` against the result. The deploy is aborted if either the sigil render or the `nginx -t` check fails, so build work is not wasted on a syntactically invalid template. When no app listeners exist yet (typical for first deploys), pre-validation injects placeholders so the rendered config parses cleanly under `nginx -t`:
- `DOKKU_APP_WEB_LISTENERS` is set to `127.0.0.1:5000` so the legacy `upstream` block has a static server entry.
- `DOKKU_APP_WEB_LISTENER_HOST` is set to `127.0.0.1` so the resolver-based `set $dokku_upstream "...";` line renders to a syntactically valid value.
Pre-validation is skipped when the proxy type is not `nginx` or when `disable-custom-config` is set to `true` for the app.
@@ -68,3 +71,49 @@ listen {{ if $.NGINX_BIND_ADDRESS_IP4 }}{{ $.NGINX_BIND_ADDRESS_IP4 }}:{{en
```
Dokku logs a deprecation warning during deploys when a custom template still uses the `listen ... http2` form, so the offending template can be located via the deploy output.
## Upstream Rendering: IP-based vs DNS-based
> [!IMPORTANT]
> New as of 0.39.0
The bundled `nginx.conf.sigil` supports two rendering paths for the upstream, and both sets of sigil variables are populated on every render. Vendored templates can use either path independently of the global `dns-resolver` setting.
### IP-based upstream (legacy path)
The classic form uses `DOKKU_APP_<PROC>_LISTENERS`: a space-separated `<ip>:<port>` list, one entry per scaled instance of the process type. The IPs come from `/home/dokku/<APP>/IP.<proc>.<index>` and the ports from `/home/dokku/<APP>/PORT.<proc>.<index>`. A typical use is an `upstream` block with one `server <ip>:<port>;` per instance:
```nginx
upstream {{ $.APP }}-{{ $upstream_port }} {
{{ range $listeners := $.DOKKU_APP_WEB_LISTENERS | split " " }}
{{ $listener_list := $listeners | split ":" }}
{{ $listener_ip := index $listener_list 0 }}
server {{ $listener_ip }}:{{ $upstream_port }};{{ end }}
}
```
This path renders the IP that Dokku saw when the config was last built. After a host reboot, an `apt upgrade docker` without live-restore, or a single container restart, those IPs can become stale until `dokku proxy:build-config` runs again. The DNS-based path below addresses that staleness.
### DNS-based upstream (resolver path)
The new variables (`DOKKU_APP_<PROC>_LISTENER_HOST` + `NGINX_DNS_RESOLVER` + `NGINX_DNS_ZONE`) describe a single hostname per process type of the form `<app>.<proctype>.<dns-zone>` (for example, `myapp.web.docker`). The hostname is served by [coredns-docker](https://github.com/dokku/coredns-docker) (>= `0.6.0`), which watches Docker events and collapses every running instance of the process type onto one multi-A record set. Combined with an nginx `resolver` directive and a variable-based `proxy_pass`, the upstream re-resolves at request time honoring DNS TTL - which gives runtime IP refresh on container restarts and DNS round-robin across scaled instances.
The bundled template emits this snippet:
```nginx
{{ if ne $.NGINX_DNS_RESOLVER "off" }}
resolver {{ $.NGINX_DNS_RESOLVER }} valid=10s ipv6=off;
set $dokku_upstream "{{ $.DOKKU_APP_WEB_LISTENER_HOST }}:{{ $upstream_port }}";
proxy_pass http://$dokku_upstream;
{{ else }}
proxy_pass http://{{ $.APP }}-{{ $upstream_port }};
{{ end }}
```
The `$dokku_upstream` variable name is intentionally namespaced (rather than `$upstream`) so vendored templates that already define `$upstream` will not collide if they adopt the resolver branch.
`NGINX_DNS_RESOLVER` defaults to `127.0.0.1:1053` and `NGINX_DNS_ZONE` defaults to `docker`; both are configurable via `dokku nginx:set --global dns-resolver ...` and `dokku nginx:set --global dns-zone ...`. The literal string `off` is the sentinel for disabling the resolver branch in the bundled template - empty strings are treated as "use the default" because the property is stored on disk and an empty value deletes the property file.
### Choosing between the two
The DNS-based path is the default for the bundled template. Vendored templates that have not been updated continue to use `DOKKU_APP_<PROC>_LISTENERS` and are unaffected by the global default. The DNS-based variables are populated regardless of whether `dns-resolver` is `off` or not, so a vendored template can adopt the resolver path on its own schedule by switching its rendering snippet, even if other apps on the host are still using IP-based upstreams.
+1
View File
@@ -88,6 +88,7 @@ Dokku is run in the following configuration:
- Image build cache is set to the data dir + `/home/dokku`.
- The docker socket is mounted into container.
- `DOKKU_HOST_ROOT` and `DOKKU_LIB_HOST_ROOT` tell Dokku how the host Docker daemon sees those same trees when creating bind mounts (app repos and plugin data such as named storage).
- `coredns-docker` (a CoreDNS plugin that serves DNS records for sibling Docker containers) is installed transitively via the dokku deb's `Depends:`, runs under runit on `127.0.0.1:1053` inside the container, and is what the `nginx-vhosts` `dns-resolver` default points at. This is what makes the `<app>.<process-type>.docker` upstream resolution work out of the box - see the [nginx proxy doc](/docs/networking/proxies/nginx.md#upstream-resolution-via-coredns-docker) for the operator-facing behavior.
Application repositories, plugin config, as well as plugin data are persisted to disk within the specified host directory for `/var/lib/dokku`. `DOKKU_LIB_HOST_ROOT` must point at that host path's `var/lib/dokku` subtree so Docker `-v` mounts resolve under the data volume rather than under the host's own `/var/lib/dokku`.
+28
View File
@@ -28,6 +28,34 @@ Nginx will proxy the requests in a [round-robin balancing fashion](http://nginx.
> [!NOTE]
> Due to how the plugin is implemented, if an app successfully starts up `web` containers but fails to deploy some other containers, nginx may eventually stop routing requests. Users should revert their code in these cases, or manually trigger `dokku proxy:build-config $APP` in order to ensure requests route to the new web containers.
### Upstream Resolution via coredns-docker
> [!IMPORTANT]
> New as of 0.39.0
Dokku resolves nginx upstreams through [coredns-docker](https://github.com/dokku/coredns-docker), a CoreDNS plugin that watches the Docker daemon and serves fresh `A` records keyed off container labels. The nginx `resolver` directive points at coredns-docker's local socket (`127.0.0.1:1053`) and the rendered upstream is a variable-based `proxy_pass`, which forces nginx to re-resolve at request time honoring DNS TTL. This is how nginx avoids serving the wrong vhost after a host reboot, after `apt upgrade docker`, or after a single container restarts mid-runtime: the on-disk nginx config no longer pins a stale IP.
`coredns-docker` (>= `0.6.0`) is installed automatically as a transitive `Depends:` of the `dokku` Debian package, so `apt install dokku` and `apt upgrade dokku` pull it in without operator action. The shipped `/etc/coredns/Corefile` already binds `127.0.0.1:1053`, serves the `docker.` zone, and synthesizes `<app>.<process-type>.docker` and `<app>.docker` `A` records from Dokku's existing `com.dokku.app-name` and `com.dokku.process-type` container labels. Multiple containers for the same process type collapse onto one multi-A record set, so scaled apps load-balance across instances via DNS without any per-container configuration. The bundled Docker image of Dokku ships coredns-docker preinstalled and runs it under runit on the same port.
No `systemd-resolved` or `/etc/resolv.conf` change is required - nginx talks to `coredns-docker` directly via the `resolver` directive in each app's `nginx.conf`.
The behavior is controlled by two global nginx properties:
- `dns-resolver` (default: `127.0.0.1:1053`) - the resolver address nginx talks to.
- `dns-zone` (default: `docker`) - the suffix appended to `<app>.<process-type>` when constructing the upstream hostname.
To revert to the legacy IP-based upstream block (for instance on a host where `coredns-docker` cannot be installed):
```shell
dokku nginx:set --global dns-resolver off
dokku proxy:build-config --all
```
> [!NOTE]
> The sentinel for disabling the feature is the literal string `off`. Setting `dns-resolver` to the empty string (`""`) does **not** disable it - that deletes the property and the default (`127.0.0.1:1053`) re-engages. Use `off` to disable, the default to re-enable.
See [the `nginx.conf.sigil` reference](/docs/appendices/file-formats/nginx-conf-sigil.md) for the sigil variables that drive the resolver-based render, and how vendored templates can adopt or skip it.
### Nginx Configuration for Undeployed Apps
> [!IMPORTANT]