feat: resolve nginx upstreams via coredns-docker
Stops nginx from routing to stale container IPs after host reboots, Docker upgrades without live-restore, or runtime container restarts. The bundled `nginx.conf.sigil` can use an nginx `resolver` directive against the local `coredns-docker` socket and a variable-based `proxy_pass` keyed on `<app>.<process-type>.docker`, so the upstream is resolved at request time honoring DNS TTL. `coredns-docker` becomes a hard `Depends:` of the dokku deb and is bundled in the Docker image under runit. Closes #8313.
This commit is contained in:
16 files changed
+419
-11
No files matched your search
@@ -7,6 +7,22 @@
|
||||
- The `--buildpacks-stack`, `--buildpacks-global-stack`, and `--buildpacks-computed-stack` report flags (and the corresponding `stack`, `global-stack`, `computed-stack` JSON keys) have been removed from `buildpacks:report`. Use `builder-herokuish:report` and `builder-pack:report` instead. The `buildpacks:report` command continues to report the buildpacks `list`.
|
||||
- The internal `buildpack-stack-name` plugin trigger has been removed. Each builder now reads its own `stack` property directly during the build.
|
||||
- Changing a builder's `stack` now clears that builder's build cache so the next deploy starts fresh. For herokuish this removes the `cache-<app>` Docker volume; for pack the next `pack build` runs with `--clear-cache`.
|
||||
- The `dokku` Debian package now declares `coredns-docker (>= 0.6.0)` as a hard `Depends:`. `apt install dokku` and `apt upgrade dokku` pull it in automatically. The deb ships a default `/etc/coredns/Corefile` that binds `127.0.0.1:1053`, serves the `docker.` zone, and synthesizes `<app>.<process-type>.docker` and `<app>.docker` `A` records from the `com.dokku.app-name` and `com.dokku.process-type` labels Dokku already stamps on every deployed container. The Docker image of Dokku ships coredns-docker preinstalled and runs it under runit on the same port.
|
||||
- The bundled `nginx.conf.sigil` now uses an nginx `resolver` directive and a variable-based `proxy_pass` against `<app>.<process-type>.docker` instead of pinning the upstream to a literal IP. This is what addresses the long-standing post-reboot and post-`apt-upgrade-docker` stale-vhost bug ([#8313](https://github.com/dokku/dokku/issues/8313)) - the rendered upstream re-resolves at request time via coredns-docker, so a host reboot or single-container restart no longer leaves nginx routing to a stale IP. See the [nginx documentation](/docs/networking/proxies/nginx.md#upstream-resolution-via-coredns-docker) for details.
|
||||
- Two new global nginx properties control the resolver path:
|
||||
|
||||
- `dns-resolver` (default: `127.0.0.1:1053`) - the address nginx talks to.
|
||||
- `dns-zone` (default: `docker`) - the suffix appended to `<app>.<process-type>` when constructing the upstream hostname.
|
||||
|
||||
Both can be set via `dokku nginx:set --global <property> <value>`.
|
||||
- The literal string `off` is the sentinel for disabling the resolver path. **Setting `dns-resolver` to the empty string does not disable it** - that deletes the property file and the new default (`127.0.0.1:1053`) re-engages. To revert to the legacy IP-based upstream block on hosts where coredns-docker cannot run (source/git installs, locally held coredns-docker package, postinst failures):
|
||||
|
||||
```shell
|
||||
dokku nginx:set --global dns-resolver off
|
||||
dokku proxy:build-config --all
|
||||
```
|
||||
|
||||
- Vendored `nginx.conf.sigil` templates are unaffected by the change. The existing `DOKKU_APP_<PROC>_LISTENERS` variable continues to be populated with the IP-based listener list exactly as before, so templates that only reference it continue to render IP-based upstreams regardless of the new property defaults. To opt a vendored template into the resolver path, switch its `proxy_pass` line to use the new `DOKKU_APP_<PROC>_LISTENER_HOST`, `NGINX_DNS_RESOLVER`, and `NGINX_DNS_ZONE` sigil variables - see the [nginx.conf.sigil reference](/docs/appendices/file-formats/nginx-conf-sigil.md#upstream-rendering-ip-based-vs-dns-based) for the exact snippet.
|
||||
|
||||
### Stack property migrated to builder plugins
|
||||
|
||||
|
||||
@@ -4,7 +4,10 @@ The `nginx.conf.sigil` file is used to configure the nginx server for an applica
|
||||
|
||||
## Validation
|
||||
|
||||
A custom `nginx.conf.sigil` is pre-validated at the start of every deploy, immediately after it is extracted from the source tree and before the build phase runs. Pre-validation renders the template via sigil with the same parameters used at deploy time, wraps the rendered config in a minimal `events`/`http` scaffold, and runs `nginx -t` against the result. The deploy is aborted if either the sigil render or the `nginx -t` check fails, so build work is not wasted on a syntactically invalid template. When no app listeners exist yet (typical for first deploys), pre-validation injects a placeholder `127.0.0.1:5000` listener for `DOKKU_APP_WEB_LISTENERS` so that the rendered upstream block has a static server entry and `nginx -t` does not bail out on "host not found in upstream".
|
||||
A custom `nginx.conf.sigil` is pre-validated at the start of every deploy, immediately after it is extracted from the source tree and before the build phase runs. Pre-validation renders the template via sigil with the same parameters used at deploy time, wraps the rendered config in a minimal `events`/`http` scaffold, and runs `nginx -t` against the result. The deploy is aborted if either the sigil render or the `nginx -t` check fails, so build work is not wasted on a syntactically invalid template. When no app listeners exist yet (typical for first deploys), pre-validation injects placeholders so the rendered config parses cleanly under `nginx -t`:
|
||||
|
||||
- `DOKKU_APP_WEB_LISTENERS` is set to `127.0.0.1:5000` so the legacy `upstream` block has a static server entry.
|
||||
- `DOKKU_APP_WEB_LISTENER_HOST` is set to `127.0.0.1` so the resolver-based `set $dokku_upstream "...";` line renders to a syntactically valid value.
|
||||
|
||||
Pre-validation is skipped when the proxy type is not `nginx` or when `disable-custom-config` is set to `true` for the app.
|
||||
|
||||
@@ -68,3 +71,49 @@ listen {{ if $.NGINX_BIND_ADDRESS_IP4 }}{{ $.NGINX_BIND_ADDRESS_IP4 }}:{{en
|
||||
```
|
||||
|
||||
Dokku logs a deprecation warning during deploys when a custom template still uses the `listen ... http2` form, so the offending template can be located via the deploy output.
|
||||
|
||||
## Upstream Rendering: IP-based vs DNS-based
|
||||
|
||||
> [!IMPORTANT]
|
||||
> New as of 0.39.0
|
||||
|
||||
The bundled `nginx.conf.sigil` supports two rendering paths for the upstream, and both sets of sigil variables are populated on every render. Vendored templates can use either path independently of the global `dns-resolver` setting.
|
||||
|
||||
### IP-based upstream (legacy path)
|
||||
|
||||
The classic form uses `DOKKU_APP_<PROC>_LISTENERS`: a space-separated `<ip>:<port>` list, one entry per scaled instance of the process type. The IPs come from `/home/dokku/<APP>/IP.<proc>.<index>` and the ports from `/home/dokku/<APP>/PORT.<proc>.<index>`. A typical use is an `upstream` block with one `server <ip>:<port>;` per instance:
|
||||
|
||||
```nginx
|
||||
upstream {{ $.APP }}-{{ $upstream_port }} {
|
||||
{{ range $listeners := $.DOKKU_APP_WEB_LISTENERS | split " " }}
|
||||
{{ $listener_list := $listeners | split ":" }}
|
||||
{{ $listener_ip := index $listener_list 0 }}
|
||||
server {{ $listener_ip }}:{{ $upstream_port }};{{ end }}
|
||||
}
|
||||
```
|
||||
|
||||
This path renders the IP that Dokku saw when the config was last built. After a host reboot, an `apt upgrade docker` without live-restore, or a single container restart, those IPs can become stale until `dokku proxy:build-config` runs again. The DNS-based path below addresses that staleness.
|
||||
|
||||
### DNS-based upstream (resolver path)
|
||||
|
||||
The new variables (`DOKKU_APP_<PROC>_LISTENER_HOST` + `NGINX_DNS_RESOLVER` + `NGINX_DNS_ZONE`) describe a single hostname per process type of the form `<app>.<proctype>.<dns-zone>` (for example, `myapp.web.docker`). The hostname is served by [coredns-docker](https://github.com/dokku/coredns-docker) (>= `0.6.0`), which watches Docker events and collapses every running instance of the process type onto one multi-A record set. Combined with an nginx `resolver` directive and a variable-based `proxy_pass`, the upstream re-resolves at request time honoring DNS TTL - which gives runtime IP refresh on container restarts and DNS round-robin across scaled instances.
|
||||
|
||||
The bundled template emits this snippet:
|
||||
|
||||
```nginx
|
||||
{{ if ne $.NGINX_DNS_RESOLVER "off" }}
|
||||
resolver {{ $.NGINX_DNS_RESOLVER }} valid=10s ipv6=off;
|
||||
set $dokku_upstream "{{ $.DOKKU_APP_WEB_LISTENER_HOST }}:{{ $upstream_port }}";
|
||||
proxy_pass http://$dokku_upstream;
|
||||
{{ else }}
|
||||
proxy_pass http://{{ $.APP }}-{{ $upstream_port }};
|
||||
{{ end }}
|
||||
```
|
||||
|
||||
The `$dokku_upstream` variable name is intentionally namespaced (rather than `$upstream`) so vendored templates that already define `$upstream` will not collide if they adopt the resolver branch.
|
||||
|
||||
`NGINX_DNS_RESOLVER` defaults to `127.0.0.1:1053` and `NGINX_DNS_ZONE` defaults to `docker`; both are configurable via `dokku nginx:set --global dns-resolver ...` and `dokku nginx:set --global dns-zone ...`. The literal string `off` is the sentinel for disabling the resolver branch in the bundled template - empty strings are treated as "use the default" because the property is stored on disk and an empty value deletes the property file.
|
||||
|
||||
### Choosing between the two
|
||||
|
||||
The DNS-based path is the default for the bundled template. Vendored templates that have not been updated continue to use `DOKKU_APP_<PROC>_LISTENERS` and are unaffected by the global default. The DNS-based variables are populated regardless of whether `dns-resolver` is `off` or not, so a vendored template can adopt the resolver path on its own schedule by switching its rendering snippet, even if other apps on the host are still using IP-based upstreams.
|
||||
Reference in new issue
Block a user