github/dokku
0
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

feat: add dns-resolver-timeout property defaulting to 5s

Browse Source 
Adds a new `dns-resolver-timeout` global nginx property that controls how long nginx waits on the configured DNS resolver before giving up and returning 502. Without an explicit timeout nginx blocks on the resolver for its built-in 30s default, which means clients see hung requests instead of a fast 502 when coredns-docker is unreachable. The 5s default keeps failure-mode requests short while leaving headroom over a healthy local resolver round trip. The bundled `nginx.conf.sigil` emits `resolver_timeout {{ $.NGINX_DNS_RESOLVER_TIMEOUT }}` next to the existing `resolver` directive, and the outage-mode bats and docker-deploy tests now expect the curl call to complete inside that window.
This commit is contained in:
Jose Diaz-Gonzalez
2026-05-13 12:01:21 -04:00
parent 1d2854fa93
commit b5315de49b
13 changed files with 73 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
docs/appendices/0.39.0-migration-guide.md
View File

@@ -9,12 +9,13 @@
- Changing a builder's `stack` now clears that builder's build cache so the next deploy starts fresh. For herokuish this removes the `cache-<app>` Docker volume; for pack the next `pack build` runs with `--clear-cache`.
- The `dokku` Debian package now declares `coredns-docker (>= 0.6.0)` as a hard `Depends:`. `apt install dokku` and `apt upgrade dokku` pull it in automatically. The deb ships a default `/etc/coredns/Corefile` that binds `127.0.0.1:1053`, serves the `docker.` zone, and synthesizes `<app>.<process-type>.docker` and `<app>.docker` `A` records from the `com.dokku.app-name` and `com.dokku.process-type` labels Dokku already stamps on every deployed container. The Docker image of Dokku ships coredns-docker preinstalled and runs it under runit on the same port.
- The bundled `nginx.conf.sigil` now uses an nginx `resolver` directive and a variable-based `proxy_pass` against `<app>.<process-type>.docker` instead of pinning the upstream to a literal IP. This is what addresses the long-standing post-reboot and post-`apt-upgrade-docker` stale-vhost bug ([#8313](https://github.com/dokku/dokku/issues/8313)) - the rendered upstream re-resolves at request time via coredns-docker, so a host reboot or single-container restart no longer leaves nginx routing to a stale IP. See the [nginx documentation](/docs/networking/proxies/nginx.md#upstream-resolution-via-coredns-docker) for details.
- Two new global nginx properties control the resolver path:
- Three new global nginx properties control the resolver path:
- `dns-resolver` (default: `127.0.0.1:1053`) - the address nginx talks to.
- `dns-resolver-timeout` (default: `5s`) - how long nginx waits for a DNS response before giving up and returning 502. The short default keeps clients from waiting on nginx's built-in 30s timeout when coredns-docker is unreachable.
- `dns-zone` (default: `docker`) - the suffix appended to `<app>.<process-type>` when constructing the upstream hostname.
Both can be set via `dokku nginx:set --global <property> <value>`.
All three can be set via `dokku nginx:set --global <property> <value>`.
- The literal string `off` is the sentinel for disabling the resolver path. **Setting `dns-resolver` to the empty string does not disable it** - that deletes the property file and the new default (`127.0.0.1:1053`) re-engages. To revert to the legacy IP-based upstream block on hosts where coredns-docker cannot run (source/git installs, locally held coredns-docker package, postinst failures):
```shell
@@ -22,7 +23,7 @@
dokku proxy:build-config --all
```
- Vendored `nginx.conf.sigil` templates are unaffected by the change. The existing `DOKKU_APP_<PROC>_LISTENERS` variable continues to be populated with the IP-based listener list exactly as before, so templates that only reference it continue to render IP-based upstreams regardless of the new property defaults. To opt a vendored template into the resolver path, switch its `proxy_pass` line to use the new `DOKKU_APP_<PROC>_LISTENER_HOST`, `NGINX_DNS_RESOLVER`, and `NGINX_DNS_ZONE` sigil variables - see the [nginx.conf.sigil reference](/docs/appendices/file-formats/nginx-conf-sigil.md#upstream-rendering-ip-based-vs-dns-based) for the exact snippet.
- Vendored `nginx.conf.sigil` templates are unaffected by the change. The existing `DOKKU_APP_<PROC>_LISTENERS` variable continues to be populated with the IP-based listener list exactly as before, so templates that only reference it continue to render IP-based upstreams regardless of the new property defaults. To opt a vendored template into the resolver path, switch its `proxy_pass` line to use the new `DOKKU_APP_<PROC>_LISTENER_HOST`, `NGINX_DNS_RESOLVER`, `NGINX_DNS_RESOLVER_TIMEOUT`, and `NGINX_DNS_ZONE` sigil variables - see the [nginx.conf.sigil reference](/docs/appendices/file-formats/nginx-conf-sigil.md#upstream-rendering-ip-based-vs-dns-based) for the exact snippet.
### Stack property migrated to builder plugins

5
docs/appendices/file-formats/nginx-conf-sigil.md
View File

@@ -96,13 +96,14 @@ This path renders the IP that Dokku saw when the config was last built. After a
### DNS-based upstream (resolver path)
The new variables (`DOKKU_APP_<PROC>_LISTENER_HOST` + `NGINX_DNS_RESOLVER` + `NGINX_DNS_ZONE`) describe a single hostname per process type of the form `<app>.<proctype>.<dns-zone>` (for example, `myapp.web.docker`). The hostname is served by [coredns-docker](https://github.com/dokku/coredns-docker) (>= `0.6.0`), which watches Docker events and collapses every running instance of the process type onto one multi-A record set. Combined with an nginx `resolver` directive and a variable-based `proxy_pass`, the upstream re-resolves at request time honoring DNS TTL - which gives runtime IP refresh on container restarts and DNS round-robin across scaled instances.
The new variables (`DOKKU_APP_<PROC>_LISTENER_HOST` + `NGINX_DNS_RESOLVER` + `NGINX_DNS_RESOLVER_TIMEOUT` + `NGINX_DNS_ZONE`) describe a single hostname per process type of the form `<app>.<proctype>.<dns-zone>` (for example, `myapp.web.docker`). The hostname is served by [coredns-docker](https://github.com/dokku/coredns-docker) (>= `0.6.0`), which watches Docker events and collapses every running instance of the process type onto one multi-A record set. Combined with an nginx `resolver` directive and a variable-based `proxy_pass`, the upstream re-resolves at request time honoring DNS TTL - which gives runtime IP refresh on container restarts and DNS round-robin across scaled instances.
The bundled template emits this snippet:
```nginx
{{ if ne $.NGINX_DNS_RESOLVER "off" }}
resolver {{ $.NGINX_DNS_RESOLVER }} valid=10s ipv6=off;
resolver_timeout {{ $.NGINX_DNS_RESOLVER_TIMEOUT }};
set $dokku_upstream "{{ $.DOKKU_APP_WEB_LISTENER_HOST }}:{{ $upstream_port }}";
proxy_pass http://$dokku_upstream;
{{ else }}
@@ -112,7 +113,7 @@ proxy_pass http://{{ $.APP }}-{{ $upstream_port }};
The `$dokku_upstream` variable name is intentionally namespaced (rather than `$upstream`) so vendored templates that already define `$upstream` will not collide if they adopt the resolver branch.
`NGINX_DNS_RESOLVER` defaults to `127.0.0.1:1053` and `NGINX_DNS_ZONE` defaults to `docker`; both are configurable via `dokku nginx:set --global dns-resolver ...` and `dokku nginx:set --global dns-zone ...`. The literal string `off` is the sentinel for disabling the resolver branch in the bundled template - empty strings are treated as "use the default" because the property is stored on disk and an empty value deletes the property file.
`NGINX_DNS_RESOLVER` defaults to `127.0.0.1:1053`, `NGINX_DNS_RESOLVER_TIMEOUT` defaults to `5s`, and `NGINX_DNS_ZONE` defaults to `docker`; all three are configurable via `dokku nginx:set --global dns-resolver ...`, `dokku nginx:set --global dns-resolver-timeout ...`, and `dokku nginx:set --global dns-zone ...`. The literal string `off` is the sentinel for disabling the resolver branch in the bundled template - empty strings are treated as "use the default" because the property is stored on disk and an empty value deletes the property file. The short default `dns-resolver-timeout` keeps clients from waiting on nginx's built-in 30s timeout when coredns-docker is unreachable; nginx returns 502 after the configured interval instead.
### Choosing between the two

3
docs/networking/proxies/nginx.md
View File

@@ -39,9 +39,10 @@ Dokku resolves nginx upstreams through [coredns-docker](https://github.com/dokku
No `systemd-resolved` or `/etc/resolv.conf` change is required - nginx talks to `coredns-docker` directly via the `resolver` directive in each app's `nginx.conf`.
The behavior is controlled by two global nginx properties:
The behavior is controlled by three global nginx properties:
- `dns-resolver` (default: `127.0.0.1:1053`) - the resolver address nginx talks to.
- `dns-resolver-timeout` (default: `5s`) - how long nginx waits for a DNS response before giving up and returning 502. The default is short so requests fail fast when coredns-docker is unreachable rather than hanging on the nginx built-in default (30s).
- `dns-zone` (default: `docker`) - the suffix appended to `<app>.<process-type>` when constructing the upstream hostname.
To revert to the legacy IP-based upstream block (for instance on a host where `coredns-docker` cannot be installed):

2
plugins/nginx-vhosts/functions
View File

@@ -325,6 +325,7 @@ fn-nginx-vhosts-render-template() {
local NGINX_BIND_ADDRESS_IP4="$(fn-nginx-computed-bind-address-ipv4 "$APP")"
local NGINX_BIND_ADDRESS_IP6="$(fn-nginx-computed-bind-address-ipv6 "$APP")"
local NGINX_DNS_RESOLVER="$(fn-nginx-computed-dns-resolver "$APP")"
local NGINX_DNS_RESOLVER_TIMEOUT="$(fn-nginx-computed-dns-resolver-timeout "$APP")"
local NGINX_DNS_ZONE="$(fn-nginx-computed-dns-zone "$APP")"
local NGINX_UNDERSCORE_IN_HEADERS="$(fn-nginx-computed-underscore-in-headers "$APP")"
local PROXY_X_FORWARDED_FOR="$(fn-nginx-computed-x-forwarded-for-value "$APP")"
@@ -357,6 +358,7 @@ fn-nginx-vhosts-render-template() {
NGINX_BIND_ADDRESS_IP4="$NGINX_BIND_ADDRESS_IP4"
NGINX_BIND_ADDRESS_IP6="$NGINX_BIND_ADDRESS_IP6"
NGINX_DNS_RESOLVER="$NGINX_DNS_RESOLVER"
NGINX_DNS_RESOLVER_TIMEOUT="$NGINX_DNS_RESOLVER_TIMEOUT"
NGINX_DNS_ZONE="$NGINX_DNS_ZONE"
HTTP2_DIRECTIVE_SUPPORTED="$HTTP2_DIRECTIVE_SUPPORTED"
HTTP2_PUSH_SUPPORTED="$HTTP2_PUSH_SUPPORTED"

21
plugins/nginx-vhosts/internal-functions
View File

@@ -193,6 +193,27 @@ fn-nginx-global-dns-resolver() {
fn-get-property --app "$APP" --global "dns-resolver"
}
fn-nginx-dns-resolver-timeout() {
declare desc="get the configured dns-resolver-timeout value"
declare APP="$1"
fn-get-property --app "$APP" "dns-resolver-timeout"
}
fn-nginx-computed-dns-resolver-timeout() {
declare desc="get the computed dns-resolver-timeout value"
declare APP="$1"
fn-get-property --app "$APP" --computed "dns-resolver-timeout"
}
fn-nginx-global-dns-resolver-timeout() {
declare desc="get the global dns-resolver-timeout value"
declare APP="$1"
fn-get-property --app "$APP" --global "dns-resolver-timeout"
}
fn-nginx-dns-zone() {
declare desc="get the configured dns-zone value"
declare APP="$1"

17
plugins/nginx-vhosts/nginx_vhosts.go
View File

@@ -167,6 +167,23 @@ func GlobalDnsResolver() string {
return common.PropertyGetDefault("nginx", "--global", "dns-resolver", "127.0.0.1:1053")
}
func AppDnsResolverTimeout(appName string) string {
return common.PropertyGet("nginx", appName, "dns-resolver-timeout")
}
func ComputedDnsResolverTimeout(appName string) string {
appValue := AppDnsResolverTimeout(appName)
if appValue != "" {
return appValue
}
return GlobalDnsResolverTimeout()
}
func GlobalDnsResolverTimeout() string {
return common.PropertyGetDefault("nginx", "--global", "dns-resolver-timeout", "5s")
}
func AppDnsZone(appName string) string {
return common.PropertyGet("nginx", appName, "dns-zone")
}

1
plugins/nginx-vhosts/report.go
View File

@@ -30,6 +30,7 @@ func nginxReportProps() []nginxReportProp {
{"client-max-body-size", AppClientMaxBodySize, ComputedClientMaxBodySize, wrapGlobal(GlobalClientMaxBodySize)},
{"disable-custom-config", AppDisableCustomConfig, ComputedDisableCustomConfig, wrapGlobal(GlobalDisableCustomConfig)},
{"dns-resolver", AppDnsResolver, ComputedDnsResolver, wrapGlobal(GlobalDnsResolver)},
{"dns-resolver-timeout", AppDnsResolverTimeout, ComputedDnsResolverTimeout, wrapGlobal(GlobalDnsResolverTimeout)},
{"dns-zone", AppDnsZone, ComputedDnsZone, wrapGlobal(GlobalDnsZone)},
{"error-log-path", AppErrorLogPath, ComputedErrorLogPath, wrapGlobal(GlobalErrorLogPath)},
{"hsts-include-subdomains", AppHSTSIncludeSubdomains, ComputedHSTSIncludeSubdomains, wrapGlobal(GlobalHSTSIncludeSubdomains)},

6
plugins/nginx-vhosts/src/nginx-property/nginx-property.go
View File

@@ -53,6 +53,8 @@ func appValue(appName string, property string) string {
value = nginx_vhosts.AppDisableCustomConfig(appName)
case "dns-resolver":
value = nginx_vhosts.AppDnsResolver(appName)
case "dns-resolver-timeout":
value = nginx_vhosts.AppDnsResolverTimeout(appName)
case "dns-zone":
value = nginx_vhosts.AppDnsZone(appName)
case "error-log-path":
@@ -129,6 +131,8 @@ func computedValue(appName string, property string) string {
value = nginx_vhosts.ComputedDisableCustomConfig(appName)
case "dns-resolver":
value = nginx_vhosts.ComputedDnsResolver(appName)
case "dns-resolver-timeout":
value = nginx_vhosts.ComputedDnsResolverTimeout(appName)
case "dns-zone":
value = nginx_vhosts.ComputedDnsZone(appName)
case "error-log-path":
@@ -205,6 +209,8 @@ func globalValue(appName string, property string) string {
value = nginx_vhosts.GlobalDisableCustomConfig()
case "dns-resolver":
value = nginx_vhosts.GlobalDnsResolver()
case "dns-resolver-timeout":
value = nginx_vhosts.GlobalDnsResolverTimeout()
case "dns-zone":
value = nginx_vhosts.GlobalDnsZone()
case "error-log-path":

6
plugins/nginx-vhosts/subcommands/set
View File

@@ -9,13 +9,13 @@ cmd-nginx-set() {
declare cmd="nginx:set"
[[ "$1" == "$cmd" ]] && shift 1
declare APP="$1" KEY="$2" VALUE="$3"
local VALID_KEYS=("access-log-format" "access-log-path" "bind-address-ipv4" "bind-address-ipv6" "client-body-timeout" "client-header-timeout" "client-max-body-size" "disable-custom-config" "dns-resolver" "dns-zone" "error-log-path" "hsts" "hsts-include-subdomains" "hsts-preload" "hsts-max-age" "keepalive-timeout" "lingering-timeout" "nginx-conf-sigil-path" "nginx-service-command" "proxy-buffer-size" "proxy-buffering" "proxy-buffers" "proxy-busy-buffers-size" "proxy-http-version" "proxy-connect-timeout" "proxy-read-timeout" "proxy-send-timeout" "proxy-keepalive" "send-timeout" "underscore-in-headers" "x-forwarded-for-value" "x-forwarded-port-value" "x-forwarded-proto-value" "x-forwarded-ssl")
local GLOBAL_KEYS=("access-log-format" "access-log-path" "bind-address-ipv4" "bind-address-ipv6" "client-body-timeout" "client-header-timeout" "client-max-body-size" "disable-custom-config" "dns-resolver" "dns-zone" "error-log-path" "hsts-include-subdomains" "hsts-max-age" "hsts-preload" "hsts" "keepalive-timeout" "lingering-timeout" "nginx-conf-sigil-path" "nginx-service-command" "proxy-buffer-size" "proxy-buffering" "proxy-buffers" "proxy-busy-buffers-size" "proxy-http-version" "proxy-connect-timeout" "proxy-read-timeout" "proxy-send-timeout" "proxy-keepalive" "send-timeout" "underscore-in-headers" "x-forwarded-for-value" "x-forwarded-port-value" "x-forwarded-proto-value" "x-forwarded-ssl")
local VALID_KEYS=("access-log-format" "access-log-path" "bind-address-ipv4" "bind-address-ipv6" "client-body-timeout" "client-header-timeout" "client-max-body-size" "disable-custom-config" "dns-resolver" "dns-resolver-timeout" "dns-zone" "error-log-path" "hsts" "hsts-include-subdomains" "hsts-preload" "hsts-max-age" "keepalive-timeout" "lingering-timeout" "nginx-conf-sigil-path" "nginx-service-command" "proxy-buffer-size" "proxy-buffering" "proxy-buffers" "proxy-busy-buffers-size" "proxy-http-version" "proxy-connect-timeout" "proxy-read-timeout" "proxy-send-timeout" "proxy-keepalive" "send-timeout" "underscore-in-headers" "x-forwarded-for-value" "x-forwarded-port-value" "x-forwarded-proto-value" "x-forwarded-ssl")
local GLOBAL_KEYS=("access-log-format" "access-log-path" "bind-address-ipv4" "bind-address-ipv6" "client-body-timeout" "client-header-timeout" "client-max-body-size" "disable-custom-config" "dns-resolver" "dns-resolver-timeout" "dns-zone" "error-log-path" "hsts-include-subdomains" "hsts-max-age" "hsts-preload" "hsts" "keepalive-timeout" "lingering-timeout" "nginx-conf-sigil-path" "nginx-service-command" "proxy-buffer-size" "proxy-buffering" "proxy-buffers" "proxy-busy-buffers-size" "proxy-http-version" "proxy-connect-timeout" "proxy-read-timeout" "proxy-send-timeout" "proxy-keepalive" "send-timeout" "underscore-in-headers" "x-forwarded-for-value" "x-forwarded-port-value" "x-forwarded-proto-value" "x-forwarded-ssl")
[[ -z "$KEY" ]] && dokku_log_fail "No key specified"
if ! fn-in-array "$KEY" "${VALID_KEYS[@]}"; then
dokku_log_fail "Invalid key specified, valid keys include: access-log-format, access-log-path, bind-address-ipv4, bind-address-ipv6, client-body-timeout, client-header-timeout, client-max-body-size, disable-custom-config, dns-resolver, dns-zone, error-log-path, hsts, hsts-include-subdomains, hsts-preload, hsts-max-age, keepalive-timeout, lingering-timeout, nginx-conf-sigil-path, nginx-service-command, proxy-buffer-size, proxy-buffering, proxy-buffers, proxy-busy-buffers-size, proxy-http-version, proxy-connect-timeout, proxy-read-timeout, proxy-send-timeout, proxy-keepalive, send-timeout, underscore-in-headers, x-forwarded-for-value, x-forwarded-port-value, x-forwarded-proto-value, x-forwarded-ssl"
dokku_log_fail "Invalid key specified, valid keys include: access-log-format, access-log-path, bind-address-ipv4, bind-address-ipv6, client-body-timeout, client-header-timeout, client-max-body-size, disable-custom-config, dns-resolver, dns-resolver-timeout, dns-zone, error-log-path, hsts, hsts-include-subdomains, hsts-preload, hsts-max-age, keepalive-timeout, lingering-timeout, nginx-conf-sigil-path, nginx-service-command, proxy-buffer-size, proxy-buffering, proxy-buffers, proxy-busy-buffers-size, proxy-http-version, proxy-connect-timeout, proxy-read-timeout, proxy-send-timeout, proxy-keepalive, send-timeout, underscore-in-headers, x-forwarded-for-value, x-forwarded-port-value, x-forwarded-proto-value, x-forwarded-ssl"
fi
if ! fn-in-array "$KEY" "${GLOBAL_KEYS[@]}" && [[ "$APP" == "--global" ]]; then

10
plugins/nginx-vhosts/template_test.go
View File

@@ -39,6 +39,7 @@ func defaultVars() map[string]interface{} {
"NGINX_BIND_ADDRESS_IP4": "",
"NGINX_BIND_ADDRESS_IP6": "::",
"NGINX_DNS_RESOLVER": "127.0.0.1:1053",
"NGINX_DNS_RESOLVER_TIMEOUT": "5s",
"NGINX_DNS_ZONE": "docker",
"NGINX_ACCESS_LOG_PATH": "/var/log/nginx/app-access.log",
"NGINX_ACCESS_LOG_FORMAT": "",
@@ -99,6 +100,7 @@ func TestTemplate_HTTPOnlyBasicProxy(t *testing.T) {
out := renderTemplate(t, defaultVars())
mustContain(t, out, "listen [::]:80;")
mustContain(t, out, "resolver 127.0.0.1:1053 valid=10s ipv6=off;")
mustContain(t, out, "resolver_timeout 5s;")
mustContain(t, out, `set $dokku_upstream "app.web.docker:5000";`)
mustContain(t, out, "proxy_pass http://$dokku_upstream;")
mustNotContain(t, out, "upstream app-5000 {")
@@ -109,6 +111,14 @@ func TestTemplate_HTTPOnlyBasicProxy(t *testing.T) {
mustNotContain(t, out, "http2_push_preload")
}
func TestTemplate_ResolverTimeoutOverride(t *testing.T) {
v := defaultVars()
v["NGINX_DNS_RESOLVER_TIMEOUT"] = "2s"
out := renderTemplate(t, v)
mustContain(t, out, "resolver_timeout 2s;")
mustNotContain(t, out, "resolver_timeout 5s;")
}
func TestTemplate_HTTPOnlyBasicProxyWithResolverOff(t *testing.T) {
v := defaultVars()
v["NGINX_DNS_RESOLVER"] = "off"

2
plugins/nginx-vhosts/templates/nginx.conf.sigil
View File

@@ -86,6 +86,7 @@ server {
{{ if ne $.NGINX_DNS_RESOLVER "off" }}
resolver {{ $.NGINX_DNS_RESOLVER }} valid=10s ipv6=off;
resolver_timeout {{ $.NGINX_DNS_RESOLVER_TIMEOUT }};
set $dokku_upstream "{{ $.DOKKU_APP_WEB_LISTENER_HOST }}:{{ $upstream_port }}";
proxy_pass http://$dokku_upstream;
{{ else }}
@@ -174,6 +175,7 @@ server {
location / {
{{ if ne $.NGINX_DNS_RESOLVER "off" }}
resolver {{ $.NGINX_DNS_RESOLVER }} valid=10s ipv6=off;
resolver_timeout {{ $.NGINX_DNS_RESOLVER_TIMEOUT }};
set $dokku_upstream "{{ $.DOKKU_APP_WEB_LISTENER_HOST }}:{{ $upstream_port }}";
grpc_pass grpc://$dokku_upstream;
{{ else }}

2
tests/test_coredns_outage
View File

@@ -47,7 +47,7 @@ start_coredns_docker() {
}
curl_status() {
curl -kSso /dev/null -w "%{http_code}" -m 5 "http://$APP.$TARGET/"
curl -kSso /dev/null -w "%{http_code}" -m 10 "http://$APP.$TARGET/"
}
wait_for_status() {

2
tests/unit/nginx-vhosts_17.bats
View File

@@ -150,7 +150,7 @@ teardown() {
echo "status: $status"
assert_success
run curl --connect-to "$TEST_APP.$DOKKU_DOMAIN:80:localhost:80" -kSso /dev/null -w "%{http_code}" -m 5 "http://$TEST_APP.$DOKKU_DOMAIN/"
run curl --connect-to "$TEST_APP.$DOKKU_DOMAIN:80:localhost:80" -kSso /dev/null -w "%{http_code}" -m 10 "http://$TEST_APP.$DOKKU_DOMAIN/"
echo "output: $output"
echo "status: $status"
assert_output "502"