From 4346a181947ecc274a598939df15484e9ac0a49f Mon Sep 17 00:00:00 2001 From: Jose Diaz-Gonzalez Date: Sat, 30 May 2026 19:35:20 -0400 Subject: [PATCH] feat: add scheduler-k3s:charts:set and :charts:report Helm chart overrides on the scheduler-k3s plugin were previously set through the generic `scheduler-k3s:set --global chart..` interface, which mixed chart-level configuration into the same command that manages scheduler properties and offered no focused way to inspect which overrides were configured. A dedicated `scheduler-k3s:charts:set` sets and clears chart-specific helm values, and a complementary `scheduler-k3s:charts:report` surfaces configured overrides per chart with optional JSON output and single-field flag queries. The legacy form on `scheduler-k3s:set` continues to work but now emits a deprecation warning pointing users at the new subcommand. --- docs/deployment/schedulers/k3s.md | 23 ++- plugins/scheduler-k3s/Makefile | 2 +- .../scheduler-k3s/src/commands/commands.go | 2 + .../src/subcommands/subcommands.go | 17 ++ plugins/scheduler-k3s/subcommands.go | 149 ++++++++++++++++++ tests/unit/scheduler-k3s-charts.bats | 107 +++++++++++++ 6 files changed, 294 insertions(+), 6 deletions(-) create mode 100644 tests/unit/scheduler-k3s-charts.bats diff --git a/docs/deployment/schedulers/k3s.md b/docs/deployment/schedulers/k3s.md index a0aa9c377..457ade4d4 100644 --- a/docs/deployment/schedulers/k3s.md +++ b/docs/deployment/schedulers/k3s.md @@ -7,6 +7,8 @@ scheduler-k3s:annotations:set () [--process-type PROCESS_TYPE] <--resource-type RESOURCE_TYPE>, Set or clear an annotation for a given app/process-type/resource-type combination scheduler-k3s:autoscaling-auth:set [<--metadata key=value>...], Set or clear a scheduler-k3s autoscaling keda trigger authentication resource for an app scheduler-k3s:autoscaling-auth:report [--format stdout|json] [--include-metadata] # Displays a scheduler-k3s autoscaling auth report for an app +scheduler-k3s:charts:report [] [--format stdout|json] # Displays a scheduler-k3s chart override report +scheduler-k3s:charts:set () # Set or clear a chart-specific helm value scheduler-k3s:cluster:add [ssh://user@host:port] # Adds a server node to a Dokku-managed cluster scheduler-k3s:cluster:list # Lists all nodes in a Dokku-managed cluster scheduler-k3s:cluster:remove [node-id] # Removes client node to a Dokku-managed cluster @@ -658,21 +660,32 @@ The default value for the `kube-context` is an empty string, and will result in ### Customizing Helm Chart Properties -Dokku includes a number of helm charts by default with settings that are optimized for Dokku. That said, it may be useful to further customize the charts for a given environment. Users can customize which charts are installed by setting properties prefixed with `chart.$CHART_NAME.` with the `--global` flag. +Dokku includes a number of helm charts by default with settings that are optimized for Dokku. That said, it may be useful to further customize the charts for a given environment. Chart overrides are managed via the `scheduler-k3s:charts:set` command, which takes a `.` argument and a value. Chart overrides are always global because helm charts are not managed on a per-app basis. ```shell -dokku scheduler-k3s:set --global chart.cert-manager.version 1.13.3 +dokku scheduler-k3s:charts:set cert-manager.version 1.13.3 ``` > [!NOTE] > Properties follow dot-notation, and are expanded according to Helm's internal logic. See the [Helm documentation](https://helm.sh/docs/helm/helm_install/#helm-install) for `helm install` for further details. -To unset a chart property, omit the value from the `scheduler-k3s:set` call: +To unset a chart property, omit the value from the `scheduler-k3s:charts:set` call: ```shell -dokku scheduler-k3s:set --global chart.cert-manager.version +dokku scheduler-k3s:charts:set cert-manager.version ``` +Configured chart overrides can be inspected with the `scheduler-k3s:charts:report` command. Without an argument, it lists every chart known to Dokku along with any configured overrides; passing a chart name scopes the report to that chart: + +```shell +dokku scheduler-k3s:charts:report +dokku scheduler-k3s:charts:report cert-manager +dokku scheduler-k3s:charts:report --format json +``` + +> [!NOTE] +> The legacy form `dokku scheduler-k3s:set --global chart.. ` continues to work but is deprecated and will be removed in a future major release. Migrate to `scheduler-k3s:charts:set` instead. + A `scheduler-k3s:ensure-charts` command with the `--force` flag is required after changing any chart properties in order to have them apply. This will install all charts, not just the ones that have changed. ```shell @@ -811,4 +824,4 @@ If unspecified for any task, the default reservation will be `.1` CPU and `128Mi | `rollback-on-failure` | app + global | `false` | `--scheduler-k3s-rollback-on-failure`, `--scheduler-k3s-global-rollback-on-failure`, `--scheduler-k3s-computed-rollback-on-failure` | When `true`, helm rolls back the release if a deploy fails | | `shm-size` | app + global | none | `--scheduler-k3s-shm-size`, `--scheduler-k3s-global-shm-size`, `--scheduler-k3s-computed-shm-size` | `/dev/shm` size override applied to app containers | | `token` | global only | none | `--scheduler-k3s-global-token` (masked as `*******` in default stdout output; the raw value is returned when queried via `--format json` or when this flag is requested explicitly) | Cluster join token used by `scheduler-k3s:cluster-add` | -| `chart..` | global only | none | `--scheduler-k3s-global-chart..` (dynamic per chart/property) | Override a value injected into the helm chart named `` (one row per chart/property pair) | +| `chart..` | global only | none | `--scheduler-k3s-global-chart..` (dynamic per chart/property) | Override a value injected into the helm chart named `` (one row per chart/property pair). Manage these via the dedicated `scheduler-k3s:charts:set` / `scheduler-k3s:charts:report` commands; the `scheduler-k3s:set`/`scheduler-k3s:report` form is deprecated. | diff --git a/plugins/scheduler-k3s/Makefile b/plugins/scheduler-k3s/Makefile index 13dc52e1b..c34a10537 100644 --- a/plugins/scheduler-k3s/Makefile +++ b/plugins/scheduler-k3s/Makefile @@ -1,4 +1,4 @@ -SUBCOMMANDS = subcommands/annotations:set subcommands/autoscaling-auth:set subcommands/autoscaling-auth:report subcommands/cluster:add subcommands/cluster:list subcommands/cluster:remove subcommands/ensure-charts subcommands/initialize subcommands/labels:set subcommands/profiles:add subcommands/profiles:list subcommands/profiles:remove subcommands/report subcommands/set subcommands/show-kubeconfig subcommands/uninstall +SUBCOMMANDS = subcommands/annotations:set subcommands/autoscaling-auth:set subcommands/autoscaling-auth:report subcommands/charts:report subcommands/charts:set subcommands/cluster:add subcommands/cluster:list subcommands/cluster:remove subcommands/ensure-charts subcommands/initialize subcommands/labels:set subcommands/profiles:add subcommands/profiles:list subcommands/profiles:remove subcommands/report subcommands/set subcommands/show-kubeconfig subcommands/uninstall TRIGGERS = triggers/core-post-deploy triggers/core-post-extract triggers/install triggers/post-app-clone-setup triggers/post-app-rename-setup triggers/post-certs-update triggers/post-certs-remove triggers/post-create triggers/post-delete triggers/report triggers/scheduler-app-status triggers/scheduler-deploy triggers/scheduler-enter triggers/scheduler-is-deployed triggers/scheduler-logs triggers/scheduler-proxy-config triggers/scheduler-proxy-logs triggers/scheduler-post-delete triggers/scheduler-run triggers/scheduler-run-list triggers/scheduler-stop triggers/scheduler-cron-write triggers/storage-create triggers/storage-destroy triggers/storage-status triggers/scheduler-storage-exec BUILD = commands subcommands triggers PLUGIN_NAME = scheduler-k3s diff --git a/plugins/scheduler-k3s/src/commands/commands.go b/plugins/scheduler-k3s/src/commands/commands.go index e5b51b68d..021ebbf4d 100644 --- a/plugins/scheduler-k3s/src/commands/commands.go +++ b/plugins/scheduler-k3s/src/commands/commands.go @@ -19,6 +19,8 @@ Additional commands:` helpContent = ` scheduler-k3s:autoscaling-auth:set [<--metadata key=value>...], Set or clear a scheduler-k3s autoscaling keda trigger authentication resource for an app scheduler-k3s:annotations:set () [--process-type PROCESS_TYPE] <--resource-type RESOURCE_TYPE>, Set or clear an annotation for a given app/process-type/resource-type combination + scheduler-k3s:charts:report [] [--format stdout|json], Displays a scheduler-k3s chart override report + scheduler-k3s:charts:set (), Set or clear a chart-specific helm value scheduler-k3s:cluster:add [--profile PROFILE] [--role ROLE] [--insecure-allow-unknown-hosts] [--server-ip SERVER_IP] [--taint-scheduling] [--kubelet-args KUBELET_ARGS] , Adds a server node to a Dokku-managed cluster scheduler-k3s:cluster:list [--format json|stdout], Lists all nodes in a Dokku-managed cluster scheduler-k3s:cluster:remove [node-id], Removes client node to a Dokku-managed cluster diff --git a/plugins/scheduler-k3s/src/subcommands/subcommands.go b/plugins/scheduler-k3s/src/subcommands/subcommands.go index c5680e66d..fc748b5ad 100644 --- a/plugins/scheduler-k3s/src/subcommands/subcommands.go +++ b/plugins/scheduler-k3s/src/subcommands/subcommands.go @@ -54,6 +54,23 @@ func main() { args.Parse(os.Args[2:]) appName := args.Arg(0) err = scheduler_k3s.CommandAutoscalingAuthReport(appName, *format, *global, *includeMetadata) + case "charts:set": + args := flag.NewFlagSet("scheduler-k3s:charts:set", flag.ExitOnError) + args.Parse(os.Args[2:]) + property := args.Arg(0) + value := args.Arg(1) + err = scheduler_k3s.CommandChartsSet(property, value) + case "charts:report": + args := flag.NewFlagSet("scheduler-k3s:charts:report", flag.ExitOnError) + format := args.String("format", "stdout", "format: [ stdout | json ]") + reportArgs, flagErr := common.ParseReportArgs("scheduler-k3s-charts", os.Args[2:]) + if flagErr == nil { + args.Parse(reportArgs.OSArgs) + chartName := args.Arg(0) + err = scheduler_k3s.CommandChartsReport(chartName, *format, reportArgs.InfoFlag) + } else { + err = flagErr + } case "cluster:add": args := flag.NewFlagSet("scheduler-k3s:cluster:add", flag.ExitOnError) allowUknownHosts := args.Bool("insecure-allow-unknown-hosts", false, "insecure-allow-unknown-hosts: allow unknown hosts") diff --git a/plugins/scheduler-k3s/subcommands.go b/plugins/scheduler-k3s/subcommands.go index a33d7377c..fb7e6aa7b 100644 --- a/plugins/scheduler-k3s/subcommands.go +++ b/plugins/scheduler-k3s/subcommands.go @@ -400,6 +400,153 @@ func CommandInitialize(ingressClass string, serverIP string, taintScheduling boo return nil } +// CommandChartsSet sets or clears a chart-specific helm value for the scheduler +func CommandChartsSet(propertyArg string, value string) error { + if propertyArg == "" { + return fmt.Errorf("Invalid property, expected format: .") + } + + dotIndex := strings.Index(propertyArg, ".") + if dotIndex <= 0 || dotIndex == len(propertyArg)-1 { + return fmt.Errorf("Invalid property, expected format: .") + } + + chartName := propertyArg[:dotIndex] + chartProperty := propertyArg[dotIndex+1:] + + knownCharts := []string{} + chartFound := false + for _, chart := range HelmCharts { + knownCharts = append(knownCharts, chart.ReleaseName) + if chart.ReleaseName == chartName { + chartFound = true + } + } + if !chartFound { + sort.Strings(knownCharts) + return fmt.Errorf("Invalid chart name %q, valid charts: %s", chartName, strings.Join(knownCharts, ", ")) + } + + key := fmt.Sprintf("chart.%s.%s", chartName, chartProperty) + if value != "" { + common.LogInfo2Quiet(fmt.Sprintf("Setting %s to %s", key, value)) + if err := common.PropertyWrite("scheduler-k3s", "--global", key, value); err != nil { + return fmt.Errorf("Unable to write property: %w", err) + } + return nil + } + + common.LogInfo2Quiet(fmt.Sprintf("Unsetting %s", key)) + if err := common.PropertyDelete("scheduler-k3s", "--global", key); err != nil { + return fmt.Errorf("Unable to delete property: %w", err) + } + return nil +} + +// CommandChartsReport displays a scheduler-k3s chart override report +func CommandChartsReport(chartName string, format string, infoFlag string) error { + if format != "stdout" && format != "json" { + return fmt.Errorf("Invalid format: %s", format) + } + + if format == "json" && infoFlag != "" { + return fmt.Errorf("--format flag cannot be specified when specifying an info flag") + } + + charts := []HelmChart{} + if chartName != "" { + found := false + for _, chart := range HelmCharts { + if chart.ReleaseName == chartName { + charts = append(charts, chart) + found = true + break + } + } + if !found { + knownCharts := []string{} + for _, chart := range HelmCharts { + knownCharts = append(knownCharts, chart.ReleaseName) + } + sort.Strings(knownCharts) + return fmt.Errorf("Invalid chart name %q, valid charts: %s", chartName, strings.Join(knownCharts, ", ")) + } + } else { + charts = append(charts, HelmCharts...) + } + + overridesByChart := map[string]map[string]string{} + flatOverrides := map[string]string{} + flagToValue := map[string]string{} + for _, chart := range charts { + properties, err := common.PropertyGetAllByPrefix("scheduler-k3s", "--global", "chart."+chart.ReleaseName+".") + if err != nil { + return fmt.Errorf("Unable to get chart properties: %w", err) + } + + chartOverrides := map[string]string{} + prefix := "chart." + chart.ReleaseName + "." + for key, value := range properties { + overrideKey := strings.TrimPrefix(key, prefix) + chartOverrides[overrideKey] = value + flatKey := chart.ReleaseName + "." + overrideKey + flatOverrides[flatKey] = value + flagToValue["--scheduler-k3s-charts-"+flatKey] = value + } + overridesByChart[chart.ReleaseName] = chartOverrides + } + + if infoFlag != "" { + value, ok := flagToValue[infoFlag] + if !ok { + validFlags := []string{} + for flag := range flagToValue { + validFlags = append(validFlags, flag) + } + sort.Strings(validFlags) + return fmt.Errorf("Invalid flag passed, valid flags: %s", strings.Join(validFlags, ", ")) + } + fmt.Println(value) + return nil + } + + if format == "json" { + b, err := json.Marshal(flatOverrides) + if err != nil { + return fmt.Errorf("Unable to marshal json: %w", err) + } + fmt.Println(string(b)) + return nil + } + + length := 31 + for _, overrides := range overridesByChart { + for key := range overrides { + label := fmt.Sprintf("Chart property %s:", key) + if len(label) > length { + length = len(label) + } + } + } + + for _, chart := range charts { + common.LogInfo2Quiet(fmt.Sprintf("%s chart information", chart.ReleaseName)) + + overrideKeys := []string{} + for key := range overridesByChart[chart.ReleaseName] { + overrideKeys = append(overrideKeys, key) + } + sort.Strings(overrideKeys) + + for _, key := range overrideKeys { + label := fmt.Sprintf("Chart property %s:", key) + common.LogVerbose(fmt.Sprintf("%s%s", common.RightPad(label, length, " "), overridesByChart[chart.ReleaseName][key])) + } + } + + return nil +} + // CommandClusterAdd adds a server to the k3s cluster func CommandClusterAdd(profileName string, role string, remoteHost string, serverIP string, allowUknownHosts bool, taintScheduling bool, kubeletArgs []string) error { if err := isK3sInstalled(); err != nil { @@ -1166,6 +1313,8 @@ func CommandSet(appName string, property string, value string) error { validProperties := DefaultProperties globalProperties := GlobalProperties if strings.HasPrefix(property, "chart.") { + common.LogWarn("scheduler-k3s:set chart.* properties are deprecated; use scheduler-k3s:charts:set . instead") + if appName != "--global" { return fmt.Errorf("Chart properties can only be set globally") } diff --git a/tests/unit/scheduler-k3s-charts.bats b/tests/unit/scheduler-k3s-charts.bats new file mode 100644 index 000000000..a15c2326b --- /dev/null +++ b/tests/unit/scheduler-k3s-charts.bats @@ -0,0 +1,107 @@ +#!/usr/bin/env bats + +load test_helper + +setup() { + global_setup +} + +teardown() { + dokku scheduler-k3s:charts:set cert-manager.installCRDs >/dev/null 2>/dev/null || true + dokku scheduler-k3s:charts:set cert-manager.version >/dev/null 2>/dev/null || true + dokku scheduler-k3s:charts:set traefik.replicas >/dev/null 2>/dev/null || true + global_teardown +} + +@test "(scheduler-k3s:charts:set) sets and unsets a chart property" { + run /bin/bash -c "dokku scheduler-k3s:charts:set cert-manager.installCRDs false" + assert_success + + run /bin/bash -c "dokku scheduler-k3s:charts:report cert-manager --format json | jq -r '.\"cert-manager.installCRDs\"'" + assert_success + assert_output "false" + + run /bin/bash -c "dokku scheduler-k3s:charts:report cert-manager" + assert_success + assert_output_contains "cert-manager chart information" + assert_output_contains "Chart property installCRDs:" + assert_output_contains "false" + + run /bin/bash -c "dokku scheduler-k3s:charts:set cert-manager.installCRDs" + assert_success + + run /bin/bash -c "dokku scheduler-k3s:charts:report cert-manager --format json" + assert_success + assert_output "{}" + + run /bin/bash -c "dokku scheduler-k3s:report --global --format json | jq -r '.\"scheduler-k3s-global-chart.cert-manager.installCRDs\" // \"missing\"'" + assert_success + assert_output "missing" +} + +@test "(scheduler-k3s:charts:set) rejects invalid chart names and malformed property args" { + run /bin/bash -c "dokku scheduler-k3s:charts:set" + assert_failure + assert_output_contains "Invalid property" + + run /bin/bash -c "dokku scheduler-k3s:charts:set cert-manager" + assert_failure + assert_output_contains "Invalid property" + + run /bin/bash -c "dokku scheduler-k3s:charts:set .version 1.0" + assert_failure + assert_output_contains "Invalid property" + + run /bin/bash -c "dokku scheduler-k3s:charts:set cert-manager. 1.0" + assert_failure + assert_output_contains "Invalid property" + + run /bin/bash -c "dokku scheduler-k3s:charts:set not-a-chart.version 1.0" + assert_failure + assert_output_contains "Invalid chart name" + + run /bin/bash -c "dokku scheduler-k3s:set --global chart.cert-manager.version 1.13.3" + assert_success + assert_output_contains "deprecated" + + run /bin/bash -c "dokku scheduler-k3s:set --global chart.cert-manager.version" + assert_success +} + +@test "(scheduler-k3s:charts:report) per-chart, single-flag query, and flat JSON" { + run /bin/bash -c "dokku scheduler-k3s:charts:set cert-manager.version 1.13.3" + assert_success + run /bin/bash -c "dokku scheduler-k3s:charts:set traefik.replicas 2" + assert_success + + run /bin/bash -c "dokku scheduler-k3s:charts:report" + assert_success + assert_output_contains "cert-manager chart information" + assert_output_contains "traefik chart information" + assert_output_contains "longhorn chart information" + assert_output_contains "ingress-nginx chart information" + assert_output_contains "keda chart information" + assert_output_contains "keda-add-ons-http chart information" + assert_output_contains "vector chart information" + + run /bin/bash -c "dokku scheduler-k3s:charts:report cert-manager" + assert_success + assert_output_contains "cert-manager chart information" + assert_output_contains "Chart property version:" + assert_output_contains "1.13.3" + + run /bin/bash -c "dokku scheduler-k3s:charts:report cert-manager" + assert_output_not_contains "traefik chart information" + + run /bin/bash -c "dokku scheduler-k3s:charts:report --scheduler-k3s-charts-cert-manager.version" + assert_success + assert_output "1.13.3" + + run /bin/bash -c "dokku scheduler-k3s:charts:report --format json | jq -r '.\"cert-manager.version\"'" + assert_success + assert_output "1.13.3" + + run /bin/bash -c "dokku scheduler-k3s:charts:report --format json | jq -r '.\"traefik.replicas\"'" + assert_success + assert_output "2" +}