feat: create skeleton of k3s plugin

docs/deployment/schedulers/k3s.md

@@ -0,0 +1,194 @@
+# K3s Scheduler
+
+```
+scheduler-k3s:initialize [<--local>|<--remote user:password@host:port>] # initializes a cluster
+scheduler-k3s:add-server [user:password@host:port]                      # adds a server node to a Dokku-managed cluster
+scheduler-k3s:add-client [user:password@host:port]                      # adds a client node to a Dokku-managed cluster
+scheduler-k3s:set [<app>|--global] <key> (<value>)                      # Set or clear a scheduler-k3s property for an app or the scheduler
+```
+
+> [!IMPORTANT]
+> New as of 0.33.0
+
+For multi-server support, Dokku provides the ability for users to setup a K3s cluster. As with all schedulers, it is set on a per-app basis. The scheduler can currently be overridden by running the following command:
+
+```shell
+dokku scheduler:set node-js-app selected k3s
+```
+
+As it is the default, unsetting the `selected` scheduler property is also a valid way to reset the scheduler.
+
+```shell
+dokku scheduler:set node-js-app k3s
+```
+
+## Tutorial
+
+### Single-node cluster
+
+Initialize the cluster in single-node mode. This will start k3s on the Dokku node itself.
+
+```shell
+dokku scheduler-k3s:initialize
+```
+
+The above command will initialize a cluster with an automatically generated token for authentication. This token should be stored securely for later recovery, and can be displayed with via the `scheduler-k3s:report` command:
+
+```shell
+dokku scheduler-k3s:report --global
+```
+
+```
+=====> --global scheduler-k3s information
+       scheduler k3s token: topsecret:server:token
+```
+
+Setup registry authentication. The K3s scheduler plugin will automatically pick up any newly configured registry backends and ensure nodes in the cluster have the credentials in place for pulling images from the cluster.
+
+```shell
+# hub.docker.com
+dokku registry:login hub.docker.com $USERNAME $PASSWORD
+```
+
+To ensure images are pushed and pulled from the correct registry, set the correct `server` registry property. This can be set on a per-app basis, but we will set it globally here for this tutorial.
+
+```shell
+dokku registry:set --global server hub.docker.com
+```
+
+If using docker hub, you'll need to use a custom repository name. As it is unique, this must be set on a per-app basis.
+
+```shell
+dokku registry:set node-js-app image-repo my-awesome-prefix/node-js-app
+```
+
+Additionally, apps should be configured to push images on the release phase via the `push-on-release` registry property.
+
+```shell
+dokku registry:set --global push-on-release true
+```
+
+As routing is handled by traefik managed on the k3s plugin, set the proxy plugin to `k3s` as well. This implements a traefik-plugin compatible layer, allowing users to otherwise interact with the `traefik` plugin as expected.
+
+```shell
+dokku proxy:set --global k3s
+```
+
+Finally, set the scheduler to `k3s` so that app deploys will work on k3s.
+
+```shell
+dokku scheduler:set --global selected k3s
+```
+
+At this point, all app deploys will be performed against the k3s cluster.
+
+### Running a multi-cluster node
+
+For high-availability, it is recommended to add both agent and server nodes to the cluster. Dokku will default to starting the cluster with an embedded Etcd database backend, and is ready to add new agent or server nodes immediately.
+
+Multiple server nodes can be added with the `scheduler-k3s:add-server` command. This will ssh onto the specified server, install k3s, and join it to the current Dokku node in server mode.
+
+> [!WARNING]
+> Certain ports must be open for cross-server communication. Refer to the [K3s networking documentation](https://docs.k3s.io/installation/requirements?os=debian#networking) for the required open ports between servers prior to running the command.
+
+```shell
+dokku scheduler-k3s:add-server root@server-1.example.com
+```
+
+Server nodes are typically used to replicate the cluster state, and it is recommended to have an odd number of nodes spread across several availability zones (datacenters in close proximity within a region). This allows for higher availability in the event of a cluster failure. Server nodes do not run any app workloads, but run control-plane services such as the traefik load balancer and the etcd backing store.
+
+> [!NOTE]
+> Only the initial Dokku server will be properly configured for deployment, and should be considered your git remote. Additional server nodes are for ensuring high-availability of the K3s etcd state. Ensure this server is properly backed up and restorable or deployments will not work.
+
+Agent nodes are used to run. To add an agent, run the `add-agent` command. This will ssh onto the specified server, install k3s, and join it to the current Dokku node in agent mode. Agents are typically used to run app workloads.
+
+```shell
+dokku scheduler-k3s:add-agent root@server-1.example.com
+```
+
+When attaching an agent or server node, the K3s plugin will look at the IP associated with the `eth0` interface and use that to connect the new node to the cluster. To change this, set the `network-interface` property to the appropriate value.
+
+```shell
+dokku scheduler-k3s:set --global network-interface eth1
+```
+
+Dokku does not manage the ssh key of the server, but the value that should be added to the remote root user's `/root/.ssh/authorized_keys` file can be checked with the `git:public-key` command:
+
+```shell
+dokku git:public-key
+```
+
+### Joining an existing cluster
+
+> [!IMPORTANT]
+> This is future planned behavior. Commands here will not work as expected.
+
+In some cases, Dokku must be setup from scratch and joined to an existing cluster. To do so, first set the k3s token value:
+
+```shell
+dokku scheduler-k3s:set --global token topsecret:server:token
+```
+
+Next, run the `scheduler-k3s:join-cluster` command. This command takes a single server node in the cluster, will SSH onto the specified server, retrieve the kubeconfig, and then configure kubectl on the Dokku node to speak with the specified k3s cluster. Dokku will also periodically retrieve all server nodes in the cluster so that server nodes can be safely replaced as needed.
+
+```shell
+dokku scheduler-k3s:join-cluster root@server-1.example.com
+```
+
+### Using kubectl remotely
+
+> [!WARNING]
+> Certain ports must be open for interacting with the remote kubernets api. Refer to the [K3s networking documentation](https://docs.k3s.io/installation/requirements?os=debian#networking) for the required open ports between servers prior to running the command.
+
+By default, Dokku assumes that all it controls all actions on the cluster, and thus does not expose the `kubectl` binary for administrators. To interact with kubectl, you will need to retrieve the `kubeconfig` for the cluster and configure your client to use that configuration.
+
+```shell
+dokku scheduler-k3s:show-kubeconfig
+```
+
+## Scheduler Interface
+
+The following sections describe implemented and unimplemented scheduler functionality for the `k3s` scheduler.
+
+### Implemented Commands and Triggers
+
+This plugin implements various functionality through `plugn` triggers to integrate with Docker for running apps on a single server. The following functionality is supported by the `scheduler-docker-local` plugin.
+
+- `apps:destroy`
+- `deploy`
+- `logs`
+- `ps:stop`
+
+### Unimplemented command functionality
+
+- `apps:clone`
+- `apps:rename`
+- `ps:inspect`
+- `enter`
+- `run`
+- `run:detached`
+
+The following Dokku functionality is not implemented at this time.
+
+- `cron`
+- `vector` log integration
+- one-off tasks
+- persistent storage
+
+### Logging support
+
+App logs for the `logs` command are fetched by Dokku from running containers via the `kubectl` cli. Persisting logs via Vector is not implemented at this time.
+
+### Supported Resource Management Properties
+
+The `docker-local` scheduler supports a minimal list of resource _limits_ and _reservations_. The following properties are supported:
+
+#### Resource Limits
+
+- cpu: is specified in number of CPUs a process can access.
+- memory: should be specified with a suffix of `b` (bytes), `k` (kilobytes), `m` (megabytes), `g` (gigabytes). Default unit is `m` (megabytes).
+
+#### Resource Reservations
+
+- cpu: is specified in number of CPUs a process can access.
+- memory: should be specified with a suffix of `b` (bytes), `k` (kilobytes), `m` (megabytes), `g` (gigabytes). Default unit is `m` (megabytes).

plugins/scheduler-k3s/.gitignore

@@ -0,0 +1,8 @@
+/commands
+/subcommands/*
+/triggers/*
+/triggers
+/scheduler-*
+/install
+/post-*
+/report

plugins/scheduler-k3s/Makefile

@@ -0,0 +1,6 @@
+SUBCOMMANDS = subcommands/report subcommands/set
+TRIGGERS = triggers/install triggers/post-app-clone-setup triggers/post-app-rename-setup triggers/post-delete triggers/report
+BUILD = commands subcommands triggers
+PLUGIN_NAME = scheduler
+
+include ../../common.mk

plugins/scheduler-k3s/go.mod

@@ -0,0 +1,20 @@
+module github.com/dokku/dokku/plugins/scheduler
+
+go 1.21
+
+require (
+	github.com/dokku/dokku/plugins/common v0.0.0-00010101000000-000000000000
+	github.com/spf13/pflag v1.0.5
+)
+
+require (
+	github.com/codegangsta/inject v0.0.0-20150114235600-33e0aa1cb7c0 // indirect
+	github.com/codeskyblue/go-sh v0.0.0-20190412065543-76bd3d59ff27 // indirect
+	github.com/joho/godotenv v1.2.0 // indirect
+	github.com/otiai10/copy v1.14.0 // indirect
+	github.com/ryanuber/columnize v2.1.2+incompatible // indirect
+	golang.org/x/sync v0.5.0 // indirect
+	golang.org/x/sys v0.13.0 // indirect
+)
+
+replace github.com/dokku/dokku/plugins/common => ../common

plugins/scheduler-k3s/plugin.toml

@@ -0,0 +1,4 @@
+[plugin]
+description = "dokku core scheduler-k3s plugin"
+version = "0.32.3"
+[plugin.config]

plugins/scheduler-k3s/report.go

@@ -0,0 +1,24 @@
+package scheduler_k3s
+
+import (
+	"github.com/dokku/dokku/plugins/common"
+)
+
+// ReportSingleApp is an internal function that displays the scheduler-k3s report for one or more apps
+func ReportSingleApp(appName string, format string, infoFlag string) error {
+	if err := common.VerifyAppName(appName); err != nil {
+		return err
+	}
+
+	flags := map[string]common.ReportFunc{}
+
+	flagKeys := []string{}
+	for flagKey := range flags {
+		flagKeys = append(flagKeys, flagKey)
+	}
+
+	trimPrefix := false
+	uppercaseFirstCharacter := true
+	infoFlags := common.CollectReport(appName, infoFlag, flags)
+	return common.ReportSingleApp("scheduler-k3s", appName, infoFlag, infoFlags, flagKeys, format, trimPrefix, uppercaseFirstCharacter)
+}

plugins/scheduler-k3s/scheduler_k3s.go

@@ -0,0 +1,12 @@
+package scheduler_k3s
+
+var (
+	// DefaultProperties is a map of all valid network properties with corresponding default property values
+	DefaultProperties = map[string]string{}
+
+	// GlobalProperties is a map of all valid global network properties
+	GlobalProperties = map[string]bool{
+		"network-interface": true,
+		"token":             true,
+	}
+)

plugins/scheduler-k3s/src/commands/commands.go

@@ -0,0 +1,56 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"os"
+	"strconv"
+	"strings"
+
+	"github.com/dokku/dokku/plugins/common"
+)
+
+const (
+	helpHeader = `Usage: dokku scheduler-k3s[:COMMAND]
+
+Manage scheduler-k3s settings for an app
+
+Additional commands:`
+
+	helpContent = `
+    scheduler-k3s:report [<app>] [<flag>], Displays a scheduler-k3s report for one or more apps
+    scheduler-k3s:set <app> <property> (<value>), Set or clear a scheduler-k3s property for an app
+`
+)
+
+func main() {
+	flag.Usage = usage
+	flag.Parse()
+
+	cmd := flag.Arg(0)
+	switch cmd {
+	case "scheduler-k3s", "scheduler-k3s:help":
+		usage()
+	case "help":
+		command := common.NewShellCmd(fmt.Sprintf("ps -o command= %d", os.Getppid()))
+		command.ShowOutput = false
+		output, err := command.Output()
+
+		if err == nil && strings.Contains(string(output), "--all") {
+			fmt.Println(helpContent)
+		} else {
+			fmt.Print("\n    scheduler-k3s, Manage scheduler-k3s settings for an app\n")
+		}
+	default:
+		dokkuNotImplementExitCode, err := strconv.Atoi(os.Getenv("DOKKU_NOT_IMPLEMENTED_EXIT"))
+		if err != nil {
+			fmt.Println("failed to retrieve DOKKU_NOT_IMPLEMENTED_EXIT environment variable")
+			dokkuNotImplementExitCode = 10
+		}
+		os.Exit(dokkuNotImplementExitCode)
+	}
+}
+
+func usage() {
+	common.CommandUsage(helpHeader, helpContent)
+}

plugins/scheduler-k3s/src/subcommands/subcommands.go

@@ -0,0 +1,50 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"strings"
+
+	"github.com/dokku/dokku/plugins/common"
+	scheduler_k3s "github.com/dokku/dokku/plugins/scheduler-k3s"
+
+	flag "github.com/spf13/pflag"
+)
+
+// main entrypoint to all subcommands
+func main() {
+	parts := strings.Split(os.Args[0], "/")
+	subcommand := parts[len(parts)-1]
+
+	var err error
+	switch subcommand {
+	case "report":
+		args := flag.NewFlagSet("scheduler-k3s:report", flag.ExitOnError)
+		format := args.String("format", "stdout", "format: [ stdout | json ]")
+		osArgs, infoFlag, flagErr := common.ParseReportArgs("scheduler-k3s", os.Args[2:])
+		if flagErr == nil {
+			args.Parse(osArgs)
+			appName := args.Arg(0)
+			err = scheduler_k3s.CommandReport(appName, *format, infoFlag)
+		}
+	case "set":
+		args := flag.NewFlagSet("scheduler-k3s:set", flag.ExitOnError)
+		global := args.Bool("global", false, "--global: set a global property")
+		args.Parse(os.Args[2:])
+		appName := args.Arg(0)
+		property := args.Arg(1)
+		value := args.Arg(2)
+		if *global {
+			appName = "--global"
+			property = args.Arg(0)
+			value = args.Arg(1)
+		}
+		err = scheduler_k3s.CommandSet(appName, property, value)
+	default:
+		err = fmt.Errorf("Invalid plugin subcommand call: %s", subcommand)
+	}
+
+	if err != nil {
+		common.LogFailWithError(err)
+	}
+}

plugins/scheduler-k3s/src/triggers/triggers.go

@@ -0,0 +1,45 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"strings"
+
+	flag "github.com/spf13/pflag"
+
+	"github.com/dokku/dokku/plugins/common"
+	scheduler_k3s "github.com/dokku/dokku/plugins/scheduler-k3s"
+)
+
+// main entrypoint to all triggers
+func main() {
+	parts := strings.Split(os.Args[0], "/")
+	trigger := parts[len(parts)-1]
+	flag.Parse()
+
+	var err error
+	switch trigger {
+	case "install":
+		err = scheduler_k3s.TriggerInstall()
+	case "post-app-clone-setup":
+		oldAppName := flag.Arg(0)
+		newAppName := flag.Arg(1)
+		err = scheduler_k3s.TriggerPostAppCloneSetup(oldAppName, newAppName)
+	case "post-app-rename-setup":
+		oldAppName := flag.Arg(0)
+		newAppName := flag.Arg(1)
+		err = scheduler_k3s.TriggerPostAppRenameSetup(oldAppName, newAppName)
+	case "post-delete":
+		appName := flag.Arg(0)
+		err = scheduler_k3s.TriggerPostDelete(appName)
+	case "report":
+		appName := flag.Arg(0)
+		err = scheduler_k3s.ReportSingleApp(appName, "", "")
+	default:
+		err = fmt.Errorf("Invalid plugin trigger call: %s", trigger)
+	}
+
+	if err != nil {
+		common.LogFailWithError(err)
+	}
+}

plugins/scheduler-k3s/subcommands.go

@@ -0,0 +1,29 @@
+package scheduler_k3s
+
+import (
+	"github.com/dokku/dokku/plugins/common"
+)
+
+// CommandReport displays a scheduler-k3s report for one or more apps
+func CommandReport(appName string, format string, infoFlag string) error {
+	if len(appName) == 0 {
+		apps, err := common.DokkuApps()
+		if err != nil {
+			return err
+		}
+		for _, appName := range apps {
+			if err := ReportSingleApp(appName, format, infoFlag); err != nil {
+				return err
+			}
+		}
+		return nil
+	}
+
+	return ReportSingleApp(appName, format, infoFlag)
+}
+
+// CommandSet set or clear a scheduler-k3s property for an app
+func CommandSet(appName string, property string, value string) error {
+	common.CommandPropertySet("scheduler-k3s", appName, property, value, DefaultProperties, GlobalProperties)
+	return nil
+}

plugins/scheduler-k3s/triggers.go

@@ -0,0 +1,51 @@
+package scheduler_k3s
+
+import (
+	"fmt"
+
+	"github.com/dokku/dokku/plugins/common"
+)
+
+// TriggerInstall runs the install step for the scheduler-k3s plugin
+func TriggerInstall() error {
+	if err := common.PropertySetup("scheduler-k3s"); err != nil {
+		return fmt.Errorf("Unable to install the scheduler-k3s plugin: %s", err.Error())
+	}
+
+	return nil
+}
+
+// TriggerPostAppCloneSetup creates new scheduler-k3s files
+func TriggerPostAppCloneSetup(oldAppName string, newAppName string) error {
+	err := common.PropertyClone("scheduler-k3s", oldAppName, newAppName)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+// TriggerPostAppRenameSetup renames scheduler-k3s files
+func TriggerPostAppRenameSetup(oldAppName string, newAppName string) error {
+	if err := common.PropertyClone("scheduler-k3s", oldAppName, newAppName); err != nil {
+		return err
+	}
+
+	if err := common.PropertyDestroy("scheduler-k3s", oldAppName); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+// TriggerPostDelete destroys the scheduler-k3s data for a given app container
+func TriggerPostDelete(appName string) error {
+	dataErr := common.RemoveAppDataDirectory("scheduler-k3s", appName)
+	propertyErr := common.PropertyDestroy("scheduler-k3s", appName)
+
+	if dataErr != nil {
+		return dataErr
+	}
+
+	return propertyErr
+}