github/dokku
0
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Merge pull request #4291 from dokku/2268-vector-logging

Browse Source 
Add log aggregation support via Vector
This commit is contained in:
Jose Diaz-Gonzalez
2021-01-04 22:56:25 -05:00
committed by GitHub
parent 96ed840d17 2d07d89d60
commit 4c28ae2630
23 changed files with 641 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/configuration/environment-variables.md
View File

@@ -87,6 +87,7 @@ The following config variables have special meanings and can be set in a variety
| `PLUGIN_CORE_AVAILABLE_PATH` | `$PLUGIN_CORE_PATH/available"` | `/etc/environment` <br /> `~dokku/.dokkurc` <br /> `~dokku/.dokkurc/*` | The directory that stores all available core plugins. |
| `PLUGIN_CORE_ENABLED_PATH` | `$PLUGIN_CORE_PATH/enabled"` | `/etc/environment` <br /> `~dokku/.dokkurc` <br /> `~dokku/.dokkurc/*` | The directory that stores all enabled core plugins. |
| `DOKKU_LOGS_DIR` | `/var/log/dokku` | `/etc/environment` <br /> `~dokku/.dokkurc` <br /> `~dokku/.dokkurc/*` | Where dokku logs should be written to. |
| `DOKKU_LOGS_HOST_DIR` | `$DOKKU_LOGS_DIR` | `/etc/environment` <br /> `~dokku/.dokkurc` <br /> `~dokku/.dokkurc/*` | A path on the host that will be mounted into the vector logging container. |
| `DOKKU_EVENTS_LOGFILE` | `$DOKKU_LOGS_DIR/events.log` | `/etc/environment` <br /> `~dokku/.dokkurc` <br /> `~dokku/.dokkurc/*` | Where the events log file is written to. |
| `DOKKU_APP_NAME` | none | `--app APP` flag | Name of application to work on. Respected by core plugins. |
| `DOKKU_APPS_FORCE_DELETE` | none | `--force` flag | Whether to force delete an application. Also used by other plugins for destructive actions. |

100
docs/deployment/logs.md
View File

@@ -2,7 +2,12 @@
```
logs <app> [-h] [-t] [-n num] [-q] [-p process] # Display recent log output
logs:failed [--parallel count] [--all|<app>] # Shows the last failed deploy logs
logs:failed [--all|<app>] # Shows the last failed deploy logs
logs:report [<app>] [<flag>] # Displays a logs report for one or more apps
logs:set [--global|<app>] <key> <value> # Set or clear a logs property for an app
logs:vector-logs # Tail the logs of the vector container
logs:vector-start # Start the vector logging container
logs:vector-stop # Stop the vector logging container
```
## Usage
@@ -40,7 +45,7 @@ will show logs continually from the web process.
In some cases, it may be useful to retrieve the logs from a previously failed deploy.
You can retrieve these logs by using the `logs:failed` command
You can retrieve these logs by using the `logs:failed` command.
```shell
dokku logs:failed node-js-app
@@ -51,3 +56,94 @@ You may also fetch all failed app logs by using the `--all` flag.
```shell
dokku logs:failed --all
```
### Vector Logging
> New as of 0.22.6
Vector is an open-source, lightweight and ultra-fast tool for building observability pipelines. Dokku integrates with it for shipping container logs for the `docker-local` scheduler. Users may configure log-shipping on a per-app or global basis, neither of which interfere with the `dokku logs` commands.
#### Starting the Vector container
> Warning: While the default vector image may be updated over time, this will not impact running vector containers. Users are encouraged to view any Dokku and Vector changelogs to ensure their system will continue running as expected.
Vector may be started via the `logs:vector-start` command.
```shell
dokku logs:vector-start
```
This will start a new container named `vector` with Dokku's vector config mounted and ready for use. If a running container already exists, this command will do nothing. Additionally, if a container exists but is not running, this command will attempt to start the container.
While the default vector image is hardcoded, users may specify an alternative via the `--vector-image` flag:
```shell
dokku logs:vector-start --vector-image timberio/vector:latest-debian
```
The `vector` container will be started with the following volume mounts:
- `/var/lib/dokku/data/logs/vector.json:/etc/vector/vector.json`
- `/var/run/docker.sock:/var/run/docker.sock`
- `/var/log/dokku/apps:/var/log/dokku/apps`
The final volume mount - `/var/log/dokku/apps` - may be used for users that wish to ship logs to a file on disk that may be later logrotated. This directory is owned by the `dokku` user and group, with permissions set to `0755`. At this time, log-rotation is not configured for this directory.
#### Stopping the Vector container
Vector may be stopped via the `logs:vector-stop` command.
```shell
dokku logs:vector-stop
```
The `vector` container will be stopped and removed from the system. If the container is not running, this command will do nothing.
#### Configuring a log sink
Vector uses the concept of log "sinks" to send logs to a given endpoint. Log sinks may be configured globally or on a per-app basis by specifying a `sink` in DSN form with the `logs:set` command. Specifying a sink value will reload any running vector container.
```shell
# setting the sink value in quotes is encouraged to avoid
# issues with ampersand encoding in shell commands
dokku logs:set node-js-app sink "console://?encoding[codec]=json"
```
A sink may be removed by setting an empty value, which will also reload the running vector container.
```shell
dokku logs:set node-js-app sink
```
Only one sink may be specified on a per-app basis at a given time.
Log sinks can also be specified globally by specifying the `--global` flag to `logs:set` with no app name specified:
```shell
dokku logs:set --global sink "console://?encoding[codec]=json"
```
As with app-specific sink settings, the global value may also be cleared by setting no value.
```shell
dokku logs:set --global sink
```
##### Log Sink DSN Format
The DSN form of a sink is as follows:
```
SINK_TYPE://?SINK_OPTIONS
```
Valid values for `SINK_TYPE` include all log vector log sinks, while `SINK_OPTIONS` is a query-string form for the sink's options. The following is a short description on how to set various values:
- `bool`: form: `key=bool`
- `string`: form: `key=string`
- `int`: form: `key=int`
- `[string]`: form: `key[]=string`
- `[int]`: form: `key[]=int`
- `table`: form: `option[key]=value
Please read the [sink documentation](https://vector.dev/docs/reference/sinks/) for your sink of choice to configure the sink as desired.