# Kubernetes logs

## Overview

In a Kubernetes setup, StackState functions are distributed across different pods and logs for each function are stored per pod and container. You can access recent logs using `kubectl`, although for long term storage it's recommended to set up log aggregation.

## Kubernetes pods for logging

StackState logs are stored per pod and container. The table below shows the pod to access for logs relating to specific StackState functions. Note that actual pod names will include a number or random string suffix (for example, `stackstate-receiver-5b9d79db86-h2hkz`) and may also include the release name specified when StackState was deployed as a prefix.

{% hint style="info" %}
Note that logs stored on pods will be regularly removed. For long term access to logs, it's advised that you set up [log aggregation](#log-aggregation) for your Kubernetes cluster.
{% endhint %}

| StackState function                           | Logs on pod                                                                                                                                 |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| API (including topology, charts and settings) | `stackstate-api`                                                                                                                            |
| Checks                                        | `stackstate-checks`                                                                                                                         |
| Data indexing into Elasticsearch              | `stackstate-mm2es` (metrics) `stackstate-e2es` (events) `stackstate-trace2es` (traces) `stackstate-sts2es` (events generated by StackState) |
| Data ingestion                                | `stackstate-receiver`                                                                                                                       |
| Event handlers                                | `stackstate-view-health`                                                                                                                    |
| Monitor                                       | `stackstate-checks`                                                                                                                         |
| State propagation                             | `stackstate-state`                                                                                                                          |
| Synchronization                               | `stackstate-sync`                                                                                                                           |
| View health state                             | `stackstate-view-health`                                                                                                                    |

You can access logs on a specific pod using the `kubectl logs` command.

For example:

```
$ kubectl logs stackstate-api-0
```

## Access recent logs

### Pod or container logs

The most recent logs can be retrieved from Kubernetes using the `kubectl logs` command. Check the [pod that you need to monitor](#kubernetes-pods-for-logging) to retrieve a specific log.

For example:

```
# Snapshot of logs for all containers of <pod-name>
$ kubectl logs <pod-name> --all-containers=true

# Stream logs for all containers of <pod-name>
$ kubectl logs -f <pod-name> --all-containers=true

# Snapshot of logs for a specific container of <pod-name>
$ kubectl logs -c <container-name> <pod-name>

# Snapshot of logs for previous terminated container of <pod-name>
$ kubectl logs -p -c <container-name> <pod-name>
```

### Synchronization logs

All synchronization logs can be found in a pod `stackstate-sync-<suffix>`. You can use the synchronization name to locate specific log information in a log snapshot.

For example:

```
# Logs of the synchronization for a specific Kubernetes cluster
$ kubectl logs stackstate-sync-0 | grep "Kubernetes - \<cluster-name\>"

# Logs of the Agent synchronization
$ kubectl logs stackstate-sync-0 | grep "Agent"
```

## Log aggregation

For long term storage of StackState log data, it's advised that you set up log aggregation on your Kubernetes cluster. This can be done using a third party system for storage such as Elasticsearch, Splunk or Logz.io and a log shipper such as Logstash or Fluentd.

For more details of how this can be done, check:

* Shipping logs with [Fluentd (fluentd.org)](https://docs.fluentd.org/container-deployment/kubernetes)
* A complete overview of setting up [log aggregation into Elasticsearch (bitnami.com)](https://docs.bitnami.com/tutorials/integrate-logging-kubernetes-kibana-elasticsearch-fluentd/)

## See also

* [kubectl command reference (kubernetes.io/docs)](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
