# Linux logs

## Overview

In a Linux setup, StackState keeps all log files in the `var/log` subdirectory of the StackState installation directory. By default, this is `/opt/stackstate/var/log`. In case of a two-node installation, logs are kept in the `var/log` directory on each node. Note that the logs are node-specific - the StackState node keeps StackState related logs and the StackGraph node keeps logs related to StackGraph.

## `var/log` details

Inside `var/log`, there is a subdirectory for each type of logs kept by StackState. By default, each subdirectory has a size cap of 2 GB; when exceeded, the oldest files are deleted. Example of `var/log` contents:

```
.:
drwxr-xr-x. 2 stackstate stackstate     4096 Dec 18 00:00 correlate
drwxr-xr-x. 2 stackstate stackstate     4096 Oct 23 00:00 elasticsearch
drwxr-xr-x. 2 stackstate stackstate    12288 Dec 18 00:00 elasticsearch7
drwxr-xr-x. 2 stackstate stackstate     4096 Dec 18 00:00 kafka
drwxr-xr-x. 2 stackstate stackstate     4096 Dec 18 00:20 kafka-to-es
drwxr-xr-x. 2 stackstate stackstate     4096 Nov 14 12:35 license-check
-rw-r--r--. 1 stackstate stackstate     2987 Dec 17 13:43 processmanager.2019-12-16.0.log
-rw-r--r--. 1 stackstate stackstate    65240 Dec 17 23:55 processmanager.2019-12-17.0.log
-rw-r--r--. 1 stackstate stackstate    10656 Dec 18 12:03 processmanager.log
drwxr-xr-x. 2 stackstate stackstate     4096 Oct 15 15:25 stackgraph
drwxr-xr-x. 2 stackstate stackstate     4096 Dec 16 09:03 stackpack
drwxr-xr-x. 2 stackstate stackstate     4096 Dec 17 13:21 stackstate
-rw-r--r--. 1 stackstate stackstate  6136841 Dec 16 23:59 stackstate.2019-12-16.11.log
-rw-r--r--. 1 stackstate stackstate 52429560 Dec 17 12:02 stackstate.2019-12-17.0.log
-rw-r--r--. 1 stackstate stackstate 52429586 Dec 17 20:39 stackstate.2019-12-17.1.log
-rw-r--r--. 1 stackstate stackstate 20252875 Dec 17 23:59 stackstate.2019-12-17.2.log
-rw-r--r--. 1 stackstate stackstate 52449712 Dec 18 00:37 stackstate.2019-12-18.0.log
-rw-r--r--. 1 stackstate stackstate 52441741 Dec 18 06:22 stackstate.2019-12-18.10.log
-rw-r--r--. 1 stackstate stackstate 52450906 Dec 18 07:27 stackstate.2019-12-18.11.log
-rw-r--r--. 1 stackstate stackstate 52478100 Dec 18 08:33 stackstate.2019-12-18.12.log
-rw-r--r--. 1 stackstate stackstate 52490055 Dec 18 09:37 stackstate.2019-12-18.13.log
-rw-r--r--. 1 stackstate stackstate 52428811 Dec 18 10:42 stackstate.2019-12-18.14.log
-rw-r--r--. 1 stackstate stackstate 52496521 Dec 18 11:48 stackstate.2019-12-18.15.log
-rw-r--r--. 1 stackstate stackstate 52445570 Dec 18 12:54 stackstate.2019-12-18.16.log
-rw-r--r--. 1 stackstate stackstate 52435555 Dec 18 01:10 stackstate.2019-12-18.1.log
-rw-r--r--. 1 stackstate stackstate 52453947 Dec 18 01:44 stackstate.2019-12-18.2.log
-rw-r--r--. 1 stackstate stackstate 52449896 Dec 18 02:19 stackstate.2019-12-18.3.log
-rw-r--r--. 1 stackstate stackstate 52429347 Dec 18 02:54 stackstate.2019-12-18.4.log
-rw-r--r--. 1 stackstate stackstate 52435159 Dec 18 03:29 stackstate.2019-12-18.5.log
-rw-r--r--. 1 stackstate stackstate 52486191 Dec 18 04:03 stackstate.2019-12-18.6.log
-rw-r--r--. 1 stackstate stackstate 52440210 Dec 18 04:38 stackstate.2019-12-18.7.log
-rw-r--r--. 1 stackstate stackstate 52436912 Dec 18 05:13 stackstate.2019-12-18.8.log
-rw-r--r--. 1 stackstate stackstate 52432884 Dec 18 05:47 stackstate.2019-12-18.9.log
-rw-r--r--. 1 stackstate stackstate   469460 Dec 18 12:54 stackstate.log
drwxr-xr-x. 2 stackstate stackstate     4096 Dec 18 00:00 stackstate-receiver
-rw-r--r--. 1 stackstate stackstate  1067419 Nov 20 10:04 stackstate-service.log
drwxr-xr-x. 2 stackstate stackstate     4096 Dec 18 12:49 sync
-rw-r--r--. 1 stackstate stackstate    50.0M Jan 11 13:27 transaction.metrics.0.log
-rw-r--r--. 1 stackstate stackstate    50.0M Jan 11 12:38 transaction.metrics.1.log
-rw-r--r--. 1 stackstate stackstate    50.0M Jan 11 11:48 transaction.metrics.2.log
-rw-r--r--. 1 stackstate stackstate    50.0M Jan 11 10:52 transaction.metrics.3.log
-rw-r--r--. 1 stackstate stackstate    50.0M Jan 11 10:01 transaction.metrics.4.log
-rw-r--r--. 1 stackstate stackstate    50.0M Jan 11 09:11 transaction.metrics.5.log
-rw-r--r--. 1 stackstate stackstate    50.0M Jan 11 08:21 transaction.metrics.6.log
-rw-r--r--. 1 stackstate stackstate    50.0M Jan 11 07:28 transaction.metrics.7.log
-rw-r--r--. 1 stackstate stackstate    50.0M Jan 11 06:28 transaction.metrics.8.log
-rw-r--r--. 1 stackstate stackstate    50.0M Jan 11 05:32 transaction.metrics.9.log
-rw-r--r--. 1 stackstate stackstate     9.2M Jan 11 13:38 transaction.metrics.log

./correlate:
-rw-r--r--. 1 stackstate stackstate 10389822 Dec 18 12:54 correlate.log
-rw-r--r--. 1 stackstate stackstate  3913232 Dec 18 12:54 worker.correlateConnections-1.log
-rw-r--r--. 1 stackstate stackstate  2629015 Dec 18 12:54 worker.correlateConnections-2.log
-rw-r--r--. 1 stackstate stackstate  3731346 Dec 18 12:54 worker.correlateConnections-3.log

./elasticsearch:
-rw-r--r--. 1 stackstate stackstate   3401 Oct 15 15:46 stackstate-2019-10-15.log
-rw-r--r--. 1 stackstate stackstate   3484 Oct 16 08:27 stackstate-2019-10-16.log
-rw-r--r--. 1 stackstate stackstate  50560 Oct 17 08:24 stackstate-2019-10-17.log
-rw-r--r--. 1 stackstate stackstate   3678 Oct 18 07:01 stackstate-2019-10-18.log
-rw-r--r--. 1 stackstate stackstate  32024 Oct 21 21:05 stackstate-2019-10-21.log
-rw-r--r--. 1 stackstate stackstate 129371 Oct 22 23:19 stackstate-2019-10-22.log
-rw-r--r--. 1 stackstate stackstate      0 Oct 15 15:46 stackstate_deprecation.log
-rw-r--r--. 1 stackstate stackstate      0 Oct 15 15:46 stackstate_index_indexing_slowlog.log
-rw-r--r--. 1 stackstate stackstate      0 Oct 15 15:46 stackstate_index_search_slowlog.log
-rw-r--r--. 1 stackstate stackstate   4664 Oct 23 10:29 stackstate.log

./elasticsearch7:
-rw-r--r--. 1 stackstate stackstate    75279 Dec 18 00:00 stackstate-2019-12-17-1.json.gz
-rw-r--r--. 1 stackstate stackstate    70897 Dec 18 00:00 stackstate-2019-12-17-1.log.gz
-rw-r--r--. 1 stackstate stackstate  2941900 Dec 18 10:27 stackstate_deprecation.json
-rw-r--r--. 1 stackstate stackstate  1932090 Dec 18 10:27 stackstate_deprecation.log
-rw-r--r--. 1 stackstate stackstate   145330 Dec 18 12:54 stackstate.log
-rw-r--r--. 1 stackstate stackstate   344879 Dec 18 12:54 stackstate_server.json

./kafka:
-rw-r--r--. 1 stackstate stackstate        0 Oct 15 15:46 authorizer.log
-rw-r--r--. 1 stackstate stackstate   538780 Dec 18 12:50 controller.log
-rw-r--r--. 1 stackstate stackstate 11640758 Nov 20 10:04 kafka-gc.log
-rw-r--r--. 1 stackstate stackstate    15856 Dec 17 16:02 log-cleaner.log
-rw-r--r--. 1 stackstate stackstate        0 Oct 15 15:46 request.log
-rw-r--r--. 1 stackstate stackstate   397450 Dec 18 12:45 server.log
-rw-r--r--. 1 stackstate stackstate   204722 Dec 17 13:45 state-change.log

./kafka-to-es:
-rw-r--r--. 1 stackstate stackstate   180046 Dec 18 12:54 kafkaEventsToES.log
-rw-r--r--. 1 stackstate stackstate   202857 Dec 17 23:58 kafkaEventsToES.log.2019-12-17.0
-rw-r--r--. 1 stackstate stackstate    13164 Dec 18 12:23 kafkaMultiMetricsToES.log
-rw-r--r--. 1 stackstate stackstate    81363 Dec 17 23:55 kafkaMultiMetricsToES.log.2019-12-17.0
-rw-r--r--. 1 stackstate stackstate    19358 Dec 18 12:18 kafkaStsEventsToES.log
-rw-r--r--. 1 stackstate stackstate   141928 Dec 17 23:52 kafkaStsEventsToES.log.2019-12-17.0

./license-check:
-rw-r--r--. 1 stackstate stackstate 4928 Nov 14 12:39 license-app.log

./stackgraph:

./stackpack:
-rw-r--r--. 1 stackstate stackstate   906 Dec 13 13:57 agent-common.2019-12-13.0.log
-rw-r--r--. 1 stackstate stackstate  4990 Dec 16 09:11 agent-common.log
-rw-r--r--. 1 stackstate stackstate  1818 Dec 16 09:03 aws.log
-rw-r--r--. 1 stackstate stackstate  3161 Dec 13 14:02 common.2019-12-13.0.log
-rw-r--r--. 1 stackstate stackstate  5418 Dec 16 09:11 common.log
-rw-r--r--. 1 stackstate stackstate  1354 Dec 13 14:02 k8s-common.log
-rw-r--r--. 1 stackstate stackstate  4274 Dec 13 14:03 kubernetes.log
-rw-r--r--. 1 stackstate stackstate   533 Oct 21 09:26 stackstate-agent-v2.2019-10-21.0.log
-rw-r--r--. 1 stackstate stackstate 72850 Dec 16 09:11 stackstate-agent-v2.log

./stackstate:

./stackstate-receiver:
-rw-r--r--. 1 stackstate stackstate 29430771 Dec 17 23:59 stackstate-receiver.2019-12-17.0.log
-rw-r--r--. 1 stackstate stackstate 11586394 Dec 18 12:54 stackstate-receiver.log

./sync:
total 1043088
-rw-r--r--. 1 stackstate stackstate 24768357 Dec 17 23:59 sync.Agent.2019-12-17.9.log
-rw-r--r--. 1 stackstate stackstate 52430757 Dec 18 02:31 sync.Agent.2019-12-18.0.log
-rw-r--r--. 1 stackstate stackstate 52428990 Dec 18 05:11 sync.Agent.2019-12-18.1.log
-rw-r--r--. 1 stackstate stackstate 52430847 Dec 18 07:48 sync.Agent.2019-12-18.2.log
-rw-r--r--. 1 stackstate stackstate 52430381 Dec 18 10:26 sync.Agent.2019-12-18.3.log
-rw-r--r--. 1 stackstate stackstate 45954315 Dec 18 12:54 sync.Agent.log
-rw-r--r--. 1 stackstate stackstate   383053 Dec 17 23:13 sync.AWS Sync for Account - 508573134510.2019-12-17.0.log
-rw-r--r--. 1 stackstate stackstate   160967 Dec 18 12:13 sync.AWS Sync for Account - 508573134510.log
-rw-r--r--. 1 stackstate stackstate   248557 Dec 17 23:30 sync.AWS Sync for Account - 766509113410.2019-12-17.0.log
-rw-r--r--. 1 stackstate stackstate   120016 Dec 18 12:30 sync.AWS Sync for Account - 766509113410.log
-rw-r--r--. 1 stackstate stackstate 52494635 Dec 18 05:12 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.2019-12-18.1.log
-rw-r--r--. 1 stackstate stackstate 52490024 Dec 18 07:48 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.2019-12-18.2.log
-rw-r--r--. 1 stackstate stackstate 52450773 Dec 18 10:22 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.2019-12-18.3.log
-rw-r--r--. 1 stackstate stackstate 52478535 Dec 18 12:49 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.2019-12-18.4.log
-rw-r--r--. 1 stackstate stackstate  2014902 Dec 18 12:54 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.log
-rw-r--r--. 1 stackstate stackstate 52438711 Dec 17 20:39 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-17.5.log
-rw-r--r--. 1 stackstate stackstate 50769208 Dec 17 23:59 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-17.6.log
-rw-r--r--. 1 stackstate stackstate 52497481 Dec 18 03:26 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-18.0.log
-rw-r--r--. 1 stackstate stackstate 52530973 Dec 18 06:53 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-18.1.log
-rw-r--r--. 1 stackstate stackstate 52539980 Dec 18 10:20 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-18.2.log
-rw-r--r--. 1 stackstate stackstate 39488493 Dec 18 12:54 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.log
-rw-r--r--. 1 stackstate stackstate 52435381 Dec 17 15:23 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-17.2.log
-rw-r--r--. 1 stackstate stackstate 52429788 Dec 17 20:32 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-17.3.log
-rw-r--r--. 1 stackstate stackstate 35254609 Dec 17 23:59 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-17.4.log
-rw-r--r--. 1 stackstate stackstate 52435568 Dec 18 05:08 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-18.0.log
-rw-r--r--. 1 stackstate stackstate 52456809 Dec 18 10:14 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-18.1.log
-rw-r--r--. 1 stackstate stackstate 27479646 Dec 18 12:54 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.log
-rw-r--r--. 1 stackstate stackstate      800 Dec 16 09:19 sync.Mysql.2019-12-16.0.log
-rw-r--r--. 1 stackstate stackstate      798 Dec 17 13:52 sync.Mysql.log
-rw-r--r--. 1 stackstate stackstate      805 Dec 16 09:19 sync.Postgresql.2019-12-16.0.log
-rw-r--r--. 1 stackstate stackstate      805 Dec 17 13:52 sync.Postgresql.log
```

{% hint style="info" %}
From StackState v1.15.0, the version of Elasticsearch used by StackState changed. Elasticsearch logs are now saved in `./elasticsearch7`. The old `./elasticsearch` subdirectory can be removed to free some disk space.
{% endhint %}

## Log files

StackState keeps logs in files that have a maximum size of 50 MB. When a log file exceeds 50 MB size cap, it's divided into ordered parts, as in the below example:

```
./sync:
total 1043088
-rw-r--r--. 1 stackstate stackstate 52430757 Dec 18 02:31 sync.Agent.2019-12-18.0.log
-rw-r--r--. 1 stackstate stackstate 52428990 Dec 18 05:11 sync.Agent.2019-12-18.1.log
-rw-r--r--. 1 stackstate stackstate 52430847 Dec 18 07:48 sync.Agent.2019-12-18.2.log
-rw-r--r--. 1 stackstate stackstate 52430381 Dec 18 10:26 sync.Agent.2019-12-18.3.log
```

## Default log pattern

StackState builds log files using the following default log pattern: `"%date [%thread] %-5level %logger{60} - %msg%n"`


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://archivedocs.stackstate.com/5.1/configure/logging/linux-logs.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.