Logging

The thingsHub follows the recommendations for Twelve Factor Applications for handling log files. This means, all logs are written to stdout and thereby passed directly to Kubernetes without storing them on disk. The further handling of these logs is the the cluster’s responsibility. On most Kubernetes clusters, logs which are generated via streaming to a container’s stdout, are stored on disk at, /var/logs/pods, with the name following the pattern /var/logs/pods/<pod-uuid>/<container-name>/<restart-counter>.log. For example,/var/log/pods/120cb494-472d-11e9-bafb-42010a9c0046/engine/0.log contains the logs for a tenant's engine service.

Inspecting Log Files

In Kubernetes kubectl command line tool, a pod’s logs can be viewed with the command

CODE

kubectl logs -n <namespace> <pod-name> [<container-name>]

In this command <namespace is the Kubernetes namespace, which is usually identifcal to the tenant name. Furthermore, the container name can be left out if there is only a single container in the pod. Because pods in the thingsHub’s pods are managed by deployments, and this adds random suffixes to pod names, it is usually more convenient to use labels for identifying the right pods more quickly:

CODE

kubectl logs -n <namespace> `kubectl -n <namespace> get pods -lapp=engine`

Finding Log Files

Should it not be possible to use kubectl to access the logs or should direct access to the files be required the can be access via /var/log/pods directly, however it is required to know the pod UI for this. This UID can be retrieved for a pod using the following command:

CODE

kubectl --namespace <namespace> get pod <podname> -o=jsonpath='{@.metadata.uid}'

If the pod name is not known, a label can be used instead to identify the right pod and find it’s UID:

CODE

kubectl get pods -n <namespace> -l=app=<deployment> -o jsonpath='{@.items[0].metadata.uid}'

In this command <namespace> is the tenant name while <deployment> indicates the service inside the tenant. This can be one of engine, iam, visualizer, ui, and a few more.

Note that the log files are present only on the node on which the pod is currently scheduled.

Log File Format

This means logs must be collected from all involved nodes.

Logs are printed out in a stackdriver compatible envelope format:

CODE

{"log":"{\"message\":\"received message\",\"module\":\"engine.components.messages.dispatcher\",\"ns_device_id\":\"A81758FFFE0364E4\",\"severity\":\"info\",\"time\":\"2019-03-26T12:53:11.878752687Z\"}\n","stream":"stdout","time":"2019-03-26T12:53:11.879040408Z"}

With proper indentation this looks like this:

CODE

{
  "log": "{\"message\":\"received message\",\"module\":\"engine.components.messages.dispatcher\",\"ns_device_id\":\"A81758FFFE0364E4\",\"severity\":\"info\",\"time\":\"2019-03-26T12:53:11.878752687Z\"}\n",
  "stream":"stdout",
  "time":"2019-03-26T12:53:11.879040408Z"
}

The log field contains the applications log message, again encoded in JSON:

CODE

{
  "time": "2019-03-26T12:53:11.878752687Z",
  "severity": "info",
  "module": "engine.components.messages.dispatcher",
  "message": "received message",
  "ns_device_id": "A81758FFFE0364E4"
}

The time field is encoded according to RFC3339, The severity can be any of "debug", "info", "warning", and "error".

Field	Type	Description
`time`	string, RFC3339 encoded	The log message's timestamp
`severity`	string, any of (debug, info, warning, error)	The log event's severity.
`module`	string	The golang package from which the log message originated.
additional fields	any	The existence and contents of additional fields depend on the specific context and purpose of the log message.

Rotation of Log Files

The rotation of log files is not done by the thingsHub or any of its services, but is responsibility of the Kubernetes cluster deployment tool, e.g. Rancher or GKE. If log files are not rotated, pods will be Evicted once the required disk space exceeds the ephemeral disk space budget.

Forwarding Logs to External Logging Servers

SmartMakers recommends using a cluster-level logging architecture with a Node Logging Agent. Fluentd is the recommended choice, if it provides a suitable plugin, for sending logs to outside of the cluster. For syslog, an easy-to-use guide is provided by Georg Goh.

Inspecting Log Files

Finding Log Files

Log File Format

Rotation of Log Files

Forwarding Logs to External Logging Servers

References