Logging
The thingsHub follows the recommendations for Twelve Factor Applications for handling log files. This means, all logs are written to stdout
and thereby passed directly to Kubernetes without storing them on disk. The further handling of these logs is the the cluster’s responsibility. On most Kubernetes clusters, logs which are generated via streaming to a container’s stdout, are stored on disk at, /var/logs/pods
, with the name following the pattern /var/logs/pods/<pod-uuid>/<container-name>/<restart-counter>.log
. For example,/var/log/pods/120cb494-472d-11e9-bafb-42010a9c0046/engine/0.log
contains the logs for a tenant's engine service.
Inspecting Log Files
In Kubernetes kubectl
command line tool, a pod’s logs can be viewed with the command
kubectl logs -n <namespace> <pod-name> [<container-name>]
In this command <namespace
is the Kubernetes namespace, which is usually identifcal to the tenant name. Furthermore, the container name can be left out if there is only a single container in the pod. Because pods in the thingsHub’s pods are managed by deployments, and this adds random suffixes to pod names, it is usually more convenient to use labels for identifying the right pods more quickly:
kubectl logs -n <namespace> `kubectl -n <namespace> get pods -lapp=engine`
Finding Log Files
Should it not be possible to use kubectl to access the logs or should direct access to the files be required the can be access via /var/log/pods
directly, however it is required to know the pod UI for this. This UID can be retrieved for a pod using the following command:
kubectl --namespace <namespace> get pod <podname> -o=jsonpath='{@.metadata.uid}'
If the pod name is not known, a label can be used instead to identify the right pod and find it’s UID:
kubectl get pods -n <namespace> -l=app=<deployment> -o jsonpath='{@.items[0].metadata.uid}'
In this command <namespace>
is the tenant name while <deployment>
indicates the service inside the tenant. This can be one of engine, iam, visualizer, ui, and a few more.
Note that the log files are present only on the node on which the pod is currently scheduled.
Log File Format
This means logs must be collected from all involved nodes.
Logs are printed out in a stackdriver compatible envelope format:
{"log":"{\"message\":\"received message\",\"module\":\"engine.components.messages.dispatcher\",\"ns_device_id\":\"A81758FFFE0364E4\",\"severity\":\"info\",\"time\":\"2019-03-26T12:53:11.878752687Z\"}\n","stream":"stdout","time":"2019-03-26T12:53:11.879040408Z"}
With proper indentation this looks like this:
{
"log": "{\"message\":\"received message\",\"module\":\"engine.components.messages.dispatcher\",\"ns_device_id\":\"A81758FFFE0364E4\",\"severity\":\"info\",\"time\":\"2019-03-26T12:53:11.878752687Z\"}\n",
"stream":"stdout",
"time":"2019-03-26T12:53:11.879040408Z"
}
The log
field contains the applications log message, again encoded in JSON:
{
"time": "2019-03-26T12:53:11.878752687Z",
"severity": "info",
"module": "engine.components.messages.dispatcher",
"message": "received message",
"ns_device_id": "A81758FFFE0364E4"
}
The time field is encoded according to RFC3339, The severity can be any of "debug", "info", "warning", and "error".
Field | Type | Description |
---|---|---|
| string, RFC3339 encoded | The log message's timestamp |
| string, any of (debug, info, warning, error) | The log event's severity. |
| string | The golang package from which the log message originated. |
additional fields | any | The existence and contents of additional fields depend on the specific context and purpose of the log message. |
Rotation of Log Files
The rotation of log files is not done by the thingsHub or any of its services, but is responsibility of the Kubernetes cluster deployment tool, e.g. Rancher or GKE. If log files are not rotated, pods will be Evicted once the required disk space exceeds the ephemeral disk space budget.
Forwarding Logs to External Logging Servers
SmartMakers recommends using a cluster-level logging architecture with a Node Logging Agent. Fluentd is the recommended choice, if it provides a suitable plugin, for sending logs to outside of the cluster. For syslog, an easy-to-use guide is provided by Georg Goh.