Timestamps

Timestamp Definitions in the thingsHub Data Pipeline

To ensure data integrity and provide clear diagnostics, it is essential to understand the different timestamps generated as data moves from a device to our platform. Each timestamp marks a specific point in the data ingestion pipeline, created by different systems (the device, the network, or our platform).

Here is a breakdown of the key timestamps you will encounter.

1. Device-Side Timestamps

These timestamps are generated by the end-device itself. When available, they are often the "source of truth" for when a real-world event actually occurred.

• Data Packet Time

  • What it is: The timestamp the device's firmware attaches to the data payload before transmission. This often represents the moment a sensor reading was taken or an event was recorded.

  • Source: The device's or sensor's main processor/clock (e.g., Real-Time Clock or RTC).

  • Key Considerations:

    • This is invaluable for understanding the true event time.

    • Not all devices provide this timestamp due to power, cost, or hardware limitations. Especially the LoRa devices usually don’t include timestamps from the device side due to power constraints.

    • Example: A Systemloco device records lightLevel at a specific time and includes this timestamp in the data packet.

2. Network-Side Timestamps

These timestamps are generated by the network infrastructure (gateways and network servers) responsible for moving the data from the device to thingsHub. They are essential for diagnosing connectivity and latency issues.

• Gateway Reception Time

  • What it is: The moment a gateway (e.g., a LoRaWAN gateway) receives the radio frequency (RF) data packet from the device.

  • Source: The gateway's system clock. Usually this can be the first calculated timestamp, for example in many of the LoRa based devices.

  • Usage: Usually, we use this timestamp when we split out multiple receptions into multiple uplinks, or when we pick the best reception. This timestamp becomes the core time for the uplink message in several scenarios.

  • Key Considerations:

    • This is the first timestamp generated by the network backend infrastructure.

    • A single device uplink can be received by multiple gateways. Each reception will have its own Gateway Reception Time, which the Network Server uses to de-duplicate the signal.

    • This is applicable mostly to the LoRa-based devices and networks. And not all devices will be connected through Gateways. For example, Digital Matter Edge devices are connected to the OEM server directly through a cellular network. In this case, there will be no multiple receptions in the uplink.

• Uplink Time or Server Time

  • What it is: The moment the Network Server (e.g., The Things Network, ChirpStack) successfully receives the data from one or more gateways/devices and processes it into a single, de-duplicated uplink for that specific device.

  • Source: The Network Server's system clock

  • Usage: For decoding data by the driver, we use the time from the internal message. The internal message time is populated either from the timestamp available in the device data payload or, if it is not available, the time will be picked from the best receptions if there’s multiple reception, and if this is also not available, fallback to the server uplink time.

  • Key Considerations:

    • Usually, this is the source of truth for all further operations in the pipeline based on the uplink.

    • This timestamp marks the completion of the "radio" portion of the data's journey as it arrives in the network server.

3. Platform-Side Timestamps (thingsHub)

These timestamps are used by our thingsHub platform as it ingests and processes the data.

• thingsHub Reception Time

  • What it is: The moment our thingsHub platform's "front door" (e.g., web-hook API endpoint to receive uplinks from the network server) first receives the raw, non-decoded uplink from the Network Server.

  • Source: The thingsHub application server's clock

  • Usage: Usually, we use it in a distinct field in the thingsHub uplink message that we store in our time series database. We usually do not use this timestamp for any time critical purpose as this timestamp is generated in our platform, and not from the devices, gateways or the network servers.

  • Key Considerations:

    • This timestamp is critical for measuring the Network Server-to-thingsHub latency.

    • It marks the "handoff" where the data enters our specific application environment.

• Decoded Data Time

  • What it is: This is not a new timestamp generated by thingsHub. It is the primary timestamp that thingsHub selects to associate with the final, decoded data in our database and further processing.

  • Source: This is a logical reference. thingsHub typically uses the Uplink Time as the source of truth, which can come either from the receptions or from the network server itself or from the device data payload. For decoding data by the driver, we use the internal message time. The internal message time is populated either from the device data frame, Uplink Time or from the Reception Time, based on the type of devices

  • Usage: The decoded data record’s time uses this timestamp for processing, storing, and forwarding.

  • Key Considerations:

    • This is a crucial design choice. By using an upstream timestamp (like Gateway Reception Time or Device Data Time), we ensure that internal processing delays (like decoding or database write-lags) do not affect or alter the official time of the data point.

    • This makes all data reporting consistent, as it's anchored to the time the network received it, not the time our platform finished processing it.