Asset Synchronization with an External System

Overview

In order to be tracked, an asset must be stored in the Asset Inventory and bound to an asset tracker/tag.

The Asset Synchronization feature allows syncing the Asset Inventory with an external system and replaces the manual process of creating/storing Asset in the Asset Inventory. While Asset Synchronization syncs the asset inventory, it is not just bulk creation of asset. Based on the current state of Asset Inventory in ThingsHub and input data in the synchronization request, the following effects can happen to the assets in Asset Inventory:

If an asset is already present in the Asset Inventory and is not provided in the Input JSON Collection, it will be deleted from the Asset Inventory.
If an asset information is provided in the Input JSON Collection but it is not present in the Asset Inventory, a new asset will be created in the Asset Inventory.
If an asset is already present in the Asset Inventory and it is also provided in the Input JSON Collection, it will be updated in the Asset Inventory with the newer data from Input JSON Object.

Let us say your external system emits the following CSV data which you want to synchronize as assets in Thingshub

convertcsv.xlsx

Asset Synchronization Terminology

Asset: Any physical object that can be tracked by using various types of tracking devices.
Asset ID: Unique identification of the asset.
Asset Name: Case sensitive name for the asset.
Metadata: Key and value type metadata that is used to logically organize and group assets by common characteristics.
Asset Import: Asset Synchronization request.
Push Synchronization: On demand synchronization request.
Pull Synchronization: Scheduled synchronization request

Input JSON Collection: The list of JSON objects provided as input for Asset Import
Example

CODE

[
    {
      "uid": "12345",
      "manufacturer": "Bosch",
      "type": "power drill"
    },
    {
      "uid": "67890",
      "manufacturer": "Black&Decker",
      "type": "power drill"
    }
  ]

Input JSON Object: A single object in the Input JSON Collection that will be mapped to a singular asset.
Example
CODE
```
{
  "uid": "12345",
  "manufacturer": "Bosch",
  "type": "power drill"
}
```

Asset ID Constraints

The Asset ID needs to be unique.
The Asset ID can contain alpha-numeric value and dashes.
The Asset ID starts and ends with an alpha-numeric value.
The Asset ID does not contain consecutive dashes.
The Asset ID can be as long as 61 characters.

Configuring Synchronization

The Asset Synchronization feature can be configured by the Admin or Tenant Operator using the Tenant Configuration yaml file. The push-synchronization is always enabled in Thingshub however, primary_id is required or Thingshub will throw error.
To enable the pull-synchronization, both schedule and url fields need to be set in the tenant configuration file.

Thingshub allows following configuration options for Asset Synchronization.

YAML

global:
  name: thingshub-tenant
  tenant_mode: trackinghub
  
  image:
    registry_type: gcr
    registry: eu.gcr.io
    repository: valid-moment-173015/things-hub
  ...
  
trackinghub:
  enabled: true
  synchronization:
    primary_id: "uid" // field from the input to be mapped to Asset ID
    schedule: // cron-job schedule
    url: // url that returns Input Json Collection

Configuration Options:

primary_id :
- The field in the Input JSON Object that will be mapped to Asset ID.
- Required for both pull-synchronization and push-synchronization
schedule :
- The cron schedule to run a pull-synchronization.
- Required for pull-synchronization
url :
- The endpoint from the 3rd party system, that returns valid Input JSON Object for a GET request.
- Required for pull-synchronization

The field set as the primary_id will be mapped to the Asset ID, so the Operater should make sure that the values of primary_id follow the Asset ID Constraints.

In our example excel data, the uid field satisfies all the Asset ID Constraints, so we have configured Thingshub to be use uid as primary_id.

Push Synchronization

You can synchronize your systems assets to the Thingshub System on demand using the REST API. This on demand synchronization is known as push-synchronization.
Push synchronization is available in the api/v3/assets-import REST endpoint.

POST api/v3/assets-import : initiate a push synchronization with the assets data in the request body.
GET api/v3/assets-import : get the status of the last asset-synchronization request.

Rest APIs for asset-import (push-synchronization)

To be able to run the API requests you’ll have to be authorized and have appropriate permissions. For authentication, the Tenant Owner will have to generate an API key first. The generated API key will be used in the authorization HTTP header, e.g. Authorization: Bearer <token> while making the above API calls.

Initiating Push Synchronization

To initiate a push synchronization request, a POST request is made to the api/v3/assets-import endpoint with all the assets information present as input JSON collection in the body.

Input Data Format

The JSON collection passed as input to the Push Synchronization request needs to follow some constraints.

All the fields in an input JSON object of the input JSON collection are processed as strings, so the input JSON objects in the collection need to be flat (i.e. Nested JSON objects are not supported).
Each input JSON object need to have the field that corresponds to the primary_id configuration in the tenant configuration.
The value of the field to be mapped as Asset ID needs to follow the Asset ID Constraints.

JSON

[
   {
    "uid": "233",
    "AuftragsNr": "50000",
    "lisence": "WVWZZZ1KZ7XXX443",
    "Hersteller": "VW",
    "Modell": "Polo",
    "description": "valid input JSON object"
  },
  {
    "uid": "6798",
    "name": "Tiaguan Car",
    "AuftragsNr": "699028",
    "lisence": "WVJKEZ1KZ7XXX443",
    "Hersteller": "VW",
    "Modell": "Tiguan",
    "description": "valid input JSON object"
  },
  {
    "AuftragsNr": "602838",
    "lisence": "NBEWZZZ1KZ7XXX443",
    "Hersteller": "VW",
    "Modell": "T-Roc",
    "description": "input JSON object without primary_id field: uid"
  },
  {
    "uid": "Fiat/67890",
    "AuftragsNr": "03985",
    "lisence": "KJEFZ1KZ7XXX443",
    "Hersteller": "Fiat",
    "Modell": "500L",
    "description": "input JSON object with invalid value of primary_id field: uid"
  }
]

Asset Creation From Input

Each input JSON object in the input JSON collection will have different fields and values. Among these fields, 2 special fields are treated as Asset ID and Asset Name of the asset being created and rest of the fields are added as Metadata in assets.

The field set as primary_id in the tenant config will be mapped to Asset ID. Any field in the input that follow the Asset ID Constraints can be set as primary_id.

If there is a name field present in the input JSON object, it will be mapped as Asset Name. If the name field is absent in the input JSON object, the Asset ID from above will be used as Asset Name as well.

Examples:

Without name field:

JSON

[
  {
    "uid": "233",
    "AuftragsNr": "50000",
    "lisence": "WVWZZZ1KZ7XXX443",
    "Hersteller": "VW",
    "Modell": "Polo"
  }
]

With name field:

CODE

[
  {
    "uid": "6798",
    "name": "Tiaguan Car",
    "AuftragsNr": "699028",
    "lisence": "WVJKEZ1KZ7XXX443",
    "Hersteller": "VW",
    "Modell": "Tiguan",
  }
]

Invalid JSON Object:
A valid JSON Object can be invalid input for asset-import if one of the 3 conditions meet:

The value of primary_id field, ( i.e. from which Asset ID is to be created), is absent in the JSON input object.
The value of primary_id field, ( i.e. from which Asset ID is to be created), do not follow the Asset ID Constraints.
The value of primary_id field, ( i.e. from which Asset ID is to be created), is duplicated in the input JSON Collection.

These invalid input JSON objects are skipped and their corresponding assets are not created during the push-synchronization. The corresponding objects in the input JSON collection are marked and returned along with asset-import status.

Pull Synchronization

Thingshub can also be configured to synchronize the Asset Inventory with an external system in regular intervals. This scheduled synchronization is known as pull-synchronization.

The pull-synchronization needs to have a schedule that will define the intervals in which the synchronization is run. This can be set using the schedule configuration field and follows the syntax of cron jobs.

When the next synchronization is to be run, Thingshub will retrieve Input JSON Collection from the address set in the url configuration field using a HTTP GET request. So the address will have to be setup to return JSON collection object that follows some constraints. (Please refer to the Input Data Format section of Push Synchronization for further details.)

Once the Input JSON Collection is received, it is processed in the similar way as done in push synchronization. (Please refer to the Asset Creation From Input section of Push Synchronization for further details.)

Getting the Status of Last Synchronization.

The push-synchronization initiation POST request will immediately return once it has been fully parsed, while the actual synchronization happens in the background. The POST request will return the immediate status of the requested synchronization.

For the pull-synchronization, the synchronization happens in the background in the set schedule.

You can also track the progress of the synchronization using the GET api/v3/assets-import endpoint. The request will return brief summary of the last synchronization request as the status.

Response:

JSON

{
  "counts": {
    "duplicated": 0,
    "failed": 0,
    "imported": 2,
    "in_progress": 0,
    "invalid_id_format": 1,
    "total": 4,
    "without_primary_id": 1
  },
  "errors": [],
  "ignored": {
    "duplicated": [],
    "invalid_id_format": [
      {
        "uid": "Fiat/67890",
        "AuftragsNr": "03985",
        "lisence": "KJEFZ1KZ7XXX443",
        "Hersteller": "Fiat",
        "Modell": "500L",
        "description": "input JSON object with invalid value of primary_id field: uid"
      }
    ],
    "without_primary_id": [
      {
        "AuftragsNr": "602838",
        "lisence": "NBEWZZZ1KZ7XXX443",
        "Hersteller": "VW",
        "Modell": "T-Roc",
        "description": "input JSON object without primary_id field: uid"
      }
    ]
  },
  "last_import": {
    "ended": "2024-03-13T03:19:13.838752603Z",
    "requested": "2024-03-13T03:19:13.838666844Z",
    "started": "2024-03-13T03:19:13.838667384Z"
  },
  "running": false
}

counts :
Counts object in the response provides a summary of how many assets were requested for synchronization, how many of those succeeded, how may failed and how many of those were ignored. The number of ignored assets is further distributed based on the reason, i.e. duplicated, without_primary_id, or invalid_id_format.
errors :
If the ThingsHub System encounters any errors during parsing the input or processing the synchronization request, they are included in this errors list in the response.
ignored : Ignored object provides the summary of all the input JSON objects that were skipped.
- duplicated : The duplicated block holds all the input JSON objects that were skipped because their primary_id value was duplicated and was not unique.
- invalid_id_format : The invalid_id_format block holds all the input JSON objects that were skipped because their primary_id value did not follow the Asset ID Constraints.
- without_primary_id: The without_primary_id block holds all the input JSON objects that were skipped because their primary_id field was missing.
last_import:
The last_import object provides the timing information about the push-synchronization request.
running :
The running field is a boolean that indicates if the synchronization has completed or not.

The status of last synchronization request is also visible in the Thingshub UI in the Asset Inventory page.