Migration Guide: Transitioning from Legacy Data Tables to Extended Data Tables

Introduction

Purpose:

This guide provides step-by-step instructions on how to migrate your thingsHub data from legacy "auto data tables" to the improved "extended data tables".

Importance:

Migrating from legacy auto data tables will ensure you benefit from the latest features, bug fixes, and improved data retention offered by the newer extended data tables.

Legacy Data Tables:
1. Also called auto data tables, these tables have been deprecated from thingsHub version 4.13 and are not actively updated anymore.
2. We cannot guarantee that legacy data tables will have new features associated with them in the future.
3. They will be completely removed from thingsHub 6, and it is recommended that customers migrate these data tables to the new extended data tables.
Extended Data Tables:
1. These have been introduced in thingsHub 4.13, and we recommend that customer tenants migrate legacy data tables to extended data tables.
2. There have been data retention issues with the legacy data tables which have been fixed in extended data tables.
3. In addition, any identified bugs will be resolved for the new data tables, but not for the legacy data tables from thingshub 6.0.

Scope: This guide covers the migration process for existing auto data tables. It does not address the creation of new tables.

Prerequisites

thingsHub Version: You must be running thingsHub version 5.3 or later to utilize the migration tools.
User Permissions: admin or tenant owner access to your thingsHub tenant is required.
Understanding of APIs: Basic familiarity with REST APIs and API calls will help use the migration tools.

API Migration Tools

thingsHub API provides tools to facilitate the migration process, which are available in the thingsHub swagger which can be accessed at https://{tenant_url}/api/index.html:

/api/v3/tenant/migrations:

Method: GET

Retrieves a list of available migration tasks with their descriptions.

Response:

JSON

{
  "collection": [
    {
      "description": "Identifies the dangling visualizer in the system, and patches them",
      "id": "script1",
      "name": "Dangling Visualizer Integration Patch"
    },
    {
      "description": "Identifies the legacy datatables in the system, and migrates them to new datatables",
      "id": "script2",
      "name": "Migrate Legacy Datatables to New Datatables"
    }
  ]
}

Currently available Migration Script Endpoints:

Script 1: Patch dangling visualizer

Endpoint: /api/v3/tenant/migrations/script1

Method: GET

Returns a list of "dangling visualizer integrations" within the tenant, and the status of the migration if it has been run before.

Response:

JSON

{
  "status": "success"                       // the status of the migration run ("success"/"running"/"failed", this field will not appear if the migration has not been run yet)
  "start_time": "2024-03-02T04:52:54.089Z",// the time when the migration was started (if the migration was previously run, otherwise this field will not appear)
  "end_time": "2024-03-02T04:52:54.089Z",  // the time when the migration ended
  "errors": [                              // the errors encountered during the migration run (if there were any)
    "Error 1",
    "Error 2"
  ],
  "items": {
    "affected": [                          // the dangling visualizers id:name that will be affected by the migration
      "1:dangling-visualizer-1",
      "2:dangling-visualizer-2",
      "3:dangling-visualizer-3",
    ]
  },                            
  "migration": {                           // the migration description
    "description": "Identifies the dangling visualizer in the system, and patches them",
    "id": "script1",
    "name": "Dangling Visualizer Integration Patch"
  }
}

Method: PATCH

Initiates the patch that deletes the dangling visualizer, and returns the status of the migration.

Response:

JSON

{
  "status": "success"                       // the status of the migration run ("success"/"running"/"failed")
  "start_time": "2024-03-02T04:52:54.089Z",// the time when the migration was started
  "end_time": "2024-03-02T04:52:54.089Z",  // the time when the migration ended
  "errors": [                              // the errors encountered during the migration run (if there were any)
    "Error 1",
    "Error 2"
  ],
  "items": {
    "affected": [                          // the dangling visualizers id:name that were affected by the migration
      "1:dangling-visualizer-1",
      "2:dangling-visualizer-2",
      "3:dangling-visualizer-3",
    ]
  },                            
  "migration": {                           // the migration description
    "description": "Identifies the dangling visualizer in the system, and patches them",
    "id": "script1",
    "name": "Dangling Visualizer Integration Patch"
  }
}

Script 2: Migrate auto data tables

Endpoint: /api/v3/tenant/migrations/script2

Method: GET

Retrieves a list of legacy data tables within the tenant.

Response:

JSON

{
  "status": "success"                       // the status of the migration run ("success"/"running"/"failed", this field will not appear if the migration has not been run yet)
  "start_time": "2024-03-02T04:52:54.089Z",// the time when the migration was started (if the migration was previously run, otherwise this field will not appear)
  "end_time": "2024-03-02T04:52:54.089Z",  // the time when the migration ended
  "errors": [                              // the errors encountered during the migration run (if there were any)
    "Error 1",
    "Error 2"
  ],
  "items": {
    "affected": [                          // the dangling visualizers id:name that will be affected by the migration
      "1:dangling-visualizer-1",
      "2:dangling-visualizer-2",
      "3:dangling-visualizer-3",
    ]
  },                            
  "migration": {                           // the migration description
    "description": "Identifies the dangling visualizer in the system, and patches them",
    "id": "script1",
    "name": "Dangling Visualizer Integration Patch"
  }
}

Method: PATCH

Migrates all legacy auto data tables to extended data tables, and returns the status of the migration

Response:

JSON

{
  "status": "success"                       // the status of the migration run ("success"/"running"/"failed", this field will not appear if the migration has not been run yet)
  "start_time": "2024-03-02T04:52:54.089Z",// the time when the migration was started (if the migration was previously run, otherwise this field will not appear)
  "end_time": "2024-03-02T04:52:54.089Z",  // the time when the migration ended
  "errors": [                              // the errors encountered during the migration run (if there were any)
    "Error 1",
    "Error 2"
  ],
  "items": {
    "affected": [                          // the auto data tables affected by this migration
      "auto-data-table-1:auto-data-table-1",
      "auto-data-table-2:auto-data-table-2",
      "auto-data-table-3:auto-data-table-3"
    ]
  },                          
  "migration": {                            // the migration description
    "description": "Identifies the legacy data tables in the system, and migrates them to extended data tables",
    "id": "script2",
    "name": "Legacy Data Table Migration"
  }
}

Steps to run the migration

Please note that you must be logged in as a user with the role tenant:owner or admin to run the API migration

Patch dangling visualizers
1. Send a GET request to /api/v3/tenant/migrations/script1 and view the response’s affected items to verify the visualizer integrations that will be deleted by this migration.
2. Send a PATCH request to /api/v3/tenant/migrations/script1 to apply the migration. If you encounter any error messages during this process, please contact smartmakers support.
Migrate auto data tables
1. Send a GET request to /api/v3/tenant/migrations/script2 and view the response’s affected items to verify the auto data tables that will be migrated to extended data tables.
2. Send a PATCH request to /api/v3/tenant/migrations/script2 to apply the migration. If you encounter any error messages during this process, please contact smartmakers support.
Restart the tsdb pod
Since this is a database-level change, you will need to restart the pod for tsdb in your kubernetes cluster to stimulate the pod to sync these changes.