Compatibility with Kubernetes

Tested Compatibility Table

How do we test for compatibility?

Premises:

• All of the compatibility tests are performed in the Google Kubernetes Engine from GCP. When the compatibility tests are done on other platforms, it is explicitly mentioned.

• The latest available thingshub patch version under each major release is tested for compatibility.

• The testing is done with the latest available Kubernetes patch version unless explicitly mentioned.

• Each Thingshub version is tested in isolation.

For each major Kubernetes release:

• We go through the release notes and mark the changes that could be related to thingshub. Then we verify if each of those changes could affect the selected thingshub versions (last 3 major releases). If any changes are required, a new thingshub patch version would be available to accommodate the changes.

• We go through the newly added features that could help to increase the stability and security of the thingshub system. If any changes are required, a new thingshub patch version would be available to accommodate the changes.

Kubernetes Versions / Releases Specific Documentation

Kubernetes 1.35

• No thingsHub-specific changes required for this Kubernetes release.

• Breaking changes — required actions before upgrade:

  • cgroup v1 removed: Kubernetes 1.35 removes cgroup v1 support entirely. Worker nodes still using cgroup v1 will have their kubelet fail to start after the cluster upgrade. You must migrate all worker nodes to cgroup v2 before upgrading your cluster to 1.35. See the Kubernetes 1.31 compatibility note for verification and migration guidance. Do not proceed with the 1.35 upgrade until all nodes confirm cgroup2fs.

  • Private image pull credentials now enforced (KubeletEnsureSecretPulledImages, beta on by default): The kubelet now verifies that a Pod carries valid image pull credentials before allowing it to use a locally cached private image. Any Pod using a private container image without an explicit imagePullSecrets entry may fail to start after a node is reprovisioned. Before upgrading, audit all tenant workloads for pods that reference private registries but do not configure imagePullSecrets, and add the missing credentials.

• Deprecated features:

  • containerd 1.x — final warning: Kubernetes 1.35 is the last release to support containerd 1.x. Support is removed in 1.36. If any cluster nodes still run containerd 1.x after the 1.35 upgrade, those nodes must be refreshed to containerd 2.x before the next minor version upgrade.

• New capabilities available (no action required):

  • In-place Pod resource resize (GA): CPU and memory requests/limits can now be updated on a running Pod without a restart. This is beneficial for stateful workloads such as EMQX brokers and InfluxDB instances. The feature is on by default and opt-in at the workload level via resizePolicy.

Kubernetes 1.34

• No thingsHub-specific changes required for this Kubernetes release.

• Pre-upgrade action (prepare for 1.35):

  • cgroup v2 verification — act now, before 1.35: Kubernetes 1.34 is the last release that supports cgroup v1. Support is removed entirely in 1.35. Use this upgrade window to verify that all worker nodes are already running cgroup v2: run stat -fc '%T' /sys/fs/cgroup/ on each node and confirm the output is cgroup2fs. If any node returns tmpfs (cgroup v1), plan and execute the migration to cgroup v2 before proceeding with a 1.35 upgrade. Refer to your Kubernetes distribution's documentation for node reconfiguration steps.

• Deprecated features:

  • containerd 1.x: containerd 1.x is deprecated as of Kubernetes 1.34. Support continues through 1.35 but will be removed in 1.36. If your cluster nodes use containerd 1.x, schedule a node image upgrade to containerd 2.x within this release cycle.

  • cgroupDriver kubelet configuration field: Manually specifying cgroupDriver in the kubelet configuration file is deprecated in favor of automatic CRI-driven detection. If you explicitly set this field in your kubelet config, plan to remove it in an upcoming release.

Kubernetes 1.33

• No changes required

Kubernetes 1.32

• No changes required

Kubernetes 1.31

• Deprecated features:

  • cgroupv2: Operators need to verify that worker nodes are configured to use cgroupv2. If nodes are using cgroupv1, plan and execute the migration to cgroupv2, consulting official Kubernetes documentation for guidance.

Kubernetes 1.30

• No changes required

Kubernetes 1.29

• Removed feature gates:

  • Based on the infrastructure setup, this change might require additional action on the operations side.

  • The KMSv1 feature gate is disabled by default in 1.29. If your cluster relies on KMSv1 for secrets management, migrate to KMSv2 for continued functionality.

Kubernetes 1.28

• Deprecated features:

  • Based on the infrastructure setup, this change might require additional action on the operations side.

  • KMSv1: This secrets management API is deprecated and will only receive security updates. Migrate to KMSv2 for full functionality.

Kubernetes 1.27

• No changes required

Kubernetes 1.26

• No changes required

Kubernetes 1.25 [ https://kubernetes.io/blog/2022/08/04/upcoming-changes-in-kubernetes-1-25/ ]

• Removals

  • PodSecurityPolicy: Thingshub does not use the PodSecurityPolicy. But since it is a cluster-level resource, it might exist separately from the thingshub installation. Additional details: https://kubernetes.io/docs/concepts/security/pod-security-policy/

• Additional Issues Faced

  • When the cluster was upgraded to 1.25, a new StorageClass was created, and it was marked "default" by the system. This does not affect the running thingshub installation and the upgrading of the existing tenants. But new tenants can't be deployed, as there are multiple "default" StorageClasses. If there is a single "default" StorageClass, the issue will be resolved. Consult the IT team on how to achieve it. Please do not remove the existing default storage class. Just marking it as a non-default storage class will resolve the issue.

Kubernetes 1.24

• No changes required

Kubernetes 1.23

• No changes required

Kubernetes Distributions Compatibility

Per the thingsHub system design definition, thingsHub will only use APIs from pure Kubernetes. So as long as your Kubernetes distribution is a CNCF-certified distribution, compatibility with your Kubernetes distribution should be generally given.

For a list of CNCF-certified Kubernetes distros: https://www.cncf.io/certification/software-conformance/. The most common CNCF-certified Kubernetes distributions are:

• Rancher / RKE Kubernetes Engine

• Google Kubernetes Engine

• Microsoft Azure Kubernetes Service

• Amazon Elastic Kubernetes Service

• Pivotal Container Service (PKS)

Kubernetes is the container orchestration system, and it is the core system that manages this orchestration. RKE, GKE and others are distributions that package Kubernetes along with many other functionalities like platform support, drivers, extensions, packaging, security, and easier and manageable installations.

For this reason, it can never be definitively ruled out that no minor adjustments or configuration changes are necessary for a specific distribution. SmartMakers therefore assumes that the respective operating team of a Kubernetes cluster has sufficient knowledge of its deployed distribution to be able to quickly identify incompatibilities and jointly find solutions for them.

In such cases, please contact your key account representative or email us at support@smartmakers.de.

FAQs

Q. Will Thingshub use PodSecurityAdmission for defining the isolation levels of the pods?

A: The PodSecurityAdmission has a stable feature state from Kubernetes 1.25. We plan to look into this when all of our supported releases are fully compatible with Kubernetes 1.25+