Skip to main content

Upgrade cluster's Kubernetes version

Overview​

Nebari runs on Kubernetes under the hood, and as administrators of this Kubernetes cluster, one of the maintenance tasks required from time to time is upgrading the version of your Kubernetes cluster. Each of the different cloud providers that Nebari can run on, has their own release cycle and support windows for their flavor of Kubernetes. That said, they tend to follow the official Kubernetes release cycle.

The Nebari development team tries to stay ahead of this by supporting the latest version when possible. However, given that many Kubernetes releases come with a set of deprecations that potentially affect Nebari and downstream plugins, there is an enforced HIGHEST_SUPPORTED_K8S_VERSION allowed.

ℹī¸note

This HIGHEST_SUPPORTED_K8S_VERSION is, at times, a minor version or two behind the officially released Kubernetes version.

Many cloud providers enable users to automatically upgrade their Kubernetes cluster (control plane), however given the potential for deprecations and other changes, Nebari deployed to these clouds has this feature automatically disabled.

This upgrade process bumps the version of the control plane along with all node groups.

đŸ”Ĩwarning
  1. Upgrading the kubernetes version of the node groups will cause downtime so please plan accordingly. We also recommend backing up your data before starting this upgrade process.
  2. Kubernetes versions can only be upgraded one minor version at a time. If you're running on 1.24, and need to upgrade to 1.26, you will first need to upgrade to 1.25.
  3. Downgrading to a lower version of Kubernetes is dangerous and we strongly advise against it!

Google Kubernetes Engine (GKE) cut their own platform specific version of Kubernetes that usually look something like: 1.26.7-gke.500; this corresponds to a Kubernetes version of 1.26.7.

You can list the supported GKE Kubernetes versions by running the following gcloud command:

gcloud container get-server-config --region <region>

We recommend selecting a validVersion from the STABLE channel:

channels:
- channel: RAPID
  defaultVersion: 1.27.4-gke.900
  validVersions:
  - 1.28.1-gke.201
  ...
- channel: REGULAR
  defaultVersion: 1.27.3-gke.100
  validVersions:
  - 1.27.4-gke.900
  - 1.27.3-gke.1700
  ...
- channel: STABLE
  defaultVersion: 1.27.3-gke.100
  validVersions:
  - 1.27.4-gke.900
  - 1.27.3-gke.100
  - 1.26.7-gke.500
  ...
  - 1.24.15-gke.1700
  - 1.24.14-gke.2700

To upgrade your GKE cluster, update the google_cloud_platform.kubernetes_version field in your nebari-config.yaml to match one of these GKE Kubernetes versions. Then run nebari deploy to apply these changes. This deployment process might take as long as 30 minutes.

❗ī¸info

You will get a validation error if you try to select a Kubernetes version that is unsupported by GKE or a version higher than HIGHEST_SUPPORTED_K8S_VERSION.

Then repeat the above process one minor version at a time. You will get a similar error otherwise:

[terraform]: │ Error: googleapi: Error 400: Master cannot be upgraded to "1.26.7-gke.500": cannot upgrade the master more than a minor version at a time.

For more information about GKE upgrades, please refer to the GKE documentation.