Skip to main content

Debug Nebari deployment with k9s

The beauty of Nebari is that most Dev Ops management is already handled for you. However, there will likely come a time that you will need to debug your Nebari instance. This debug guide contains some tips and tricks to help you in your debug endeavors.

Debugging the Kubernetes Cluster using k9s​

If you need more information about your Kubernetes cluster, k9s is a terminal-based UI that is extremely useful for debugging. It simplifies navigating, observing, and managing your applications in Kubernetes. k9s continuously monitors Kubernetes clusters for changes and provides shortcut commands to interact with the observed resources. It's a fast way to review and resolve day-to-day issues in Kubernetes, a huge improvement to the general workflow, and a best-to-have tool for debugging your Kubernetes cluster sessions.

Setting up k9s for Nebari​

First, follow the installation instructions on the k9s documentation to install k9s on your local machine.

â„đïļnote

If you deployed manually from your local machine, the configuration file kubeconfig will have been autogenerated during the nebari deploy process. You can skip to the Getting started with k9s below.

Generating the kubeconfig​

For Google GCP (also known as Google Cloud Platform or GCP), you'll need to have gcloud installed.

Next, install the gke-gcloud-auth-plugin component (this step will become outdated once GKE (Google Kubernetes Engine) version 1.26 is released).

gcloud components install gke-gcloud-auth-plugin

To enable this plugin, you'll need to update gcloud by:

  1. Set export USE_GKE_GCLOUD_AUTH_PLUGIN=True in ~/.bashrc (or in Environment variables for Windows).

  2. Run the following command:

    source ~/.bashrc

    For Windows environments, start a new terminal.

  3. Update gcloud to the latest version.

    gcloud components update

Now you'll need to authenticate to GCP using:

gcloud auth login

Finally, you can generate the kubeconfig which will be used by k9s:

gcloud container clusters get-credentials <project-name>-<namespace> --region <region>

Getting started with k9s​

Launch k9s from a terminal on your local machine. It will look for the configuration in the kubeconfig to dial the connection with the cluster.

$ k9s

To view all the current processes

Use the shortcut 0 (this may be the default).

Image of the k9s terminal UI

Using k9s you can inspect any services launched by your cluster. For example, if your cluster has problems with the network traffic tunnel configuration, it may limit or block a user's access to destination resources such as files or workers, using k9s you can inspect the network configuration as well as the connection logs.

To debug processes running on pod

Use the arrow keys to navigate to a pod for your Nebari instance. Then, you can investigate the pod by clicking Enter or by typing the hotkey l.

Advanced k9s usage​

From within k9s, the port-forward keystroke, shift + f, allows you to access and interact with your Kubernetes cluster from your local machine. You can then use this method to investigate issues and adjust your services locally without the need to expose them beforehand.

Let's suppose you are deploying a new service to Nebari, and would like to quickly debug any connection related problems around it, for example a 404 Error. With port-forwarding you can attest if the origin of the problem lies within the service itself or with the networking aspects of the connection, for example an Ingress route misconfiguration. This allows you to check the service status without any extra modifications to its structure.

Troubleshooting​

If your k9s instance fails to connect to your cluster, ensure that you have the authentication set up properly. k9s may give you a warning similar to: "Dial K9s Toast [X/5]".

If your k9s instance closes for unknown reasons, you can check the k9s logs. You can specify where your log file is located by:

k9s --logFile string

or you can inspect the default temporary directory by looking up the logFile attribute description in k9s --help.