Set up healthchecks with Kuberhealthy

Phase out notice

Nebari is evolving. The original Nebari project is now called Nebari Classic, and is in maintenance mode.

We'll continue to provide security updates and key bug fixes; however, no new features are planned at this time. We're using the lessons learned from Nebari Classic to build a new Nebari ecosystem of tools, on a more robust foundation with a modular architecture. Nebari Classic's maintenance window will remain open until the new architecture reaches stability and feature parity, and current users are able to migrate, expected through the end of 2026.

See the current Nebari documentation for details on the new platform.

Caution

This feature is in beta status. It should be used with caution.

Nebari integrates Kuberhealthy to perform internal healthchecks on Nebari. This is an extensible Kubernetes native framework for continuous synthetic testing. Kuberhealthy is set up to export metrics to Prometheus. This allows them to be seen in Grafana.

Enabling

Healthchecks are currently considered a beta feature that we are testing. Due to this, they are disabled by default. To enable healthchecks, add the following configuration under the monitoring configuration in your nebari-config.yaml.

monitoring:
  healthchecks:
    enabled: true

Checking status of Healthchecks

All healthchecks are exported as metrics to Prometheus and can be viewed in Grafana.

For example: To see the uptime for the conda-store service, you can run:

1 - (sum(count_over_time(kuberhealthy_check{check="dev/conda-store-http-check", status="0"}[30d])) OR vector(0))/(sum(count_over_time(kuberhealthy_check{check="dev/conda-store-http-check", status="1"}[30d])) * 100)

in Grafana, which will show you the following chart.

To see what other healthchecks are available, you can use the metric explorer in Grafana. Select the metric type of kuberhealthy_check and the label filter of check. The values list will be a list of the checks that have metrics available.

Note

If you have previously deployed Nebari without healthchecks, You may need to restart your Prometheus service to get it to pick up the kuberhealthy metrics.

Summary of available healthchecks

Below is an explanation of the available healthchecks. This list may not be comprehensive as work on this feature is ongoing.

Check Label	Description
conda-store-http-check	verifies that conda-store is accessible via it's REST API
jupyterhub-http-check	verifies JupyterHub is running
dns-status-internal	verifies internal DNS is accessible
daemonset	verifies that a daemonset can be created, fully provisioned, and torn down. This checks the full kubelet functionality of every node in your Kubernetes cluster
deployment	verifies that a fresh deployment can run, deploy multiple pods, pass traffic, do a rolling update (without dropping connections), and clean up successfully
keycloak-http-check	verifies Keycloak is accessible