Skip to content

Debug a deployment

Updated 3 min read

This guide covers debugging a stuck or failing Nebari Kubernetes Platform (NKP) deployment. It is scoped to NKP only. For Classic Nebari, see Troubleshooting.

nic deploy phase

Section titled “nic deploy phase”

Reading nic output

Section titled “Reading nic output”

nic deploy streams structured progress to stdout. When a deploy fails, the error appears at the end of that output. Capture it for sharing or offline review:

Terminal window
nic deploy -f config.yaml 2>&1 | tee deploy.log

For deeper traces that show exactly which internal step hung, enable the built-in OpenTelemetry console exporter (OTEL_EXPORTER is a nic-specific variable; valid values are none, console, otlp, and both):

Terminal window
OTEL_EXPORTER=console nic deploy -f config.yaml

This emits span data for every internal step, including provider API calls and ArgoCD bootstrap.

Common nic deploy failures

Section titled “Common nic deploy failures”
Symptom Cause Fix
Exits immediately with a config or credential error Pre-flight validation failed Run nic validate -f config.yaml and fix the reported fields before re-deploying
context deadline exceeded Large cluster or slow network Re-run with --timeout 1h
Cloud API error (quota exceeded, insufficient permissions) Provider-specific Check your cloud provider's console; fix IAM permissions or request a quota increase, then re-run (nic deploy is idempotent)
Hangs after "ArgoCD installed" ArgoCD can't reach the GitOps repository Verify GIT_TOKEN has read/write access to the repo, then check ArgoCD server logs (see ArgoCD applications below)

Post-deploy phase

Section titled “Post-deploy phase”

After nic deploy returns, the cluster still needs ArgoCD to sync and start all foundational services. This section covers debugging that process in dependency order.

ArgoCD applications

Section titled “ArgoCD applications”

Start here: ArgoCD is the engine that delivers everything else.

List all applications and their sync and health status:

Terminal window
kubectl get applications -n argocd

Inspect a stuck or degraded application:

Terminal window
kubectl describe application <name> -n argocd

What to look for:

  • SyncStatus: Synced (good), OutOfSync (hasn't applied the latest manifests), Unknown (can't reach the source repo; check the ArgoCD server logs).
  • Health: Healthy (good), Progressing (still starting up), Degraded (something is wrong).
  • A Degraded application includes a human-readable message in Status.Conditions. Read it before looking elsewhere.

Check ArgoCD server logs directly:

Terminal window
kubectl logs -n argocd -l app.kubernetes.io/component=server --tail=100

Common causes of stuck ArgoCD applications: GIT_TOKEN expired or missing repo access, or the cluster can't pull images from the OCI registry.

Foundational software pods

Section titled “Foundational software pods”

Each foundational component runs in its own namespace. Check for pods that are not Running:

Component Namespace
cert-manager cert-manager
Envoy Gateway envoy-gateway-system
Keycloak keycloak
OpenTelemetry Collector monitoring

General pattern for any component:

Terminal window
kubectl get pods -n <namespace>
kubectl logs -n <namespace> <pod-name>

cert-manager

Section titled “cert-manager”

A healthy ArgoCD application for cert-manager does not guarantee that cert-manager is issuing TLS certificates. Also check:

Terminal window
kubectl get certificates -A
kubectl describe certificate <name> -n <namespace>

A False ready status usually means a DNS-01 challenge timeout or a Let's Encrypt rate limit. The describe output includes the challenge URL and the specific error message from the ACME server.

Keycloak

Section titled “Keycloak”

If Keycloak pods are running but the sign-in page is unreachable, check whether the HTTPRoute was created:

Terminal window
kubectl get httproutes -A

Nebari Operator

Section titled “Nebari Operator”

Check the operator pod:

Terminal window
kubectl get pods -n nebari-operator-system
kubectl logs -n nebari-operator-system -l control-plane=controller-manager --tail=100

The most common failure is the operator starting before Keycloak is healthy and failing to connect. Fix: wait for the keycloak ArgoCD application to reach Healthy, then delete the operator pod to trigger a restart:

Terminal window
kubectl delete pod -n nebari-operator-system -l control-plane=controller-manager

NebariApp status

Section titled “NebariApp status”

List all NebariApps across namespaces:

Terminal window
kubectl get nebariapps -A

Inspect the status conditions on a specific NebariApp:

Terminal window
kubectl describe nebariapp <name> -n <namespace>

The four status conditions and what each one tracks:

Condition What it tracks
RoutingReady HTTPRoute created and accepted by Envoy Gateway
TLSReady TLS certificate issued by cert-manager
AuthReady OIDC client provisioned in Keycloak
Ready All three above are True

Each condition includes a Message field explaining why it is False. Read that before checking logs.

Symptom index

Section titled “Symptom index”
Symptom Where to look
nic deploy exits with an error immediately nic deploy common failures
nic deploy times out nic deploy common failures, timeout row
nic deploy hangs after "ArgoCD installed" nic deploy common failures; ArgoCD applications
ArgoCD app stuck in Progressing or Degraded ArgoCD applications
Pod in CrashLoopBackOff or ImagePullBackOff Foundational software pods
Nebari Operator pod crashing or not connecting to Keycloak Nebari Operator
TLS certificate not issuing cert-manager
Can't sign in / Keycloak error Keycloak
NebariApp not Ready NebariApp status
Section titled “Related pages”