Troubleshooting Devtron

Problem

This occurs most of the time because one or more Devtron installation jobs fail.

Root Cause

Incomplete installation jobs or failed migrator execution.

Resolution

1. Run: kubectl get jobs -n devtroncd and check for jobs with 0/1 completions.

2. Check pods of incomplete jobs: kubectl get pods -n devtroncd.

3. If pods are still running, wait; if not, delete the incomplete jobs: kubectl delete jobs <job1> <job2> -n devtroncd.

4. Download migrator: wget https://raw.githubusercontent.com/devtron-labs/devtron/main/manifests/yamls/migrator.yaml.

5. Edit migrator.yaml and remove creation of postgresql-migrator secret.

6. Apply YAML: kubectl apply -f migrator.yaml -n devtroncd.

7. Wait for recreated jobs to complete and retry saving configuration.

Problem

Deployment metrics or application metrics fail to appear.

Root Cause

Rollout CRDs are outdated on the cluster.

Resolution

1. Apply updated Rollout CRDs:

2. kubectl apply -f https://raw.githubusercontent.com/devtron-labs/devtron/main/manifests/yamls/rollout.yaml -n devtroncd

Problem

Login fails with errors related to token absence or incorrect OIDC config.

Root Cause

Stale or invalid tokens, Dex failing to refresh configuration.

Resolution

1. Delete Devtron pod: kubectl delete pod -n devtroncd -l app=devtron.

2. Wait for pod restart and retry login.

Problem

No logs are visible and builds cannot be aborted.

Root Cause

kubewatch or NATS-based log forwarding components malfunction.

Resolution

1. Check CI pods: kubectl get pods -n devtron-ci.

2. If pods are created, delete kubewatch and devtron pods:

3. kubectl delete pod -n devtroncd -l app=devtron

4. kubectl delete pod -n devtroncd -l app=kubewatch

5. If still unresolved, delete:

6. kubectl delete pod -n devtroncd devtron-nats-0

7. kubectl delete pod -n devtroncd devtron-stan-0

8. kubectl delete pod -n devtroncd -l app=devtron

9. kubectl delete pod -n devtroncd -l app=kubewatch

10. Trigger a new build.

Problem

Dashboards show panel errors or 404.

Root Cause

Grafana org settings or Prometheus datasource mismatch.

Resolution

1. Check Prometheus configuration.

2. Open <devtron-url>/grafana?orgId=2.

3. If 'Not Found', follow Grafana org recreation steps.

4. Run setup commands inside curl utility pod to reconfigure datasources.

5. Revisit Grafana and ensure datasource and dashboards exist.

6. Update environment from Devtron UI to sync datasources.

Screenshots

Problem

Login fails despite correct password.

Root Cause

Stale argocd token stored in browser cookies.

Resolution

1. Open browser developer tools.

2. Delete ArgoCD cookie as per your browser:

3. Firefox → Storage → Cookies → Delete All Session Cookies.

4. Chrome → Application → Cookies → Delete token.

5. Safari → Enable Develop menu → Storage → Cookies → Delete token.

6. Retry login.

Screenshots

Problem

Discover section shows empty chart list.

Root Cause

Chart repositories not refreshed.

Resolution

1. Go to Application Management → Configurations → Chart Repository.

2. Click 'Refresh Chart'.

3. Wait 4–5 minutes for charts to sync.

Screenshots

Problem

Updating cluster throws datasource reload errors.

Root Cause

Devtron pod caching issues.

Resolution

1. Edit cluster settings and click Save.

2. If error appears, delete Devtron pod:

3. kubectl -ndevtroncd delete po -l app=devtron

4. Refresh Devtron UI and try again.

Problem

Postgres goes into CrashLoopBackOff.

Root Cause

Incorrect or outdated PostgreSQL image.

Resolution

1. Patch StatefulSet with new images:

2. kubectl patch ...

3. Delete Postgres pod so it restarts with new image.

Problem

Devtron is not showing new commits.

Root Cause

Git sensor pod malfunction.

Resolution

1. Save Git repository again.

2. Check logs: kubectl logs -n devtroncd -l app=git-sensor

3. Bounce git-sensor pod:

4. kubectl delete pod -n devtroncd git-sensor-0

Problem

SSO fails in clusters with IP-restricted service.

Root Cause

NAT gateway IP ranges not whitelisted.

Resolution

1. Whitelist NAT gateway IPs of the cluster.

Problem

CPU graphs remain empty.

Root Cause

image!="" filter blocks CPU metric results.

Resolution

1. Login to Grafana.

2. Edit CPU graphs.

3. Remove filter image!="".

4. Save dashboard.

Problem

Uploads fail for larger files.

Root Cause

Ingress proxy-body-size limit.

Resolution

1. Add annotation:

2. nginx.ingress.kubernetes.io/proxy-body-size: 100m

Problem

ALB provisioning fails with unauthorized error.

Root Cause

Nodegroup IAM role missing permissions.

Resolution

1. Attach required IAM policy:

2. https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.3.1/docs/install/iam_policy.json

Problem

Metrics fail to appear for workloads.

Root Cause

Kube-prometheus stack value mismatch.

Resolution

1. Set serviceMonitorSelectorNilUsesHelmValues: false in Prometheus chart values.

Problem

Metrics-server installation fails.

Root Cause

Certificate validation requiring secure kubelet endpoint.

Resolution

1. Disable certificate validation:

2. Add --kubelet-insecure-tls to metrics-server chart values.

Problem

Database shows 'being accessed by other users' error.

Root Cause

Open backend connections.

Resolution

1. Terminate sessions:

2. SELECT pg_terminate_backend(pg_stat_activity.pid)...

3. Delete DB using: drop database <db>

Problem

Login fails or new admin password is needed.

Root Cause

ADMIN_PASSWORD fields in secret outdated.

Resolution

1. Backup devtron-secret.

2. Edit secret and remove ADMIN_PASSWORD keys.

3. Restart Dex server.

4. Retrieve new admin password.

Problem

'base64 not recognized' error appears.

Root Cause

Windows missing base64 binary.

Resolution

1. Install base64 utility OR

2. Get encoded password:

3. kubectl -n devtroncd get secret devtron-secret -o jsonpath='{.data.ACD_PASSWORD}'

4. Use online base64 decoder.

Problem

StatefulSet cannot be patched due to immutable fields.

Root Cause

Annotations or volume size mismatch.

Resolution

1. Ensure all Devtron resources are annotated/labeled.

2. Add parameter:

3. --set components.postgres.persistence.volumeSize=20Gi

4. Retry upgrade.

Problem

Blob storage not configured or misconfigured.

Root Cause

Missing or incorrect provider-specific values.

Resolution

1. Follow provider-specific configuration instructions (MinIO, AWS, Azure, GCP).

Problem

Rollout controller throws lua runtime errors.

Root Cause

Old CRDs incompatible with K8s >=1.22.

Resolution

1. Check rollout chart version.

2. Update to devtron/rollout latest chart.

3. Ensure Devtron helm repo is added.

Problem

Pod status shows ImagePullBackOff.

Root Cause

Invalid or missing registry credentials.

Resolution

1. Use 'How to resolve?' from Devtron UI.

2. Go to Manage Access → select cluster.

3. Redeploy application.

Screenshots

Problem

Terminal disconnects frequently.

Root Cause

timeoutSec in BackendConfig defaults to 30 seconds.

Resolution

1. Create BackendConfig YAML with higher timeout.

2. Apply BackendConfig.

3. Annotate Devtron service with backend-config name.

Problem

ArgoCD certs expire, causing login or GitOps failures.

Root Cause

Old tls.crt/tls.key in argocd-secret.

Resolution

1. Edit argocd-secret and remove tls data.

2. Delete ArgoCD server pod.

3. Delete Devtron pod.

Problem

Exit status 128 shown in git-sensor logs.

Root Cause

Token invalid or Git sensor pod locked.

Resolution

1. Save Git repository again.

2. Check git-sensor logs.

3. Bounce git-sensor pod.

Problem

Git-sensor shows disk full error.

Root Cause

PVC too small or volumeExpansion disabled.

Resolution

1. Check storageclass attributes.

2. Ensure allowVolumeExpansion = true.

3. Edit PVC to increase size.

4. Bounce git-sensor pod.

Screenshots

Problem

Rollout sync fails with 'Invalid JSON Document'.

Root Cause

Manually edited or corrupted JSON annotation.

Resolution

1. Copy annotation value.

2. Validate via JSON validator.

3. Fix malformed JSON.

4. Apply corrected configuration.

Screenshots

Problem

'tls: handshake failure' when syncing Bitnami charts.

Root Cause

Insecure flag erroneously enabled.

Resolution

1. Devtron version must be >=0.7.1.

2. Go to Chart Repository → Bitnami.

3. Disable 'Allow Insecure Connection'.

4. Sync charts again.

Screenshots

Problem

Deployment template fields show empty values.

Root Cause

Chart missing required app-values.yaml.

Resolution

1. Add app-values.yaml to your deployment chart.

2. Upload chart again.

Screenshots

Problem

GitOps pipeline creation fails.

Root Cause

GitOps and Cluster configuration not re-saved after enabling module.

Resolution

1. Save GitOps configuration again.

2. Save Cluster configuration again.

3. Retry creating pipeline.