Application Metrics

While deploying kube-prometheus-stack chart, the deployment status may show as Timed out, and some CustomResourceDefinitions (CRDs) may appear as Failed.

This behavior is expected and do not require any action from you.

This occurs because certain Prometheus CRDs are large in size, which can lead to temporary sync issues during deployment, but, this does not impact the functionality of the Prometheus components.

ArgoCD handles such cases automatically and the kube-prometheus-stack will continue to function as expected.

Update the rollout CRDs to the latest version, run the following command:

kubectl apply -f https://raw.githubusercontent.com/devtron-labs/devtron/main/manifests/yamls/rollout.yaml -n devtroncd

If the graphs are not visible check if Prometheus is configured properly. Then go to Global Configurations > Clusters & Environments > Click on any environment for the cluster where you added Prometheus endpoint and simply click Update. If the charts are still not visible, try visiting the URL: <devtron-url>/grafana?orgId=2 If you see Not Found on this page, then follow all the given steps or if the page is accessible, and you are getting panel with id 2 not found then follow from step 6:

1. Get Grafana password using kubectl -n devtroncd get secret devtron-secret -o jsonpath='{.data.GRAFANA_PASSWORD}' | base64 -d

2. kubectl run --rm -it --image quay.io/devtron/k8s-utils:tutum-curl curl Run this command, and it will create a pod for using curl

3. Copy the following and change grafana-password with your password of Grafana and change the value of prometheusUrl with your Prometheus endpoint, and run in the pod that we created above in step 2.

   Copy

   cat << EOF
   grafanaUrl="http://admin:[email protected]/grafana"
   prometheusUrl="http://prometheus.example.com"
   
   ORG_ID=$( curl -d '{"name":"devtron-metrics-view"}' -H "Content-Type: application/json" -X POST "${grafanaUrl}/api/orgs" )
   
   echo $ORG_ID
   
   curl -X POST "${grafanaUrl}/api/user/using/2";
   
   curl -X PUT -H "Content-Type: application/json" -d '{"homeDashboardId":0,"theme":"light","timezone":"browser"}' "${grafanaUrl}/api/org/preferences";
   
   curl "${grafanaUrl}/api/datasources" -H 'content-type: application/json' -H 'x-grafana-org-id: 2' --data '{"name":"Prometheus-devtron-demo","type":"prometheus","access":"proxy","isDefault":true}'
   
   curl "${grafanaUrl}/api/datasources/2" -X PUT \
       -H 'content-type: application/json' \
       -H 'x-grafana-org-id: 2' \
       --data '{"id": 2 ,
       "orgId": 2,
       "name":"Prometheus-devtron-demo","type":"prometheus","access":"proxy",
       "url":${prometheusUrl},
       "basicAuth":false,"jsonData":{},"version":1}'
   EOF

4. Now visit <devtron-url>/grafana?orgId=2 again, and you'll see Grafana login page. Login using username admin and password from step 1 and check if Prometheus URL is updated in data sources. If not, update it in the default data source.

5. Now from Devtron UI, update any of the environment again and its data source will be created automatically.

6. In Grafana UI you need to be logged in and Go to Dashboards > Manage then click Import and Import the given dashboards one by one.

   Copy

   https://grafana.com/api/dashboards/13322/revisions/4/download
   https://grafana.com/api/dashboards/13320/revisions/4/download
   https://grafana.com/api/dashboards/13325/revisions/4/download
   https://grafana.com/api/dashboards/13321/revisions/6/download

After that, your issue should be resolved, and you should be able to see all the graphs on UI.

Do the following:-

1. Go to Grafana and Login with the credentials.

2. Edit the CPU graphs and remove image!=”” from the query.

3. Save the dashboard.

CPU metrics should start showing up in a while.