1 of 79

v0.5

Overview

Devtron 🚀

Devtron is a tool integration platform for Kubernetes.

Devtron deeply integrates with products across the lifecycle of microservices,i.e., CI, CD, security, cost, debugging, and observability via an intuitive web interface.

To quickly get started, refer to the Devtron Installation Guide ⎈ ⎈

Why Devtron?

To improve the use of Kubernetes, we employ several tools. Using these tools at the same time, however, is cumbersome and complex. This is because these tools do not communicate with one another to manage different aspects of the application lifecycle, such as CI, CD, security, cost, observability, and stabilization.

Devtron is a one-stop solution for the complexity of the tools mentioned above!

Devtron is an open-source modular product that provides a 'seamless' and 'implementation agnostic uniform interface', that can be integrated with both open-source and commercial tools across the entire lifecycle. All this is achieved while focusing on a slick user experience, including a self-serve model.

You can efficiently handle security, stability, cost, and more in a unified experience.

Devtron Features:

Zero code software delivery workflow for Kubernetes

Workflow which understands the domain of Kubernetes, testing, CD, SecOps so that you don't have to write scripts
Reusable and composable components so that workflows are easy to construct and reason through

Multi cloud deployment

Deploy to multiple Kubernetes clusters on multiple cloud/on-prem from one Devtron setup.
Works for all cloud providers and on-premise Kubernetes clusters.

Easy DevSecOps integration

Multi-level security policy at global, cluster, environment, and application-level for efficient hierarchical policy management
Behavior-driven security policy
Define policies and exceptions for Kubernetes resources
Define policies for events for faster resolution

Application debugging dashboard

One place for all historical Kubernetes events
Access all manifests securely, such as secret obfuscation
Application metrics for CPU, RAM, HTTP status code, and latency with a comparison between new and old
Advanced logging with grep and JSON search
Intelligent correlation between events, logs for faster triangulation of issue
Auto issue identification

Enterprise-Grade security and compliances

Fine-grained access control; control who can edit the configuration and who can deploy.
Audit log to know who did what and when
History of all CI and CD events
Kubernetes events impacting application
Relevant cloud events and their impact on applications
Advanced workflow policies like blackout window, branch environment relationship to secure build and deployment pipelines

GitOps aware

GitOps exposed through API and UI so that you don't have to interact with git CLI
GitOps backed by Postgres for easy analysis
Enforce finer access control than Git

Operational insights

Deployment metrics to measure the success of the agile process. It captures MTTR, change failure rate, deployment frequency, and deployment size out of the box.
Audit log to understand the failure causes
Monitor changes across deployments and reverts easily

Compatibility notes

It uses a modified version of argo rollout.
Application metrics only work for k8s 1.16+

What's next

Install Devtron

Contribute

Check out our contributing guidelines. Directions for opening issues, coding standards, and notes on our development processes are all included.

Community

Get updates on Devtron's development and chat with the project maintainers, contributors, and community members.

Join the Discord Community
Follow @DevtronL on Twitter
Raise feature requests, suggest enhancements, report bugs at GitHub issues
Read the Devtron blog

Vulnerability Reporting

We at Devtron take security and our users' trust very seriously. If you believe you have found a security issue in Devtron, please responsibly disclose it by contacting us at security@devtron.ai.

License

Devtron is available under the Apache License, Version 2.0.

Getting Started

Install Devtron

Devtron is installed over a Kubernetes cluster and can be installed standalone or along with CI/CD integration:

Devtron with CI/CD: Devtron installation with the CI/CD integration is used to perform CI/CD, security scanning, GitOps, debugging, and observability.
Devtron: The Devtron installation includes functionalities to deploy, observe, manage, and debug existing Helm applications in multiple clusters and deeply integrate with multiple tools using extensions.

Recommended resources

The minimum requirements for Devtron and Devtron with CI/CD integration in production and non-production environments include:

Non-production

Integration

CPU

Memory

Production (assumption based on 5 clusters)

Integration

CPU

Memory

Refer to the Override Configurations section for more information.

Note: It is NOT recommended to use brustable CPU VMs (T series in AWS, B Series in Azure and E2/N1 in GCP) for Devtron installation.

Before you begin

Create a Kubernetes cluster (preferably K8s 1.16 or higher) if you haven't done that already!

Refer to the Creating a Production grade EKS cluster using EKSCTL article to set up a cluster in the production environment.

Install Devtron with CI/CD integration

Are you installing Devtron on Minikube, Microk8s, K3s, Kind? See Instructions

Before you begin

Install .

Installing Devtron using Helm

Add Devtron repository
Install Devtron

This installation will use Minio for storing build logs and cache.

This installation will use AWS s3 buckets for storing build logs and cache. Refer to the AWS specific parameters on the page.

This installation will use Azure Blob Storage for storing build logs and cache. Refer to the Azure specific parameters on the page.

Append the command with --set installer.release="vX.X.X" to install a particular version of Devtron. Where vx.x.x is the .

For those countries/users where Github is blocked, you can use Gitee as the installation source:

Check Devtron installation status

The install commands start Devtron-operator, which takes about 20 minutes to spin up all of the Devtron microservices one by one. You can use the following command to check the status of the installation:

The command executes with one of the following output messages, indicating the status of the installation:

Check the installer logs

To check the installer logs, run the following command:

Devtron dashboard

Use the following command to get the dashboard URL:

You will get an output similar to the one shown below:

The hostname aaff16e9760594a92afa0140dbfd99f7-305259315.us-east-1.elb.amazonaws.com as mentioned above is the Loadbalancer URL where you can access the Devtron dashboard.

If you don't see any results or receive a message that says "service doesn't exist," it means Devtron is still installing; please check back in 5 minutes.

Note: You can also do a CNAME entry corresponding to your domain/subdomain to point to this Loadbalancer URL to access it at a custom domain.

Devtron Admin credentials

For admin login, use the username:admin, and run the following command to get the admin password:

Cleaning Devtron Helm installer

Please make sure that you do not have anything inside namespaces devtroncd, devtron-cd, devtron-ci, and devtron-demo as the below steps will clean everything inside these namespaces:

What's next

FAQs

1. How will I know when the installation is finished?

Run the following command to check the status of the installation:

The above command will print Applied once the installation process is complete. The installation process could take up to 30 minutes.

2. How do I track the progress of the installation?

Run the following command to check the logs of the Pod:

3. How can I restart the installation if the Devtron installer logs contain an error?

First run the below command to clean up components installed by Devtron installer:

Install Devtron

Are you installing Devtron on Minikube, Microk8s, K3s, Kind? See Instructions

This page helps you to install Devtron without any integrations. Integrations can be added later using .

Before you begin

Install if you haven't done that already!

Devtron dashboard

Use the following command to get the dashboard URL:

You will get the result something as shown below:

The hostname aaff16e9760594a92afa0140dbfd99f7-305259315.us-east-1.elb.amazonaws.com as mentioned above is the Loadbalancer URL where you can access the Devtron dashboard.

You can also do a CNAME entry corresponding to your domain/subdomain to point to this Loadbalancer URL to access it at a custom domain.

Host

Type

Points to

Devtron Admin credentials

For admin login, use the username as admin, and run the following command to get the admin password:

Cleaning Helm installer

Please make sure that you do not have anything inside namespaces devtroncd, devtron-cd devtron-ci, and devtron-demo as the below steps will clean everything inside these namespaces

Upgrade

Install Devtron on Minikube, Microk8s, K3s, Kind

You can install and try Devtron on a high-end machine or on a Cloud VM. If you install it on a laptop/PC, it may start to respond slow, so it is recommended to uninstall Devtron from your system before shutting it down.

System Configurations for Devtron Installation

2 vCPUs
4GB+ of free memory
20GB+ free disk space

Before you begin

Before we get started and install Devtron, we need to set up the cluster in our servers & install required tools

Create cluster using Minikube or Kind or K3s
Install Helm3
Install kubectl

Install Devtron on your machine

Add Devtron repository
Install Devtron
Port-forward the devtron-service to access dashboard

To install devtron on Minikube/kind Cluster use the Following commands

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set components.devtron.service.type=NodePort

To install devtron on k3s Cluster use the Following commands

kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set components.devtron.service.type=NodePort

Devtron dashboard

To access dashboard when using Minikube as Cluster use this command, dashboard will automatically open on default browser.

minikube service devtron-service --namespace devtroncd

To access dashboard when using Kind/k3s as Cluster, use this command to port forward the devtron service to port 8000

kubectl -ndevtroncd port-forward service/devtron-service 8000:80

Dashboard http://127.0.0.1:8000.

Devtron Admin credentials

For admin login, use the username:admin, and run the following command to get the admin password:

kubectl -n devtroncd get secret devtron-secret -o jsonpath='{.data.ACD_PASSWORD}' | base64 -d

Install Devtron on Cloud VM (AWS ec2, Azure VM, GCP VM)

It is preferd to use Cloud VM with 2vCPU+, 4GB+ free Memory, 20GB+ Storage, Compute Optimized VM type & Ubuntu Flavoured OS.

Create Microk8s Cluster

sudo snap install microk8s --classic --channel=1.22
sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube
newgrp microk8s
microk8s enable dns storage helm3
echo "alias kubectl='microk8s kubectl '" >> .bashrc
echo "alias helm='microk8s helm3 '" >> .bashrc
source .bashrc

Install devtron

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set components.devtron.service.type=NodePort

Ensure that the port on which the devtron-service runs is open in the VM's security group or network Security group.

Commad to get the devtron-service Port number

kubectl get svc -n devtroncd devtron-service -o jsonpath='{.spec.ports[0].nodePort}'

Installation Configurations

Configuration

Configure Secrets

For helm installation this section referes to secrets section of values.yaml. For kubectl based installation it refers to kind: secret in install/devtron-operator-configs.yaml.

Configure the following properties:

Parameter

Description

Default

Configure ConfigMaps

For helm installation this section refers to configs section of values.yaml. For kubectl based installation it refers to kind: ConfigMap in install/devtron-operator-configs.yaml.

Configure the following properties:

Parameter

Description

Default

Configure Overrides

For helm installation this section refers to customOverrides section of values.yaml. In this section you can override values of devtron-cm which you want to keep persistent. For example:

You can configure the following properties:

Storage for Logs and Cache

AWS SPECIFIC

While installing Devtron and using the AWS-S3 bucket for storing the logs and caches, the below parameters are to be used in the ConfigMap.

NOTE: For using the S3 bucket it is important to add the S3 permission policy to the IAM role attached to the nodes of the cluster.

AZURE SPECIFIC

While installing Devtron using Azure Blob Storage for storing logs and caches, the below parameters will be used in the ConfigMap.

To convert string to base64 use the following command:

echo -n "string" | base64 -d

Note:
Ensure that the cluster has read and write access to the S3 buckets/Azure Blob storage container mentioned in DEFAULT_CACHE_BUCKET, DEFAULT_BUILD_LOGS_BUCKET or AZURE_BLOB_CONTAINER_CI_LOG, or AZURE_BLOB_CONTAINER_CI_CACHE.
Ensure that the cluster has read access to AWS secrets backends (SSM & secrets manager).

The following tables contain parameters and their details for Secrets and ConfigMaps that are configured during the installation of Devtron. While installing Devtron using kubectl the following parameters can be tweaked in devtron-operator-configs.yaml file. If the installation is proceeded using helm3, the values can be tweaked in values.yaml file.

We can use the --set flag to override the default values when installing with Helm. For example, to update POSTGRESQL_PASSWORD and BLOB_STORAGE_PROVIDER, use the install command as:

helm install devtron devtron/devtron-operator --create-namespace --namespace devtroncd \
--set secrets.POSTGRESQL_PASSWORD=change-me \
--set configs.BLOB_STORAGE_PROVIDER=S3 \

Secrets

ConfigMaps

Dashboard Configurations

RECOMMEND_SECURITY_SCANNING=false
FORCE_SECURITY_SCANNING=false
HIDE_DISCORD=false

Override Configurations

In certain cases, you may want to override default configurations provided by Devtron. For example, for deployments or statefulsets you may want to change the memory or CPU requests or limit or add node affinity or taint tolerance. Say, for ingress, you may want to add annotations or host. Samples are available inside the directory.

To modify a particular object, it looks in namespace devtroncd for the corresponding configmap as mentioned in the mapping below:

component

configmap name

purpose

apiVersion, kind, metadata.name in the multiline string is used to match the object which needs to be modified. In this particular case it will look for apiVersion: extensions/v1beta1, kind: Ingress and metadata.name: devtron-ingress and will apply changes mentioned inside update: as per the example inside the metadata: it will add annotations owner: app1 and inside spec.rules.http.host it will add http://change-me.

Once we have made these changes in our local system we need to apply them to a Kubernetes cluster on which Devtron is installed currently using the below command:

Run the following command to make these changes take effect:

Our changes would have been propagated to Devtron after 20-30 minutes.

Recommended Resources for Production use

The overall resources required for the recommended production overrides are:

The production overrides can be applied as pre-devtron installation as well as post-devtron installation in the respective namespace.

Pre-Devtron Installation

If you want to install a new Devtron instance for production-ready deployments, this is the best option for you.

Create the namespace and apply the overrides files as stated above:

After files are applied, you are ready to install your Devtron instance with production-ready resources.

Post-Devtron Installation

If you have an existing Devtron instance and want to migrate it for production-ready deployments, this is the right option for you.

In the existing namespace, apply the production overrides as we do it above.

Ingress setup for Devtron Installation

After Devtron is installed, Devtron is accessible through service devtron-service. If you want to access devtron through ingress, edit devtron-service and change the loadbalancer to ClusterIP. You can do this using kubectl patch command like :

After that create ingress by applying the ingress yaml file. You can use to create ingress to access devtron:

Optionally you also can access devtron through a specific host like :

Enable HTTPS For Devtron

Once ingress setup for devtron is done and you want to run Devtron over https, you need to add different annotations for different ingress controllers and load balancers.

1. Nginx Ingress Controller

In case of nginx ingress controller, add following annotations under service.annotations under nginx ingress controller to run devtron over https.

(i) Amazon Web Services (AWS)

If you are using AWS cloud, add the following annotations under service.annotations under nginx ingress controller.

(ii) Digital Ocean

If you are using Digital Ocean cloud, add the following annotations under service.annotations under nginx ingress controller.

2. AWS Application Load Balancer (AWS ALB)

In case of AWS application load balancer, add following annotations under ingress.annotations to run devtron over https.

3. Azure Application Gateway

In case of AWS application load balancer, the following annotations need to be added under ingress.annotations to run devtron over https.

For an Ingress resource to be observed by AGIC (Application Gateway Ingress Controller) must be annotated with kubernetes.io/ingress.class: azure/application-gateway. Only then AGIC will work with the Ingress resource in question.

Note: Make sure NOT to use port 80 with HTTPS and port 443 with HTTP on the Pods.

Global Configurations

This documentation consists of the Global Configurations available in Devtron.

Parts of the Documentation

GitOps

Why Devtron takes GitOps Configuration?

Devtron uses GitOps and stores configurations in git; Git Credentials can be entered at Global Configuration > GitOps which is used by Devtron for configuration management and storing desired state of the application configuration. In case GitOps is not configured, Devtron cannot deploy any application or charts.

Areas impacted by GitOps are:

Deployment Template, to learn more.
Charts, to learn more.

Add Git Configuration

Select the GitOps section of global configuration. At the top of the section, four Git providers are available.

GitHub
GitLab
Azure
BitBucket Cloud

Select one of the Git providers. To add git account, You need to provide the following inputs as given below:

Git Host / Azure DevOps Organisation Url / BitBucket Host
GitHub Organization Name / Gitlab Group id / Azure DevOps Project Name / BitBucket Workspace ID
BitBucket Project Key (only for BitBucket Cloud)
Git access credential

1. Git Host:

This field is filled by default, Showing the URL of the selected git providers. For example- https://github.com for GitHub, https://gitlab.com for GitLab, https://dev.azure.com/ for Azure & https://bitbucket.org for BitBucket. Please replace them(not available for GitHub & BitBucket) if they are not the url you want to use.

2. GitHub Organization Name / GitLab Group ID / Azure DevOps Project Name / BitBucket Workspace ID:

3. BitBucket Project Key:

4. Git access credential

Provide Git Username and Personal Access Token of your git account.

(a) Username Username for your git account.

(b) Personal Access Token A personal access token (PAT) is used as an alternate password to authenticate into your git accounts.

repo - Full control of private repositories(Access commit status , Access deployment status and Access public repositories).
admin:org - Full control of organizations and teams(Read and write access).
delete_repo - Grants delete repo access on private repositories.

api - Grants complete read/write access to the scoped project API.
write_repository - Allows read-write access (pull, push) to the repository.

repo - Full control of repositories (Read, Write, Admin, Delete access).

Click on Save to save your gitOps configuration details.

Note: A Green tick will appear on the active gitOps provider.

Projects

Projects are nothing but a logical grouping of your applications so that you can manage and control the access level of users. We will discuss User Access in the next step.

Add Project:

Click on the Projects inside the Global configuration tab. Click on Add projects and give a name to your project and press the Save button to save your project

Cluster And Environments

The Global configuration provides a feature of Cluster & Environment in which you can add your Kubernetes clusters and environment.

Select the Cluster & Environment section of global configuration and click on Add Cluster to add your cluster.

Add Cluster:

To add a cluster on devtron, you must have superadmin access.

Navigate to the Global Configurations → Clusters and Environments on devtron and click on Add Cluster. Provide the below informations to add your kubernetes cluster:

Name
Kubernetes Cluster Info
- Server URL
- Bearer token
Prometheus Info
- Prometheus endpoint
  - Basic
    Username
    Password
  - Anonymous
- TLS Key
- TLS Certificate

1. Name

Give a name to your cluster inside the name box.

2. Kubernetes Cluster Info

Provide your kubernetes cluster’s credentials.

Server URL

Provide the endpoint/URL of your kubernetes cluster.It is recommended to use a self-hosted URL instead of cloud hosted. Self-hosted URL will provide the following benefits.

(a) Disaster Recovery - It is not possible to edit the server-url of a cluster. So if you're using an eks url, For eg- *****.eu-west-1.elb.amazonaws.com it will be a tedious task to add a new cluster and migrate all the services one by one. While using a self-hosted url For eg- clear.example.com you can just point to the new cluster's server url in DNS manager and update the new cluster token and sync all the deployments.

(b) Easy cluster migrations - Cluster url is given in the name of the cloud provider used, so migrating your cluster from one provider to another will result in waste of time and effort. On the other hand, if using a self-hosted url migrations will be easy as the url is of single hosted domain independent of the cloud provider.

To get the Server URL, run the following the command :

Bearer token

Provide your kubernetes cluster’s Bearer token for authentication purposes so that Devtron is able to communicate with your kubernetes cluster and can deploy your application in your kubernetes cluster.

Generate the Bearer Token by running the following command:

Please ensure that kubectl and jq are installed on the bastion on which you are running the command.

If you are using a microk8s cluster, run the following command to generate the bearer token:

3. Prometheus Info

Prometheus is a powerful solution to provide graphical insight into your application behavior. If you want to see your application matrix against your applications deployed in kubernetes, install Prometheus in your kubernetes cluster. The below inputs are required to configure your prometheus into Devtron’s tool.

Prometheus endpoint

Provide the URL of your prometheus. Prometheus supports two types of authentication Basic and Anonymous. Select the authentication type for your Prometheus setup.

Basic

If you select the basic type of authentication then you have to provide the Username and Password of prometheus for authentication.

Anonymous

If you select Anonymous then you do not have to provide any username and password for authentication.

TLS Key & TLS Certificate

TLS key and TLS certificate both options are optional, these options are used when you use a custom URL, in that case, you can pass your TLS key and TLS certificate.

K8s Version

on saving or update a cluster there is a call to fetch k8s version, it will store corresponding to cluster on db. used in listing api's and app detail page for grafana url.

Check the below screenshots to know how it looks like If you select the Basic authentication type

If you select the Anonymous authentication type

Now click on Save Cluster to save your cluster information.

Note:

Your kubernetes cluster gets mapped with the Devtron when you save your kubernetes cluster Configuration. Now the agents of devtron will be installed on your cluster so that the components of devtron can communicate to your cluster. When the agent starts installing on your cluster, you can check the status of the agents in the Cluster & Environment tab also.

Click on Details to check what got installed inside the agents. A new window will be popped up displaying all the details about these agents.

Add Environment

Once you have added your cluster in Cluster & Environment, you can add the environment also. Click on Add Environment, a window will be opened. Give a name to your environment in the Environment Name box and provide a namespace corresponding to your environment in the Namespace input box. Now choose if your environment is for Production purposes or for Non-production purposes. Production and Non-production options are only for tagging purposes. Click on Save and your environment will be created.

You can update an already created environment, Select and click on the environment which you want to update. You can only change Production and Non-production options here.

Note

You can not change the Environment name and Namespace name.

Click on Update to update your environment.

Namespaces And Environments

Namespaces

Kubernetes namespaces can be seen as a logical entity used to represent cluster resources for usage of a particular set of users. This logical entity can also be termed as a virtual cluster. One physical cluster can be represented as a set of multiple such virtual clusters (namespaces).

Multiple Namespaces

Namespaces are intended for use in environments with many users spread across multiple teams, or projects. Names of resources need to be unique within a namespace, but not across namespaces. Namespaces can not be nested inside one another and each Kubernetes resource can only be in one namespace

Environments

One of the advantages that Kubernetes provides is the ability to manage various environments easier and better than traditional deployment strategies. For most nontrivial applications, you have test, staging, and production environments. You can spin up a separate cluster of resources, such as VMs, with the same configuration in staging and production, but that can be costly and managing the differences between the environments can be difficult. Kubernetes includes a cool feature called namespaces, which enables you to manage different environments within the same cluster. For example, you can have different test and staging environments in the same cluster of machines, potentially saving resources.

Environments in Devtron can be accessed from:-

Global Configuration->Clusters & Environments

Here multiple environments can be created.

Git Accounts

Git Accounts allow you to connect your code source with Devtron. You will be able to use these git accounts to build the code using the CI pipeline.

Git Account Configuration

Global Configuration helps you to add a Git provider. Click on Add git account button at the top of the Git Account Section. To add a new git provider, add the details as mentioned below.

Name
Git Host
URL
Authentication type

1. Name

Provide a Name to your Git provider. This name will be displayed in the the Git Provider drop-down inside the Git Material configuration section.

2. Git Host

It is the git provider on which corresponding application git repository is hosted. By default you will get Bitbucket and GitHub but you can add many as you want clicking on [+ Add Git Host].

3. URL

4. Authentication type

Here provide the type of authentication required by your version controller. Devtron supports three types of authentications. You can choose the one that suits you the best.

Anonymous

If authentication type is set as Anonymous then you do not need to provide any username, password/authentication token or SSH key. Just click on Save to save the git account provider details.

If authentication type is set as Anonymous, only public git repository will be accessible.

User Auth

If you select User Auth then you have to provide the Username and either of Password or Auth Token for the authentication of your version controller account. Click on Save to save the git account provider details.

SSH Key

If you choose SSH Key then you have to provide the Private SSH Key corresponding to the public key added in your version controller account. Click on Save to save the git account provider details.

Update Git Account

You can update your saved git account settings at anytime. To update the git account:

Click on the git account which you want to update.
Make the required changes
Click on Update to save the changes.

Updates can only be made within one Authentication type or one protocol type, i.e. HTTPS(Anonymous or User Auth) & SSH. You can update from Anonymous to User Auth & vice versa, but not from Anonymous/User Auth to SSH or reverse.

Note:

Disabled git accounts will be unavailable for use in future applications. Applications already using a disabled git account will not be affected.

Container Registries

Container registries are used to store images built by the CI Pipeline. Here you can configure the container registry you want to use for storing images.

When configuring an application, you can choose which registry and repository it should use in the App Configuration > section.

Add Container Registry configuration:

Go to the Container Registry section of Global Configuration. Click on Add container registry.

You will see below the input fields to configure the container registry.

Name
Registry type
- ecr
  - AWS region
  - Access key ID
  - Secret access key
- docker hub
  - Username
  - Password
- Others
  - Username
  - password
Registry URL
Set as default

Name

Provide a name to your registry, this name will be shown to you in Docker Build Config as a drop-down.

Registry type

Here you can select the type of the Registry. We are supporting three types- docker hub, ecr and others. You can select any one of them from the drop-down. By default, this value is ecr. If you select ecr then you have to provide some information like- AWS region, Access Key, and Secret Key. If you select docker hub then you have to provide Username and Password. And if you select others then you have to provide the Username and Password.

Registry URL

Select any type of Registry from the drop-down, you have to provide the URL of your registry. Create your registry and provide the URL of that registry in the URL box.

Registry Type: ECR

To add an Amazon Elastic Container Registry (ECR), select the ECR Registry type. Amazon ECR is an AWS-managed container image registry service. The ECR provides resource-based permissions to the private repositories using AWS Identity and Access Management (IAM). ECR allows both Key-based and Role-based authentications.

To set this ECR as the default registry hub for your images, select [x] Set as default registry.

Select Save.

Registry Type- Docker Hub

You have to provide the below information if you select the registry type as Docker Hub.

Username

Give the username of the docker hub account you used for creating your registry in.

Password

Registry Type Others:

You have to provide the below information if you select the registry type as others.

Username

Give the username of your account, where you have created your registry in.

Password

Give the password corresponding to the username of your registry.

Set as default:

If you enable the Set as default option, then this registry name will be set as default in the Container Registry section inside the Docker build config page. This is optional. You can keep it disabled.

Advance Registry Url connection options:

If you enable the Allow Only Secure Connection option, then this registry allows only secure connections.
If you enable the Allow Secure Connection With CA Certificate option, then you have to upload/provide private CA certificate (ca.crt).
If the container registry is insecure (for eg : SSL certificate is expired), then you enable the Allow Insecure Connection option.

Now click on Save to save the configuration of the Container registry.

Note:

You can use any registry which can be authenticated using docker login -u <username> -p <password> <registry-url>. However these registries might provide a more secured way for authentication, which we will support later. Some popular registries which can be used using username and password mechanism:

Integrating With External Container Registry

If you want to use a private registry for container registry other than ecr, this will be used to push image and then create a secret in same environment to pull the image to deploy. To create secret, go to charts section and search for chart ‘dt-secrets’ and configure the chart. Provide an App Name and select the Project and Environment in which you want to deploy this chart and then configure the values.yaml as shown in example. The given example is for DockerHub but you can configure similarly for any container registry that you want to use.

Chart Repositories

This feature allows you to add more chart repositories to Devtron. Once added they will be available in the Discover section of the Chart Store.

Learn more about

Note : After the successfull installation of Devtron, click on Refresh Charts to sync & download all the default charts listed on the dashboard.

Add Chart Repository

Select the Chart Repository section of global configuration and click on Add Repository button at the top of the Chart Repository Section. To add new chart, you need to provide three inputs as below:

Name
URL
Authentication type

1. Name

Provide a Name to your Chart Repository. This name is added as prefix to the name of the chart in the listing on the helm chart section of application.

2. URL

3. Authentication type

Here you have to provide the type of Authentication required by your version controller. We support three types of authentications, You can choose the one that suits you the best.

Anonymous

If you select Anonymous then you do not have to provide any username, password, or authentication token. Just click on Save to save your chart repository details.

Password/Auth token

If you select Password/Auth token, then you have to provide the Access Token for the authentication of your version controller account inside the Access token box. Click on Save to save your chart repository details.

User Auth

If you choose User Auth then you have to provide the Username and Password of your version controller account. Click on Save to save your chart repository details.

Update Chart Repository

You can update your saved chart repository settings at any point in time. Just click on the chart repository which you want to update. Make the required changes and click on Update to save you changes.

Note: You can enable and disable your chart repository setting. If you enable it, then you will be able to see that enabled chart in Discover section of Chart Store.

Custom charts

Devtron includes predefined helm charts that cover the majority of use cases. For any use case not addressed by the default helm charts, you can upload your own helm chart and use it as a custom chart in Devtron.

Who can upload a custom chart - Super admins
Who can use the custom chart - All users

A super admin can upload multiple versions of a custom helm chart.

Prerequisites

A valid helm chart, which contains Chart.yaml file with name and version fields.
Image descriptor template file - .image_descriptor_template.json.
Custom chart packaged in the *.tgz format.

1. How to create a helm chart

2. Create the image descriptor template file - `.image_descriptor_template.json`

It's a GO template file that should produce a valid JSON file upon rendering. This file is passed as the last argument in helm install -f myvalues.yaml -f override.yaml command.

Place the .image_descriptor_template.json file in the root directory of your chart.

You can use the following variables in the helm template (all the placeholders are optional):

The values from the CD deployment pipeline are injected at the placeholder specified in the .image_descriptor_template.json template file.

For example:
To create a template file to allow Devtron to only render the repository name and the tag from the CI/CD pipeline that you created, edit the .image_descriptor_template.json file as:

3. Package the custom chart in the `*.tgz` format

Before you begin, ensure that your helm chart includes both Chart.yaml (with name and version fields) and .image_descriptor_template.json files.

The helm chart to be uploaded must be packaged as a versioned archive file in the format - <helm-chart-name>-vx.x.x.tgz.

The above command will create a my-custom-chart-0.1.0.tgz file.

Uploading a custom chart

A custom chart can only be uploaded by a super admin.

On the Devtron dashboard, select Global Configurations > Custom charts.
Select Import Chart.
Choose Select tar.gz file... and upload the packaged custom chart in the *.tgz format.

The chart is being uploaded and validated. You may also Cancel upload if required.

Validation

The uploaded archive will be validated against:

Supported archive template should be in *.tgz format.
Chart.yaml must include the name and the version number.
image_descriptor_template.json file should be present and the field format must match the format listed in the image builder template section.

The following are the validation results:

View the custom charts

All users can view the custom charts.

To view a list of available custom charts, go to Global Configurations > Custom charts page.

The charts can be searched with their name, version, or description.

Use the custom chart in an application

Info:

Authorization

This documentaion consist of authorizations available in Devtron

Parts of the documentaion

Usage

Resources

Deprecated

User Permissions

Like any enterprise product, Devtron supports fine grained access control to the resources based on

Type of action allowed on the Devtron resources (Create Vs View)
Sensitivity of the data (Editing image Vs Editing memory)

Access can be added to the User either directly or via Groups.

Role-based Access Levels

Devtron supports 5 levels of access:

View: User with view only access has the least privilege. This user can only view combination of environments, applications and helm charts on which access has been granted to the user. This user cannot view sensitive data like secrets used in applications or charts.
Build and Deploy: In addition to view privilege mentioned in above, user with build and deploy permission can build and deploy the image of permitted applications and helm charts to permitted environments.
Admin: User with admin access can create, edit, delete and view permitted applications in permitted projects.
Manager: User with manager access can do everything that an admin type user can do, in addition they can also give and revoke access of users for the applications and environments of which they are manager.
Super Admin: User with super admin privilege has unrestricted access to all Devtron resources. Super admin can create, modify, delete and view any Devtron resource without any restriction; its like Superman without the weakness of Kryptonite. Super Admin can also add and delete user access across any Devtron resource, add delete git repository credentials, container registry credentials, cluster and environment.

User Roles And Permissions

1. Custom Applications

2. Helm Charts

3. User Access

4. Global Configurations

To control the access of User and Group-

Go to the left main panel -> Select Global Configurations -> Select User Access

Users

1. Add new user

Click on Add User, to add one or multiple users.

2. Create User Permissions

When you click on Add User, you will see 6 options to set permission for users which are as follow:

Email addresses
Assign super admin permissions
Group Permissions
Devtron Apps Permissions
- Project
- Environment
- Applications
- Roles
Helm Apps Permissions
- Project
- Environment or cluster/namespace
- Applications
- Permission
Chart group permissions

Email addresses:

In the Email address box, you have to provide the mail ID of the user to whom you want to give access to your applications.

IMP Please note that Email address should be same as that in the email field in the JWT token returned by OIDC provider.

Assign super admin permissions

If you check the option Assign super admin permissions, the user will get full access to your system and the rest of the options will disappear. Please check above to see permission levels. Only users with super admin permissions can assign super admin permissions to a user.

Click on Save and your user will be saved with super admin permissions.

We suggest that super admin privileges should be given to only select few.

If you don’t want to assign super admin permissions then you have to provide the rest of the information.

Devtron Apps permissions

Access to devtron applications can be given to user by attaching permission directly to his/her email id through the Devtron Apps section. This section has 4 options to manage the permissions of your users.

Project

Select a project from the drop-down to which you want to give permission to the users. You can select only one project at a time if you want to select more than one project then click Add row.

Environment

In the Environment section, you can select one or more than one or all environments at a time. Click on the environment section, you will see a drop-down of your environments and select any environment on which you want to give permission to the user.

IMP If all environments option is selected then user gets access to all current environments and any new environment which gets associated with this application later.

Applications

Similarly, you can select Applications from the drop-down corresponding to your selected Environments. In this section, you can also give permissions to one or more than one or to all applications at a time.

IMP If all applications option is selected then user gets access to all current applications and any new application which gets associated with this project later.

Roles
Inside the Role, you actually choose which type of permissions you want to give to the users.

There are four different view access levels/Role available for both User and Group as described above:

You can add multiple rows, for Devtron app permission.

Once you have finished assigning the appropriate permissions for the listed users, Click on Save.

Helm Apps Permissions

Project

Select a project from the drop-down to which you want to give permission to the users. You can select only one project at a time if you want to select more than one project then click Add row.

Environment or cluster/namespace

IMP If all environments option is selected then user gets access to all current environments and any new environment which gets associated with this application later.

Applications

IMP If all applications option is selected then user gets access to all current applications and any new application which gets associated with this project later.

Permission
Inside the Role, you actually choose which type of permissions you want to give to the users.

There are four different view access levels/Role available for both User and Group as described above:

Chart Group Permissions

You can also manage the access of users to Chart Groups in your project.

NOTE: You can only give users the ability to create or edit, not both.

Click on the checkbox of Create, if you want the users to create, view, edit, or delete the chart groups.

To permit a user to only edit the chart groups, check Specific chart group from Edit drop-down. In the following field, select the chart group for which you want to grant the user edit permission.

Go to Edit drop-down, if you want to allow or deny users to edit the chart groups.

Select on Deny option from the drop-down, if you want to restrict the users to edit the chart groups.

Select the Specific Charts option from the drop-down and then select the chart groups for which you want to allow users to edit, from the other drop-down menu.

Click on Save, once you have configured all the required permissions for the users.

3. Edit User Permissions

You can edit the user permissions, by clicking on the downward arrow.

Then you can edit the user permissions here.

After you have done editing the user permissions, click on Save.

If you want to delete the user/users with particular permissions, click on Delete.

Rollout Deployment

Deployment configuration is the Manifest for the application, it defines the runtime behavior of the application. You can define application behavior by providing information in three sections:

Chart Version
Yaml file
Show application metrics

1. Chart version

Devtron uses helm charts for the deployments. And we are having multiple chart versions based on features it is supporting with every chart version.

One can see multiple chart version options available in the drop-down. you can select any chart version as per your requirements. By default, the latest version of the helm chart is selected in the chart version option.

Every chart version has its own YAML file. Helm charts are used to provide specifications for your application. To make it easy to use, we have created templates for the YAML file and have added some variables inside the YAML. You can provide or change the values of these variables as per your requirement.

Application Metrics is not supported for Chart version older than 3.7 version.

2. Yaml file

Container Ports

This defines ports on which application services will be exposed to other services

EnvVariables

EnvVariables provide run-time information to containers and allow to customize how the application works and the behavior of the applications on the system.

Here we can pass the list of env variables , every record is an object which contain the name of variable along with value.

To set environment variables for the containers that run in the Pod.

Example of env variables

IMP Docker image should have env variables, whatever we want to set.

But ConfigMap and Secret are the prefered way to inject env variables. So we can create this in App Configuration Section

ConfigMap

It is a centralized storage, specific to k8s namespace where key-value pairs are stored in plain text.

Secret

It is a centralized storage, specific to k8s namespace where we can store the key-value pairs in plain text as well as in encrypted(Base64) form.

IMP All key-values of Secret and CofigMap will reflect to your application.

Liveness Probe

If this check fails, kubernetes restarts the pod. This should return error code in case of non-recoverable error.

MaxUnavailable

The maximum number of pods that can be unavailable during the update process. The value of "MaxUnavailable: " can be an absolute number or percentage of the replicas count. The default value of "MaxUnavailable: " is 25%.

MaxSurge

The maximum number of pods that can be created over the desired number of pods. For "MaxSurge: " also, the value can be an absolute number or percentage of the replicas count. The default value of "MaxSurge: " is 25%.

Min Ready Seconds

This specifies the minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing, for it to be considered available. This defaults to 0 (the Pod will be considered available as soon as it is ready).

Readiness Probe

If this check fails, kubernetes stops sending traffic to the application. This should return error code in case of errors which can be recovered from if traffic is stopped.

Autoscaling

This is connected to HPA and controls scaling up and down in response to request load.

Fullname Override

fullnameOverride replaces the release fullname created by default by devtron, which is used to construct Kubernetes object names. By default, devtron uses {app-name}-{environment-name} as release fullname.

Image

Image is used to access images in kubernetes, pullpolicy is used to define the instances calling the image, here the image is pulled when the image is not present,it can also be set as "Always".

imagePullSecrets

imagePullSecrets contains the docker credentials that are used for accessing a registry.

Ingress

This allows public access to the url, please ensure you are using right nginx annotation for nginx class, its default value is nginx

Legacy deployment-template ingress format

Ingress Internal

This allows private access to the url, please ensure you are using right nginx annotation for nginx class, its default value is nginx

Init Containers

Specialized containers that run before app containers in a Pod. Init containers can contain utilities or setup scripts not present in an app image. One can use base image inside initContainer by setting the reuseContainerImage flag to true.

Pause For Seconds Before Switch Active

To wait for given period of time before switch active the container.

Resources

These define minimum and maximum RAM and CPU available to the application.

Resources are required to set CPU and memory usage.

Limits

Limits make sure a container never goes above a certain value. The container is only allowed to go up to the limit, and then it is restricted.

Requests

Requests are what the container is guaranteed to get.

Service

This defines annotations and the type of service, optionally can define name also.

Note - If loadBalancerSourceRanges is not set, Kubernetes allows traffic from 0.0.0.0/0 to the LoadBalancer / Node Security Group(s).

Volumes

It is required when some values need to be read from or written to an external disk.

Volume Mounts

It is used to provide mounts to the volume.

Affinity and anti-affinity

Spec is used to define the desire state of the given container.

Node Affinity allows you to constrain which nodes your pod is eligible to schedule on, based on labels of the node.

Inter-pod affinity allow you to constrain which nodes your pod is eligible to be scheduled based on labels on pods.

Key

Key part of the label for node selection, this should be same as that on node. Please confirm with devops team.

Values

Value part of the label for node selection, this should be same as that on node. Please confirm with devops team.

Tolerations

Taints are the opposite, they allow a node to repel a set of pods.

A given pod can access the given node and avoid the given taint only if the given pod satisfies a given taint.

Taints and tolerations are a mechanism which work together that allows you to ensure that pods are not placed on inappropriate nodes. Taints are added to nodes, while tolerations are defined in the pod specification. When you taint a node, it will repel all the pods except those that have a toleration for that taint. A node can have one or many taints associated with it.

Arguments

This is used to give arguments to command.

Command

It contains the commands to run inside the container.

Containers

Containers section can be used to run side-car containers along with your main container within same pod. Containers running within same pod can share volumes and IP Address and can address each other @localhost.

Prometheus

It is a kubernetes monitoring tool and the name of the file to be monitored as monitoring in the given case.It describes the state of the prometheus.

rawYaml

Accepts an array of Kubernetes objects. You can specify any kubernetes yaml here and it will be applied when your app gets deployed.

Grace Period

Kubernetes waits for the specified time called the termination grace period before terminating the pods. By default, this is 30 seconds. If your pod usually takes longer than 30 seconds to shut down gracefully, make sure you increase the GracePeriod.

A Graceful termination in practice means that your application needs to handle the SIGTERM message and begin shutting down when it receives it. This means saving all data that needs to be saved, closing down network connections, finishing any work that is left, and other similar tasks.

There are many reasons why Kubernetes might terminate a perfectly healthy container. If you update your deployment with a rolling update, Kubernetes slowly terminates old pods while spinning up new ones. If you drain a node, Kubernetes terminates all pods on that node. If a node runs out of resources, Kubernetes terminates pods to free those resources. It’s important that your application handle termination gracefully so that there is minimal impact on the end user and the time-to-recovery is as fast as possible.

Server

It is used for providing server configurations.

Deployment

It gives the details for deployment.

Service Monitor

It gives the set of targets to be monitored.

Db Migration Config

It is used to configure database migration.

Application Metrics

Application metrics can be enabled to see your application's metrics-CPUService Monito usage,Memory Usage,Status,Throughput and Latency.

Deployment Metrics

It gives the realtime metrics of the deployed applications

Addon features in Deployment Template Chart version 3.9.0

Service Account

A service account provides an identity for the processes that run in a Pod.

When you access the cluster, you are authenticated by the API server as a particular User Account. Processes in containers inside pod can also contact the API server. When you are authenticated as a particular Service Account.

When you create a pod, if you do not create a service account, it is automatically assigned the default service account in the namespace.

Pod Disruption Budget

You can create PodDisruptionBudget for each application. A PDB limits the number of pods of a replicated application that are down simultaneously from voluntary disruptions. For example, an application would like to ensure the number of replicas running is never brought below the certain number.

You can specify maxUnavailable and minAvailable in a PodDisruptionBudget.

With minAvailable of 1, evictions are allowed as long as they leave behind 1 or more healthy pods of the total number of desired replicas.

With maxAvailable of 1, evictions are allowed as long as at most 1 unhealthy replica among the total number of desired replicas.

Application metrics Envoy Configurations

Envoy is attached as a sidecar to the application container to collect metrics like 4XX, 5XX, Throughput and latency. You can now configure the envoy settings such as idleTimeout, resources etc.

Prometheus Rule

Alerting rules allow you to define alert conditions based on Prometheus expressions and to send notifications about firing alerts to an external service.

In this case, Prometheus will check that the alert continues to be active during each evaluation for 1 minute before firing the alert. Elements that are active, but not firing yet, are in the pending state.

Pod Labels

Labels are key/value pairs that are attached to pods. Labels are intended to be used to specify identifying attributes of objects that are meaningful and relevant to users, but do not directly imply semantics to the core system. Labels can be used to organize and to select subsets of objects.

Pod Annotations

Pod Annotations are widely used to attach metadata and configs in Kubernetes.

Custom Metrics in HPA

HPA, by default is configured to work with CPU and Memory metrics. These metrics are useful for internal cluster sizing, but you might want to configure wider set of metrics like service latency, I/O load etc. The custom metrics in HPA can help you to achieve this.

Wait For Seconds Before Scaling Down

Wait for given period of time before scaling down the container.

3. Show Application Metrics

If you want to see application metrics like different HTTP status codes metrics, application throughput, latency, response time. Enable the Application metrics from below the deployment template Save button. After enabling it, you should be able to see all metrics on App detail page. By default it remains disabled.

Helm Chart Json Schema Table

Helm Chart json schema is used to validate the deployment template values.

Other Validations in Json Schema

The values of CPU and Memory in limits must be greater than or equal to in requests respectively. Similarly, In case of envoyproxy, the values of limits are greater than or equal to requests as mentioned below.

Addon features in Deployment Template Chart version 4.11.0

KEDA Autoscaling

Example for autosccaling with KEDA using Prometheus metrics is given below:

Example for autosccaling with KEDA based on kafka is given below :

Security Context

A security context defines privilege and access control settings for a Pod or Container.

To add a security context for main container:

To add a security context on pod level:

Topology Spread Constraints

You can use topology spread constraints to control how Pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains. This can help to achieve high availability as well as efficient resource utilization.

CI Pipeline

Info:
For Devtron version older than v0.4.0, please refer the page.

A CI Pipeline can be created in one of the three ways:

Each of these methods has different use-cases that can be tailored to the needs of the organization.

1. Continuous Integration

Continuous Integration Pipeline allows you to build the container image from a source code repository.

From the Applications menu, select your application.
On the App Configuration page, select Workflow Editor.
Select + New Build Pipeline.
Select Continuous Integration.
Enter the following fields on the Create build pipeline screen:

Advanced Options

The advanced CI Pipeline includes the following stages:

Pre-build stage: The tasks in this stage run before the image is built.
Build stage: In this stage, the build is triggered from the source code that you provide.
Post-build stage: The tasks in this stage are triggered once the build is complete.

Scan for vulnerabilities

To Perform the security scan after the container image is built, enable the Scan for vulnerabilities toggle in the build stage.

Build stage

The Build stage allows you to configure a build pipeline from the source code.

From the Create build pipeline screen, select Advanced Options.
Select Build stage.

Select Update Pipeline.

Source type: Branch Fixed

The Source type - "Branch Fixed" allows you to trigger a CI build whenever there is a code change on the specified branch.

Select the Source type as "Branch Fixed" and enter the Branch Name.

Source type: Branch Regex

Branch Regex allows users to easily switch between branches matching the configured Regex before triggering the build pipeline. In case of Branch Fixed, users cannot change the branch name in ci-pipeline unless they have admin access for the app. So, if users with Build and Deploy access should be allowed to switch branch name before triggering ci-pipeline, Branch Regex should be selected as source type by a user with Admin access.

For example if the user sets the Branch Regex as feature-*, then users can trigger from branches such as feature-1450, feature-hot-fix etc.

Configuring Webhook

Info: If you choose "Pull Request" or "Tag Creation" as the source type, you must first configure the Webhook for GitHub/Bitbucket as a prerequisite step.

1. Configure Webhook for GitHub

Go to the Settings page of your repository and select Webhooks.
Select Add webhook.
In the Payload URL field, enter the Webhook URL that you get on selecting the source type as "Pull Request" or "Tag Creation" in Devtron the dashboard.
Change the Content-type to application/json.
In the Secret field, enter the secret from Devtron the dashboard when you select the source type as "Pull Request" or "Tag Creation".

Under Which events would you like to trigger this webhook?, select Let me select individual events. to trigger the webhook to build CI Pipeline.
Select Branch or tag creation and Pull Requests.
Select Add webhook.

2. Configure Webhook for Bitbucket cloud

Go to the Repository settings page of your Bitbucket repository.
Select Webhooks and then select Add webhook.

Enter a Title for the webhook.
In the URL field, enter the Webhook URL that you get on selecting the source type as "Pull Request" or "Tag Creation" in the Devtron dashboard.
Select the event triggers for which you want to trigger the webhook.
Select Save to save your configurations.

Source type: Pull Request

The Source type - "Pull Request" allows you to configure the CI Pipeline using the PR raised in your repository.

To trigger the build from specific PRs, you can filter the PRs based on the following keys:

Select the appropriate filter and pass the matching condition as a regular expression (regex).

Select Create Pipeline.

Source type: Tag Creation

The Source type - "Tag Creation" allows you to build the CI pipeline from a tag.

To trigger the build from specific tags, you can filter the tags based on the author and/or the tag name.

Select the appropriate filter and pass the matching condition as a regular expression (regex).

Select Create Pipeline.

Note
(a) You can provide pre-build and post-build stages via the Devtron tool’s console or can also provide these details by creating a file devtron-ci.yaml inside your repository. There is a pre-defined format to write this file. And we will run these stages using this YAML file. You can also provide some stages on the Devtron tool’s console and some stages in the devtron-ci.yaml file. But stages defined through the Devtron dashboard are first executed then the stages defined in the devtron-ci.yaml file.
(b) The total timeout for the execution of the CI pipeline is by default set as 3600 seconds. This default timeout is configurable according to the use case. The timeout can be edited in the configmap of the orchestrator service in the env variable as env:"DEFAULT_TIMEOUT" envDefault:"3600"

2. Linked CI Pipeline

If one code is shared across multiple applications, Linked CI Pipeline can be used, and only one image will be built for multiple applications because if there is only one build, it is not advisable to create multiple CI Pipelines.

From the Applications menu, select your application.
On the App Configuration page, select Workflow Editor.
Select + New Build Pipeline.
Select Linked CI Pipeline.
Enter the following fields on the Create linked build pipeline screen:

Select the application in which the source CI pipeline is present.
Select the source CI pipeline from the application that you selected above.
Enter a name for the linked CI pipeline.

Select Create Linked CI Pipeline.

After creating a linked CI pipeline, you can create a CD pipeline. Builds cannot be triggered from a linked CI pipeline; they can only be triggered from the source CI pipeline. There will be no images to deploy in the CD pipeline created from the 'linked CI pipeline' at first. To see the images in the CD pipeline of the linked CI pipeline, trigger build in the source CI pipeline. The build images will now be listed in the CD pipeline of the 'linked CI pipeline' whenever you trigger a build in the source CI pipeline.

3. Incoming Webhook

The CI pipeline receives container images from an external source via a webhook service.

You can use Devtron for deployments on Kubernetes while using your CI tool such as Jenkins. External CI features can be used when the CI tool is hosted outside the Devtron platform.

From the Applications menu, select your application.
On the App Configuration page, select Workflow Editor.
Select + New Build Pipeline.
Select Incoming Webhook.

Select Save and Generate URL. This generates the Payload format and Webhook URL.

You can send the Payload script to your CI tools such as Jenkins and Devtron will receive the build image every time the CI Service is triggered or you can use the Webhook URL which will build an image every time CI Service is triggered using Devtron Dashboard.

Update CI Pipeline

You can update the configurations of an existing CI Pipeline except for the pipeline's name. To update a pipeline, select your CI pipeline. In the Edit build pipeline window, edit the required stages and select Update Pipeline.

Delete CI Pipeline

You can only delete a CI pipeline if there is no CD pipeline created in your workflow.

To delete a CI pipeline, go to App Configurations > Workflow Editor and select Delete Pipeline.

CD Pipeline

Once you are done creating your CI pipeline, you can move start building your CD pipeline. Devtron enables you to design your CD pipeline in a way that fully automates your deployments.

Creating CD Pipeline

Click on “+” sign on CI Pipeline to attach a CD Pipeline to it. A basic Create deployment modal will pop up.

This section expects two inputs:

Select Environment
Deployment Strategy

1. Select Environment

This section further including two inputs:

(a) Deploy to Environment

Select the environment where you want to deploy your application.

(b) Namespace

This field will be automatically populated with the Namespace corresponding to the Environment selected in the previous step.

Click on Create Pipeline to create a CD pipeline.

One can have a single CD pipeline or multiple CD pipelines connected to the same CI Pipeline. Each CD pipeline corresponds to only one environment, or in other words, any single environment of an application can have only one CD pipeline. So, the images created by the CI pipeline can be deployed into multiple environments through different CD pipelines originating from a single CI pipeline. If you already have one CD pipeline and want to add more, you can add them by clicking on the + sign and then choosing the environment in which you want to deploy your application. Once a new CD Pipeline is created for the environment of your choosing, you can move ahead and configure the CD pipeline as required. Your CD pipeline can be configured for the pre-deployment stage, the deployment stage, and the post-deployment stage. You can also select the deployment strategy of your choice. You can add your configurations as explained below:

To configure the advance CD option click on Advance Options at the bottom.

1. Pipeline Name

Pipeline name will be autogenerated.

2. Deploy to Environment

As we discussed above, Select the environment where you want to deploy your application. Once you select the environment, it will display the Namespace corresponding to your selected environment automatically.

Stages

There are 3 dropdowns given below:

Pre-deployment stage
Deployment stage
Post-deployment stage

3. Pre-deployment stage

Sometimes one has a requirement where certain actions like DB migration are to be executed before deployment, the Pre-deployment stage should be used to configure these actions.

Pre-deployment stages can be configured to be executed automatically or manually.

If you select automatic, Pre-deployment Stage will be triggered automatically after the CI pipeline gets executed and before the CD pipeline starts executing itself. But, if you select a manual, then you have to trigger your stage via console.

If you want to use some configuration files and secrets in pre-deployment stages or post-deployment stages, then you can use the Config Maps & Secrets options.

Config Maps can be used to define configuration files. And Secrets can be defined to store the private data of your application.

Once you are done defining Config Maps & Secrets, you will get them as a drop-down in the pre-deployment stage and you can select them as part of your pre-deployment stage.

These Pre-deployment CD / Post-deployment CD pods can be created in your deployment cluster or the devtron build cluster. It is recommended that you run these pods in the Deployment cluster so that your scripts (if there are any) can interact with the cluster services that may not be publicly exposed.

If you want to run it inside your application, then you have to check the Execute in application Environment option else leave it unchecked to run it within the Devtron build cluster.

Make sure your cluster has devtron-agent installed if you check the Execute in the application Environment option.

4. Deployment stage

(a) Deploy to Environment

Select the environment where you want to deploy your application. Once you select the environment, it will display the Namespace corresponding to your selected environment automatically.

(b)We support two methods of deployments - Manual and Automatic. If you choose automatic, it will trigger your CD pipeline automatically once the corresponding CI pipeline has been executed successfully.

If you have defined pre-deployment stages, then the CD Pipeline will be triggered automatically after the successful execution of your CI pipeline followed by the successful execution of your pre-deployment stages. But if you choose the manual option, then you have to trigger your deployment manually via console.

(c) Deployment Strategy

Devtron's tool has 4 types of deployment strategies. Click on Add Deployment strategy and select from the available options:

(a) Recreate

(b) Canary

(d) Rolling

5. Post-deployment Stage

If you want to Configure actions like Jira ticket close, that you want to run after the deployment, you can configure such actions in the post-deployment stages.

Post-deployment stages are similar to pre-deployment stages. The difference is, pre-deployment executes before the CD pipeline execution and post-deployment executes after the CD pipeline execution. The configuration of post-deployment stages is similar to the pre-deployment stages.

You can use Config Map and Secrets in post deployments as well, as defined in the Pre-Deployment stages.

Once you have configured the CD pipeline, click on Create Pipeline to save it. You can see your newly created CD Pipeline on the Workflow tab attached to the corresponding CI Pipeline.

Update CD Pipeline

You can update the deployment stages and the deployment strategy of the CD Pipeline whenever you require it. But, you cannot change the name of a CD Pipeline or its Deployment Environment. If you need to change such configurations, you need to make another CD Pipeline from scratch.

To Update a CD Pipeline, go to the App Configurations section, Click on Workflow editor and then click on the CD Pipeline you want to Update.

Make changes as needed and click on Update Pipeline to update this CD Pipeline.

Delete CD Pipeline

If you no longer require the CD Pipeline, you can also Delete the Pipeline.

To Delete a CD Pipeline, go to the App Configurations and then click on the Workflow editor. Now click on the pipeline you want to delete. A pop will be displayed with CD details. Verify the name and the details to ensure that you are not accidentally deleting the wrong CD pipeline and then click on the Delete Pipeline option to delete the CD Pipeline.

Deployment Strategies

A deployment strategy is a way to make changes to an application, without downtime in a way that the user barely notices the changes. There are different types of deployment strategies like Blue/green Strategy, Rolling Strategy, Canary Strategy, Recreate Strategy. These deployment configuration-based strategies are discussed in this section.

Blue Green Strategy

Blue-green deployments involve running two versions of an application at the same time and moving traffic from the in-production version (the green version) to the newer version (the blue version).

blueGreen:
  autoPromotionSeconds: 30
  scaleDownDelaySeconds: 30
  previewReplicaCount: 1
  autoPromotionEnabled: false

Rolling Strategy

A rolling deployment slowly replaces instances of the previous version of an application with instances of the new version of the application. Rolling deployment typically waits for new pods to become ready via a readiness check before scaling down the old components. If a significant issue occurs, the rolling deployment can be aborted.

rolling:
  maxSurge: "25%"
  maxUnavailable: 1

Canary Strategy

Canary deployments are a pattern for rolling out releases to a subset of users or servers. The idea is to first deploy the change to a small subset of servers, test it, and then roll the change out to the rest of the servers. The canary deployment serves as an early warning indicator with less impact on downtime: if the canary deployment fails, the rest of the servers aren't impacted.

canary:
  maxSurge: "25%"
  maxUnavailable: 1
  steps:
    - setWeight: 25
    - pause:
        duration: 15 # 1 min
    - setWeight: 50
    - pause:
        duration: 15 # 1 min
    - setWeight: 75
    - pause:
        duration: 15 # 1 min

Recreate

The recreate strategy is a dummy deployment that consists of shutting down version A then deploying version B after version A is turned off. A recreate deployment incurs downtime because, for a brief period, no instances of your application are running. However, your old code and new code do not run at the same time.

recreate:

It terminates the old version and releases the new one.

Does your app has different requirements in different Environments? Also read Environment Overrides

Creating Sequential Pipelines

Devtron now supports attaching multiple deployment pipelines to a single build pipeline, in its workflow editor. This feature lets you deploy an image first to stage, run tests and then deploy the same image to production.

Please follow the steps mentioned below to create sequential pipelines :

After creating CI/build pipeline, create a CD pipeline by clicking on the + sign on CI pipeline and configure the CD pipeline as per your requirements.
To add another CD Pipeline sequentially after previous one, again click on + sign on the last CD pipeline.
Similarly, you can add multiple CD pipelines by clicking + sign of the last CD pipeline, each deploying in different environments.

Note: Deleting a CD pipeline also deletes all the K8s resources associated with it and will bring a disruption in the deployed micro-service. Before deleting a CD pipeline, please ensure that the associated resources are not being used in any production workload.

CI Pipeline (Legacy)

CI Pipeline can be created in three different ways, Continuous Integration, Linked CI Pipeline and Incoming Webhook.

Each of these methods have different use-cases which can be used according to the needs of the organization. Let’s begin with Continuous Integration.

A. Continuous Integration

Click on Continuous Integration, a prompt comes up in which we need to provide our custom configurations. Below is the description of some configurations which are required.

[Note] Options such as pipeline execution, stages and scan for vulnerabilities, will be visible after clicking on advanced options present in the bottom left corner.

I. Pipeline Name

Pipeline name is an auto-generated name which can also be renamed by clicking on Advanced options.

II. Pipeline Execution

You can select the method you want to execute the pipeline. By default the value is automatic. In this case it will get automatically triggered if any changes are made to the respective git repository. You can set it to manual if you want to trigger the pipeline manually.

III. Source Type

In source type, we can observe that we have three types of mechanisms which can be used for building your CI Pipeline. In the drop-down you can observe we have Branch Fixed, Pull Request and Tag Creation.

i) Branch Fixed

If you select the Branch Fixed as your source type for building CI Pipeline, then you need to provide the corresponding Branch Name.

Branch Name is the name of the corresponding branch (eg. main or master, or any other branch)

ii) Pull Request

[Note] It only works if Git Host is Github or Bitbucket Cloud as of now. In case you need support for any other Git Host, please create a github issue.

If you select the Pull Request option, you can configure the CI Pipeline using the generated PR. For this mechanism you need to configure a webhook for the repository added in the Git Material.

Prerequisites for Pull Request

If using GitHub - To use this mechanism, as stated above you need to create a webhook for the corresponding repository of your Git Provider. In Github to create a webhook for the repository -

Go to settings of that particular repository
Click on webhook section under options tab
In the Payload URL section, please copy paste the Webhook URL which can be found at Devtron Dashboard when you select source type as Pull Request as seen in above image.
Change content type to - application/json
Copy paste the Secret as well from the Dashboard when you select the source type as Pull Request

Now, scroll down and select the custom events for which you want to trigger the webhook to build CI Pipeline -

Check the radio button for Let me select individual events
Then, check the Branch or Tag Creation and Pull Request radio buttons under the individual events as mentioned in image below.

[Note] If you select Branch or Tag Creation, it will work for the Tag Creation mechanism as well.

After selecting the respective options, click on the generate the webhook button to create a webhook for your respective repository.

If using Bitbucket Cloud - If you are using Bitbucket cloud as your git provider, you need to create a webhook for that as we created for Github in the above section. Follow the steps to create webhook -

Go to Repository Settings on left sidebar of repository window
Click on Webhooks and then click on Add webhook as shown in the image.

Give any appropriate title as per your choice and then copy-paste the url which you can get from Devtron Dashboard when you select Pull Request as source type in case of Bitbucket Cloud as Git Host.
Check the Pull Request events for which you want to trigger the webhook and then save the configurations.

Filters

Now, coming back to the Pull Request mechanism, you can observe we have the option to add filters. In a single repository we have multiple PRs generated, so to have the exact PR for which you want to build the CI Pipeline, we have this feature of filters.

You can add a few filters which can be seen in the dropdown to sort the exact PR which you want to use for building the pipeline.

Below are the details of different filters which you can use as per your requirement. Please select any of the filters and pass the value in regex format as one has already given for example and then click on Create Pipeline.

Devtron uses regexp library, view regexp cheatsheet. You can test your custom regex from here.

iii) Tag Creation

The third option i.e, Tag Creation. In this mechanism you need to provide the tag name or author to specify the exact tag for which you want to build the CI Pipeline. To work with this feature as well, you need to configure the webhook for either Github or Bitbucket as we did in the previous mechanism i.e, Pull Request.

In this process as well you can find the option to filter the specific tags with certain filter parameters. Select the appropriate filter as per your requirement and pass the value in form of regex, one of the examples is already given.

Select the appropriate filter and pass the value in the form of regex and then click on Create Pipeline.

Advanced Options

When you click on the advanced options button which can be seen at the bottom-left of the screen, you can see some more configuration options which includes pipeline execution, stages and scan for vulnerabilities.

Stages:

There are 3 dropdowns given below:

Pre-build
Docker build
Post-build

(a) Pre-build

This section is used for those steps which you want to execute before building the Docker image. To add a Pre-build stage, click on Add Stage and provide a name to your pre-stage and write your script as per your requirement. These stages will run in sequence before the docker image is built. Optionally, you can also provide the path of the directory where the output of the script will be stored locally.

You can add one or more than one stage in a CI Pipeline.

(b) Docker build

Though we have the option available in the Docker build configuration section to add arguments in key-value pairs for the docker build image. But one can also provide docker build arguments here as well. This is useful, in case you want to override them or want to add new arguments to build your docker image.

(c) Post-build

The post-build stage is similar to the pre-build stage. The difference between the post-build stage and the pre-build stage is that the post-build will run when your CI pipeline will be executed successfully.

Adding a post-build stage is similar to adding a pre-build stage. Click on Add Stage and provide a name to your post-stage. Here you can write your script as per your requirement, which will run in sequence after the docker image is built. You can also provide the path of the directory in which the output of the script will be stored in the Remote Directory column. And this is optional to fill because many times you run scripts that do not provide any output.

NOTE:

(a) You can provide pre-build and post-build stages via the Devtron tool’s console or can also provide these details by creating a file devtron-ci.yaml inside your repository. There is a pre-defined format to write this file. And we will run these stages using this YAML file. You can also provide some stages on the Devtron tool’s console and some stages in the devtron-ci.yaml file. But stages defined through the Devtron dashboard are first executed then the stages defined in the devtron-ci.yaml file.

(b) The total timeout for the execution of the CI pipeline is by default set as 3600 seconds. This default timeout is configurable according to the use-case. The timeout can be edited in the configmap of the orchestrator service in the env variable env:"DEFAULT_TIMEOUT" envDefault:"3600"

Scan for vulnerabilities

Scan for vulnerabilities adds a security feature to your application. If you enable this option, your code will be scanned for any vulnerabilities present in your code. And you will be informed about these vulnerabilities. For more details please check doc

You have provided all the details required to create a CI pipeline, now click on Create Pipeline.

Update CI Pipeline

You can also update any configuration of an already created CI Pipeline, except the pipeline name. The pipeline name can not be edited.

Click on your CI pipeline, to update your CI Pipeline. A window will be popped up with all the details of the current pipeline.

Make your changes and click on Update Pipeline at the bottom to update your Pipeline.

Delete CI Pipeline

You can only delete CI pipeline if you have no CD pipeline created in your workflow.

To delete a CI pipeline, go to the App Configurations and then click on Workflow editor

Click on Delete Pipeline at the bottom to delete the CD pipeline

Automated Test suite integration in the CI step using devtron-ci.yaml

Users can run the test case using the Devtron dashboard or by including the test cases in the devtron.ci.yaml file in the source git repository. For reference, check: https://github.com/kumarnishant/getting-started-nodejs/blob/master/devtron-ci.yaml

The test cases given in the script will run before the test cases given in the devtron.ci.yaml

B. Linked CI Pipeline

To create a Linked CI Pipeline, please follow the steps mentioned below :

Click on + New Build Pipeline button.

Select Linked CI Pipeline.

Select the application in which the source CI pipeline is present.
Select the source CI pipeline.
Provide a name for linked CI pipeline.
Click on Create Linked CI Pipeline button to create linked CI pipeline.

After creating a linked CI pipeline, you can create a CD pipeline. You cannot trigger build from linked CI pipeline, it can be triggered only from source CI pipeline. Initially you will not see any images to deploy in CD pipeline created from linked CI pipeline. Trigger build in source CI pipeline to see the images in CD pipeline of linked CI pipeline. After this, whenever you trigger buld in source CI pipeline, the build images will be listed in CD pipeline of linked CI pipeline too.

C. Incoming Webhook

You can use Devtron for deployments on Kubernetes while using your own CI tool such as Jenkins. External CI features can be used for cases where the CI tool is hosted outside the Devtron platform.

You can send the ‘Payload script’ to your CI tools such as Jenkins and Devtron will receive the build image every time the CI Service is triggered or you can use the Webhook URL which will build an image every time CI Service is triggered using Devtron Dashboard.

Troubleshooting

We always try to make your experience of using Devtron as smooth as possible but still if you face any issues, follow the troubleshooting guide given below or join our if you couldn't find the solution for the issue you are facing.

1. How to resolve unauthorized errors, while trying to save global configurations like hostname, GitOps etc. after successful devtron installation

This occurs most of the time because any one or multiple jobs get failed during installation. To resolve this, you'll need to first check which jobs have failed. Follow these steps:

Run the following command and check which are the jobs with 0/1 completions:

Note down or remember the names of jobs with 0/1 completions and check if their pods are in running state still or not by running the command: kubectl get pods -n devtroncd
If they are in running condition, please wait for the jobs to be completed as it may be due to internet issue and if not in running condition, then delete those incomplete jobs using: kubectl delete jobs -n devtroncd
Now download migrator.yaml file from our github repository using the command:

Now edit the file you downloaded in step 3 and remove the postgresql-migrator secret resource creation and then apply the yaml file using the command: kubectl apply -f migrator.yaml -n devtroncd
It will re-create the failed jobs and you’ll see their pods created again. Just wait for a few minutes until the jobs gets completed then you are good to go. You should be able to save your global configurations now.

2. Not able to see deployment metrics on production environment or Not able to enable application-metrics or Not able to deploy the app after creating a configmap or secret with data-volume option enabled

Update the rollout crds to latest version, run the following command:

error: user/UserAuthHandler.go:236","msg":"service err, AuthVerification","err":"no token provided Or error: Failed to query provider "api/dex": Get "api/dex/.well-known/openid-configuration": unsupported protocol scheme

Delete devtron pod once to reload the configurations using:

4. Logs are not Visible on UI while running the build and not even able to abort the same

Check if the pods are being created when you start a new build, run the command and look if a new pod is created when you started the build:

If yes, delete kubewatch and devtron pod so that kubewatch can restart and start sharing the logs again:

Wait for 5 minutes and then trigger a new build again, if still not resolved then run the following commands one by one

Again wait for 5 minutes and your issue should be resolved

5. Grafana dashboards not visible in App Details page even after adding prometheus endpoint or Graphs showing error panel with id 2 not found

If the graphs are not visible check if prometheus is configured properly. Then go to Global Configurations > Clusters & Environments > Click on any environment for the cluster where you added prometheus endpoint and simply click Update. If the charts are still not visible, try visiting the url: /grafana?orgId=2 If you see Not Found on this page, then follow all the given steps or if the page is accessible and you are getting panel with id 2 not found then follow from step 6:

Get grafana password using kubectl -n devtroncd get secret devtron-secret -o jsonpath='{.data.GRAFANA_PASSWORD}' | base64 -d
kubectl run --rm -it --image quay.io/devtron/k8s-utils:tutum-curl curl Run this command and it will create a pod for using curl
Copy the following and change grafana-password with your password of grafana and change the value of prometheusUrl with your prometheus endpoint

and run in the pod that we created above in step 2. 4. Now visit /grafana?orgId=2 again and you'll see grafana login page. Login using username admin and password from step 1 and check if prometheus url is updated in datasources. If not, update it in the default datasource. 5. Now from devtron UI, update any of the environment again and it's datasource will be created automatically. 6. In Grafana UI you need to be logged in and Go to Dashboards > Manage then click Import and Import the given dashboards one by one.

After that, your issue should be resolved and you should be able to see all the graphs on UI.

If you are not able to login into Devtron dashboard even after giving the correct password, it is possible that the argocd token of previous session has been stored in the cookies and is not able to override the new token that is generated for the new session. If you are facing this issue, follow the steps below -

If using Firefox -

Goto login page of Devtron and open inspect.
Navigate to storage tab in inspect.
Click on url where Devtron has been installed under Cookies tab and you could see an argocd token with its value, something similar to below image.

Now right click on token, and click on Delete All Session Cookies option.

If using Chrome -

Goto login page of Devtron and open inspect.
Navigate to Application tab, and under Storage tab click on Cookies.
Click on url under Cookie and you would be able tto see an argocd token with its value, as shown in the image below.

Now right click on token and click on delete option.

If using Safari -

Goto Safari preferences >> Advanced options and check the show develop menu as shown in the image below.

Now goto login page of Devtron and press option+command+I. It will open inspect element.
Then navigate to Storage, click on Cookies and you would be able to see an argocd token with its value as shown in the image below.

Now right click on token and select delete option.

After clearing Cookies, try again to login, you should be able to login now.

7. No charts found in Charts Discover Section

In the Devtron's Discover Chart section, if you are not able to see any charts available, goto Global Configuration >> Chart Repositories and click on Refresh Chart at the top-right as shown in the image below. After clicking the button, it might take 4-5mins to show all the charts in Discover section depending upon the chart repositories added.

8. Not able to update cluster

In Global Configurations >> Cluters & Environments, if you try to update a cluster which has been already added in Devtron, you might get an error as {"message":"Failed to update datasource. Reload new version and try again"}. If you are facing such issue, please follow the following steps -

Edit the changes you want to make in respective cluster
Click on save after making changes and you may get error message stated above.
Go to cluster where devtron has been installed and execute - kubectl -ndevtroncd delete po -l app=devtron
Now refresh the page and you should be able to save it.

[Note: If you already have created some environments in that cluster, it needs to be updated again]

9. Postgresql is in crashloop with error - Failed to pull image

Then delete postgresql pod so that it can fetch the updated images:

You can also delete other pods which are in crashloop after postgresql is up and running so that they can restart and connect to postgresql and Devtron will be up and running again in a few moments.

10. Unable to fetch the latest commit and not able to trigger auto build.

To solve this, bounce the git-sensor-0 pod.

Whitelist the NAT-gateway IPs of the cluster (There can be multiple NAT-gateways if your cluster is multi-AZ)

12. If CPU metrics are not showing but memory metrics are visible in graphs.

Do the following:-

Go to Grafana and Login with the credentials.
Edit the CPU graphs and remove image!=”” from the query.
Save the dashboard.

CPU metrics should start showing up in a while.

13. If user not able to upload a file more than specific size.

Please use below annotation in ingress

Note:- Where m is MiB.

14. If AWS Load balancer controller is unable to provision ALB and getting message in alb controller as unauthorized, attach these IAM policy to the nodegroup IAM Role.

15. When app metrics is not coming on grafana and devtron dashboard, set the value of the following parameter as false in kube prometheus stack values.

16. Unable to deploy metrics-server using chart on devtron

To solve

Disable certificate validation by passing --kubelet-insecure-tls argument to metrics server chart.

17. Unable to delete a database from postgres

Description of issue

ERROR: database <db-name> is being accessed by other users

DETAIL: There is 1 other session using the database.

You have to terminate the connections to the database first, for that you can use the command.

Then run the command to delete database - drop databases <db-name>

Debug

Run the command for admin credentials and use it for login in dashboard:

If you are getting an error message of “invalid username or password”, follow the solution to solve it.

Solution:

Run kubectl get secret -n devtroncd and then edit the argocd-secret, remove both the admin.password lines.

Run kubectl delete po your-argocd-server-pod -n devtroncd, it will create a new pod after deletion and reset your admin password. Re-run the command for admin credentials again to get the new password.

19. After installing Devtron using helm, getting the admin password does not work.(if using windows)

Debug

'base64' is not recognized as an internal or external command, operable program or batch file.

Solution

The first way to debug is either install base64 encode and decode into your windows machine and use the appropriate cmd to get the admin password.

The other way is to get the password in the encoded form using the cmd

20. Getting `UPGRADE FAILED: cannot patch "postgresql-postgresql"` while upgrading Devtron to newer versions

Debug:

Description of error

Solution: Verify if annotations & lables are set to all k8s resources in devtroncd namespace and add --set components.postgres.persistence.volumeSize=20Gi parameter in Devtron upgrade command.