1. Job

A Job creates one or more Pods and will continue to retry execution of the Pods until a specified number of them successfully terminate. As pods successfully complete, the Job tracks the successful completions. When a specified number of successful completions is reached, the task (ie, Job) is complete. Deleting a Job will clean up the Pods it created. Suspeding a Job will delete its active Pods until the Job is resumed again.

Example:

kind: Job
jobConfigs:
    activeDeadlineSeconds: 120
    backoffLimit: 6
    completions: 1
    parallelism: 1
    suspend: false
    ttlSecondsAfterFinished: 100

2. Cronjob

A Cronjob creates Jobs on a repeating schedule , One Cronjob object is like one line of a crontab (cron table) file. It runs a job periodically on a given schedule, written in Cron format. Cronjobs are meant for performing regular scheduled actions such as backups, report generation, and so on. Each of those tasks should be configured to recur indefinitely (for example: once a day / week / month); you can define the point in time within that interval when the job should start.

Example:

kind: CronJob
cronjobConfigs:
    concurrencyPolicy: Allow
    failedJobsHistoryLimit: 1
    restartPolicy: OnFailure
    schedule: 32 8 * * *
    startingDeadlineSeconds: 100
    successfulJobsHistoryLimit: 3
    suspend: false

Git Repository is used to pull your application source code during the CI step. Select Git Repository section of the App Configuration. Inside Git Repository when you click on Add Git Repository you will see three options as shown below:

1. Git Account

2. Git Repo URL

3. Checkout Path

Devtron also supports multiple git repositories in a single deployment. We will discuss this in detail in the multi git option below.

1. Git Account

In this section, you have to select the git account of your code repository. If the authentication type of the Git account is anonymous, only public git repository will be accessible. If you are using a private git repository, you can configure your git provider via git accounts.

2. Git Repo URL

Inside the git repo URL, you have to provide your code repository’s URL. For Example- https://github.com/devtron-labs/django-repo

You can find this URL by clicking on the '⤓ code' button on your git repository page.

Note:

• Copy the HTTPS/SSH url of the repository

• Please make sure that you've added your dockerfile in the repo.

3. Checkout Path

After clicking on checkbox, git checkout path field appears. The git checkout path is the directory where your code is pulled or cloned for the repository you specified in the previous step.

This field is optional in case of a single git repository application and you can leave the path as default. Devtron assigns a directory by itself when the field is left blank. The default value of this field is ./

If you want to go with a multi-git approach, then you need to specify a separate path for each of your repositories. The first repository can be checked out at the default ./ path as explained above. But, for all the rest of the repositories, you need to ensure that you provide unique checkout paths. In failing to do so, you may cause Devtron to checkout multiple repositories in one directory and overwriting files from different repositories on each other.

4. Pull Modules Recursively:

This checkbox is optional and is used for pulling git submodules present in a repo. The submodules will be pulled recursively and same auth method which is used for parent repo will be used for submodules.

5. Multi Git:

As we discussed, Devtron also supports multiple git repositories in a single application. To add multiple repositories, click on add repo and repeat steps 1 to 3. Repeat the process for every new git repository you add. Ensure that the checkout paths are unique for each.

Note: Even if you add multiple repositories, only one image will be created based on the docker file as shown in the docker build config.

Why do we need Multi Git support-

Let’s look at this with an example:

Due to security reasons, you may want to keep sensitive configurations like third party API keys in a separate access restricted git repositories and the source code in a git repository that every developer has access to. To deploy this application, code from both the repositories is required. A multi-git support will help you to do that.

Few other examples, where you may want to have multiple repositories for your application and will need multi git checkout support:

• To make code modularize, you are keeping front-end and back-end code in different repositories.

• Common Library extracted out in different repo so that it can be used via multiple other projects.

• Due to security reasons you are keeping configuration files in different access restricted git repositories.

How Devtron's 'Checkout Path' Works

The checkout path is used by Devtron to assign a directory to each of your git repositories. Once you provide different checkout paths for your repositories, Devtron will clone your code at those locations and these checkout paths can be referenced in the docker file to create docker image for the application. Whenever a change is pushed to any the configured repositories, the CI will be triggered and a new docker image file will be built based on the latest commits of the configured repositories and pushed to the container registry.

Workflow is a logical sequence of different stages used for continuous integration and continuous deployment of an application.

Click on New Build Pipeline to create a new workflow

On clicking New Build Pipeline, three options appear as mentioned below:

• Continuous Integration: Choose this option if you want Devtron to build the image of source code.

• Linked CI Pipeline: Choose this option if you want to use an image created by an existing CI pipeline in Devtron.

• Incoming Webhook: Choose this if you want to build your image outside Devtron, it will receive a docker image from an external source via the incoming webhook.

Then, create CI/CD Pipelines for your application.

Info:

For Devtron version older than v0.4.0, please refer the page.

A CI Pipeline can be created in one of the three ways:

Each of these methods has different use-cases that can be tailored to the needs of the organization.

1. Continuous Integration

Continuous Integration Pipeline allows you to build the container image from a source code repository.

1. From the Applications menu, select your application.

2. On the App Configuration page, select Workflow Editor.

3. Select + New Build Pipeline.

4. Select Continuous Integration.

5. Enter the following fields on the Create build pipeline screen:

Advanced Options

The advanced CI Pipeline includes the following stages:

• Pre-build stage: The tasks in this stage run before the image is built.

• Build stage: In this stage, the build is triggered from the source code that you provide.

• Post-build stage: The tasks in this stage are triggered once the build is complete.

Scan for vulnerabilities

To Perform the security scan after the container image is built, enable the Scan for vulnerabilities toggle in the build stage.

Build stage

The Build stage allows you to configure a build pipeline from the source code.

1. From the Create build pipeline screen, select Advanced Options.

2. Select Build stage.

Select Update Pipeline.

Source type: Branch Fixed

The Source type - "Branch Fixed" allows you to trigger a CI build whenever there is a code change on the specified branch.

Select the Source type as "Branch Fixed" and enter the Branch Name.

Configuring Webhook

Info: If you choose "Pull Request" or "Tag Creation" as the source type, you must first configure the Webhook for GitHub/Bitbucket as a prerequisite step.

1. Configure Webhook for GitHub

1. Go to the Settings page of your repository and select Webhooks.

2. Select Add webhook.

3. In the Payload URL field, enter the Webhook URL that you get on selecting the source type as "Pull Request" or "Tag Creation" in Devtron the dashboard.

4. Change the Content-type to application/json.

5. In the Secret field, enter the secret from Devtron the dashboard when you select the source type as "Pull Request" or "Tag Creation".

1. Under Which events would you like to trigger this webhook?, select Let me select individual events. to trigger the webhook to build CI Pipeline.

2. Select Branch or tag creation and Pull Requests.

3. Select Add webhook.

2. Configure Webhook for Bitbucket cloud

1. Go to the Repository settings page of your Bitbucket repository.

2. Select Webhooks and then select Add webhook.

1. Enter a Title for the webhook.

2. In the URL field, enter the Webhook URL that you get on selecting the source type as "Pull Request" or "Tag Creation" in the Devtron dashboard.

3. Select the event triggers for which you want to trigger the webhook.

4. Select Save to save your configurations.

Source type: Pull Request

The Source type - "Pull Request" allows you to configure the CI Pipeline using the PR raised in your repository.

To trigger the build from specific PRs, you can filter the PRs based on the following keys:

Select the appropriate filter and pass the matching condition as a regular expression (regex).

Select Create Pipeline.

Source type: Tag Creation

The Source type - "Tag Creation" allows you to build the CI pipeline from a tag.

To trigger the build from specific tags, you can filter the tags based on the author and/or the tag name.

Select the appropriate filter and pass the matching condition as a regular expression (regex).

Select Create Pipeline.

Note

(a) You can provide pre-build and post-build stages via the Devtron tool’s console or can also provide these details by creating a file devtron-ci.yaml inside your repository. There is a pre-defined format to write this file. And we will run these stages using this YAML file. You can also provide some stages on the Devtron tool’s console and some stages in the devtron-ci.yaml file. But stages defined through the Devtron dashboard are first executed then the stages defined in the devtron-ci.yaml file.

(b) The total timeout for the execution of the CI pipeline is by default set as 3600 seconds. This default timeout is configurable according to the use case. The timeout can be edited in the configmap of the orchestrator service in the env variable as env:"DEFAULT_TIMEOUT" envDefault:"3600"

2. Linked CI Pipeline

If one code is shared across multiple applications, Linked CI Pipeline can be used, and only one image will be built for multiple applications because if there is only one build, it is not advisable to create multiple CI Pipelines.

1. From the Applications menu, select your application.

2. On the App Configuration page, select Workflow Editor.

3. Select + New Build Pipeline.

4. Select Linked CI Pipeline.

5. Enter the following fields on the Create linked build pipeline screen:

• Select the application in which the source CI pipeline is present.

• Select the source CI pipeline from the application that you selected above.

• Enter a name for the linked CI pipeline.

Select Create Linked CI Pipeline.

After creating a linked CI pipeline, you can create a CD pipeline. Builds cannot be triggered from a linked CI pipeline; they can only be triggered from the source CI pipeline. There will be no images to deploy in the CD pipeline created from the 'linked CI pipeline' at first. To see the images in the CD pipeline of the linked CI pipeline, trigger build in the source CI pipeline. The build images will now be listed in the CD pipeline of the 'linked CI pipeline' whenever you trigger a build in the source CI pipeline.

3. Incoming Webhook

The CI pipeline receives container images from an external source via a webhook service.

You can use Devtron for deployments on Kubernetes while using your CI tool such as Jenkins. External CI features can be used when the CI tool is hosted outside the Devtron platform.

1. From the Applications menu, select your application.

2. On the App Configuration page, select Workflow Editor.

3. Select + New Build Pipeline.

4. Select Incoming Webhook.

• Select Save and Generate URL. This generates the Payload format and Webhook URL.

You can send the Payload script to your CI tools such as Jenkins and Devtron will receive the build image every time the CI Service is triggered or you can use the Webhook URL which will build an image every time CI Service is triggered using Devtron Dashboard.

Update CI Pipeline

You can update the configurations of an existing CI Pipeline except for the pipeline's name. To update a pipeline, select your CI pipeline. In the Edit build pipeline window, edit the required stages and select Update Pipeline.

Delete CI Pipeline

You can only delete a CI pipeline if there is no CD pipeline created in your workflow.

To delete a CI pipeline, go to App Configurations > Workflow Editor and select Delete Pipeline.

Welcome! This is the documentation for Creating Applications

Parts of Documentation

A deployment configuration is a manifest for the application. It defines the runtime behavior of the application.

Devtron includes deployment template for both default as well as custom charts created by a super admin.

To configure a deployment chart for your application:

• Go to Applications and create a new application.

• Go to App Configuration page and configure your application.

• On the Deployment Template page, select the drop-down under Chart type.

You can select a chart in one of the following ways:

Select chart by Devtron

3. Knative

Select chart from custom charts

Custom charts are added by a super admin from the section.

Users can select the available custom charts from the drop-down list.

Upload custom chart

Application Metrics

Enable show application metrics toggle to view the application metrics on the App Details page.

IMPORTANT: Enabling Application metrics introduces a sidecar container to your main container which may require some additional configuration adjustments, we recommend you to do load test after enabling it in a non-prod environment before enabling it in production environment.

Select Save to save your configurations.

In the previous step, we discussed Git Configurations. In this section, we will provide information on the Docker Build Configuration.

Docker build configuration is used to create and push docker images in the docker registry of your application. You will provide all the docker related information to build and push docker images in this step.

Only one docker image can be created even for multi-git repository applications as explained in the .

To add docker build configuration, You need to provide three sections as given below:

• Image store

• Checkout path

• Advanced

Image Store

In Image store section, You need to provide two inputs as given below:

1. Docker registry

2. Docker repository

1. Docker Registry

2. Docker Repository

In this field, add the name of your docker repository. The repository that you specify here will store a collection of related docker images. Whenever an image is added here, it will be stored with a new tag version.

If you are using docker hub account, you need to enter the repository name along with your username. For example - If my username is kartik579 and repo name is devtron-trial, then enter kartik579/devtron-trial instead of only devtron-trial.

Checkout path

Checkout path including inputs:

1. Git checkout path

2. Docker file (relative)

1. Git checkout path

In this field, you have to provide the Git checkout path of your repository. This repository is the same that you had defined earlier in git configuration details.

2. Docker File Path

Here, you provide a relative path where your docker file is located. Ensure that the dockerfile is present on this path.

Advanced

Set Target Platform for the build

Before selecting a custom target platform, please ensure that the architecture and the operating system is supported by the registry type you are using, otherwise builds will fail. Devtron uses BuildX to build images for mutiple target Platforms, which requires higher CI worker resources. To allocate more resources, you can increase value of the following parameters in the devtron-cm configmap in devtroncd namespace.

• LIMIT_CI_CPU

• REQ_CI_CPU

• REQ_CI_MEM

• LIMIT_CI_MEM

To edit the devtron-cm configmap in devtroncd namespace:

If target platform is not set, Devtron will build image for architecture and operating system of the k8s node on which CI is running.

The Target Platform feature might not work in minikube & microk8s clusters as of now.

Docker build arguments is a collapsed view including

• Key

• Value

Key-value

Deployment configuration is the Manifest for the application, it defines the runtime behavior of the application. You can define application behavior by providing information in three sections:

1. Chart Version

2. Yaml file

3. Show application metrics

1. Chart version

Devtron uses helm charts for the deployments. And we are having multiple chart versions based on features it is supporting with every chart version.

One can see multiple chart version options available in the drop-down. you can select any chart version as per your requirements. By default, the latest version of the helm chart is selected in the chart version option.

Every chart version has its own YAML file. Helm charts are used to provide specifications for your application. To make it easy to use, we have created templates for the YAML file and have added some variables inside the YAML. You can provide or change the values of these variables as per your requirement.

Application Metrics is not supported for Chart version older than 3.7 version.

2. Yaml file

Container Ports

This defines ports on which application services will be exposed to other services

EnvVariables

EnvVariables provide run-time information to containers and allow to customize how the application works and the behavior of the applications on the system.

Here we can pass the list of env variables , every record is an object which contain the name of variable along with value.

To set environment variables for the containers that run in the Pod.

Example of env variables

IMP Docker image should have env variables, whatever we want to set.

But ConfigMap and Secret are the prefered way to inject env variables. So we can create this in App Configuration Section

ConfigMap

It is a centralized storage, specific to k8s namespace where key-value pairs are stored in plain text.

Secret

It is a centralized storage, specific to k8s namespace where we can store the key-value pairs in plain text as well as in encrypted(Base64) form.

IMP All key-values of Secret and CofigMap will reflect to your application.

Liveness Probe

If this check fails, kubernetes restarts the pod. This should return error code in case of non-recoverable error.

MaxUnavailable

The maximum number of pods that can be unavailable during the update process. The value of "MaxUnavailable: " can be an absolute number or percentage of the replicas count. The default value of "MaxUnavailable: " is 25%.

MaxSurge

The maximum number of pods that can be created over the desired number of pods. For "MaxSurge: " also, the value can be an absolute number or percentage of the replicas count. The default value of "MaxSurge: " is 25%.

Min Ready Seconds

This specifies the minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing, for it to be considered available. This defaults to 0 (the Pod will be considered available as soon as it is ready).

Readiness Probe

If this check fails, kubernetes stops sending traffic to the application. This should return error code in case of errors which can be recovered from if traffic is stopped.

Autoscaling

This is connected to HPA and controls scaling up and down in response to request load.

Fullname Override

fullnameOverride replaces the release fullname created by default by devtron, which is used to construct Kubernetes object names. By default, devtron uses {app-name}-{environment-name} as release fullname.

Image

Image is used to access images in kubernetes, pullpolicy is used to define the instances calling the image, here the image is pulled when the image is not present,it can also be set as "Always".

imagePullSecrets

imagePullSecrets contains the docker credentials that are used for accessing a registry.

Ingress

This allows public access to the url, please ensure you are using right nginx annotation for nginx class, its default value is nginx

Legacy deployment-template ingress format

Ingress Internal

This allows private access to the url, please ensure you are using right nginx annotation for nginx class, its default value is nginx

Init Containers

Specialized containers that run before app containers in a Pod. Init containers can contain utilities or setup scripts not present in an app image. One can use base image inside initContainer by setting the reuseContainerImage flag to true.

Pause For Seconds Before Switch Active

To wait for given period of time before switch active the container.

Resources

These define minimum and maximum RAM and CPU available to the application.

Resources are required to set CPU and memory usage.

Limits

Limits make sure a container never goes above a certain value. The container is only allowed to go up to the limit, and then it is restricted.

Requests

Requests are what the container is guaranteed to get.

Service

This defines annotations and the type of service, optionally can define name also.

Note - If loadBalancerSourceRanges is not set, Kubernetes allows traffic from 0.0.0.0/0 to the LoadBalancer / Node Security Group(s).

Volumes

It is required when some values need to be read from or written to an external disk.

Volume Mounts

It is used to provide mounts to the volume.

Affinity and anti-affinity

Spec is used to define the desire state of the given container.

Node Affinity allows you to constrain which nodes your pod is eligible to schedule on, based on labels of the node.

Inter-pod affinity allow you to constrain which nodes your pod is eligible to be scheduled based on labels on pods.

Key

Key part of the label for node selection, this should be same as that on node. Please confirm with devops team.

Values

Value part of the label for node selection, this should be same as that on node. Please confirm with devops team.

Tolerations

Taints are the opposite, they allow a node to repel a set of pods.

A given pod can access the given node and avoid the given taint only if the given pod satisfies a given taint.

Taints and tolerations are a mechanism which work together that allows you to ensure that pods are not placed on inappropriate nodes. Taints are added to nodes, while tolerations are defined in the pod specification. When you taint a node, it will repel all the pods except those that have a toleration for that taint. A node can have one or many taints associated with it.

Arguments

This is used to give arguments to command.

Command

It contains the commands to run inside the container.

Containers

Containers section can be used to run side-car containers along with your main container within same pod. Containers running within same pod can share volumes and IP Address and can address each other @localhost.

Prometheus

It is a kubernetes monitoring tool and the name of the file to be monitored as monitoring in the given case.It describes the state of the prometheus.

rawYaml

Accepts an array of Kubernetes objects. You can specify any kubernetes yaml here and it will be applied when your app gets deployed.

Grace Period

Kubernetes waits for the specified time called the termination grace period before terminating the pods. By default, this is 30 seconds. If your pod usually takes longer than 30 seconds to shut down gracefully, make sure you increase the GracePeriod.

A Graceful termination in practice means that your application needs to handle the SIGTERM message and begin shutting down when it receives it. This means saving all data that needs to be saved, closing down network connections, finishing any work that is left, and other similar tasks.

There are many reasons why Kubernetes might terminate a perfectly healthy container. If you update your deployment with a rolling update, Kubernetes slowly terminates old pods while spinning up new ones. If you drain a node, Kubernetes terminates all pods on that node. If a node runs out of resources, Kubernetes terminates pods to free those resources. It’s important that your application handle termination gracefully so that there is minimal impact on the end user and the time-to-recovery is as fast as possible.

Server

It is used for providing server configurations.

Deployment

It gives the details for deployment.

Service Monitor

It gives the set of targets to be monitored.

Db Migration Config

It is used to configure database migration.

Application Metrics

Application metrics can be enabled to see your application's metrics-CPUService Monito usage,Memory Usage,Status,Throughput and Latency.

Deployment Metrics

It gives the realtime metrics of the deployed applications

Addon features in Deployment Template Chart version 3.9.0

Service Account

A service account provides an identity for the processes that run in a Pod.

When you access the cluster, you are authenticated by the API server as a particular User Account. Processes in containers inside pod can also contact the API server. When you are authenticated as a particular Service Account.

When you create a pod, if you do not create a service account, it is automatically assigned the default service account in the namespace.

Pod Disruption Budget

You can create PodDisruptionBudget for each application. A PDB limits the number of pods of a replicated application that are down simultaneously from voluntary disruptions. For example, an application would like to ensure the number of replicas running is never brought below the certain number.

You can specify maxUnavailable and minAvailable in a PodDisruptionBudget.

With minAvailable of 1, evictions are allowed as long as they leave behind 1 or more healthy pods of the total number of desired replicas.

With maxAvailable of 1, evictions are allowed as long as at most 1 unhealthy replica among the total number of desired replicas.

Application metrics Envoy Configurations

Envoy is attached as a sidecar to the application container to collect metrics like 4XX, 5XX, Throughput and latency. You can now configure the envoy settings such as idleTimeout, resources etc.

Prometheus Rule

Alerting rules allow you to define alert conditions based on Prometheus expressions and to send notifications about firing alerts to an external service.

In this case, Prometheus will check that the alert continues to be active during each evaluation for 1 minute before firing the alert. Elements that are active, but not firing yet, are in the pending state.

Pod Labels

Labels are key/value pairs that are attached to pods. Labels are intended to be used to specify identifying attributes of objects that are meaningful and relevant to users, but do not directly imply semantics to the core system. Labels can be used to organize and to select subsets of objects.

Pod Annotations

Pod Annotations are widely used to attach metadata and configs in Kubernetes.

Custom Metrics in HPA

HPA, by default is configured to work with CPU and Memory metrics. These metrics are useful for internal cluster sizing, but you might want to configure wider set of metrics like service latency, I/O load etc. The custom metrics in HPA can help you to achieve this.

Wait For Seconds Before Scaling Down

Wait for given period of time before scaling down the container.

3. Show Application Metrics

If you want to see application metrics like different HTTP status codes metrics, application throughput, latency, response time. Enable the Application metrics from below the deployment template Save button. After enabling it, you should be able to see all metrics on App detail page. By default it remains disabled.

Helm Chart Json Schema Table

Helm Chart json schema is used to validate the deployment template values.

Other Validations in Json Schema

The values of CPU and Memory in limits must be greater than or equal to in requests respectively. Similarly, In case of envoyproxy, the values of limits are greater than or equal to requests as mentioned below.

Addon features in Deployment Template Chart version 4.11.0

KEDA Autoscaling

Example for autosccaling with KEDA using Prometheus metrics is given below:

Example for autosccaling with KEDA based on kafka is given below :

Security Context

A security context defines privilege and access control settings for a Pod or Container.

To add a security context for main container:

To add a security context on pod level:

Topology Spread Constraints

You can use topology spread constraints to control how Pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains. This can help to achieve high availability as well as efficient resource utilization.

The CI pipeline includes Pre and Post-build steps to validate and introduce checkpoints in the build process.

The pre/post plugins allow you to execute some standard tasks, such as Code analysis, Load testing, Security scanning, and so on. You can build custom pre/post tasks or use one from the standard preset plugins provided by Devtron.

Before you begin

Create a if you haven't done that already!

Configuring Pre/Post-build stages

Each Pre/Post-build stage is executed as a series of events called tasks and includes custom scripts. You could create one or more tasks that are dependent on one another for execution. In other words, the output variable of one task can be used as an input for the next task to build a CI runner.

The tasks will run following the execution order.

The tasks may be re-arranged by using drag-and-drop; however, the order of passing the variables must be followed.

Creating a task

1. Go to Applications and select your application from the Devtron Apps tabs.

2. From the App Configuration tab select Workflow Editor.

3. Select the build pipeline for editing the stages.

Devtron CI pipeline includes the following build stages:

• Pre-build stage: The tasks in this stage run before the image is built.

• Build stage: In this stage, the build is triggered from the source code that you provide.

• Post-build stage: The tasks in this stage are triggered once the build is complete.

You can create a task either by selecting one of the available preset plugins or by creating a custom script.

Preset plugins

Prerequisite: Set up Sonarqube, or get the API keys from an admin.

The example shows a Post-build stage with a task created using a preset plugin - Sonarqube.

1. On the Edit build pipeline screen, select the Post-build stage (or Pre-build).

2. Select + Add task.

3. Select Sonarqube from PRESET PLUGINS.

Input Variables

Select Update Pipeline.

Execute custom script

1. On the Edit build pipeline screen, select the Pre-build stage.

2. Select + Add task.

3. Select Execute custom script.

Custom script - Shell

• Select the Task type as Shell.

Consider an example that creates a Shell task to stop the build if the database name is not "mysql". The script takes 2 input variables, one is a global variable (DOCKER_IAMGE), and the other is a custom variable (DB_NAME) with a value "mysql". The task triggers only if the database name matches "mysql". If the trigger condition fails, this Pre-build task will be skipped and the build process will start. The variable DB_NAME is declared as an output variable that will be available as an input variable for the next task. The task fails if DB_NAME is not equal to "mysql".

• Select Update Pipeline.

Here is a screenshot with the failure message from the task:

Custom script - Container image

• Select the Task type as Container image.

This example creates a Pre-build task from a container image. The output variable from the previous task is available as an input variable.

• Select Update Pipeline.

What's next

The ConfigMap API resource holds key-value pairs of the configuration data that can be consumed by pods or used to store configuration data for system components such as controllers. ConfigMap is similar to Secrets, but designed to more conveniently support working with strings that do not contain sensitive information.

Click on Add ConfigMap to add a config map to your application.

Configure the ConfigMap

You can configure a configmap in two ways-

(a) Using data type Kubernetes ConfigMap

(b) Using data type Kubernetes External ConfigMap

(A) Using Kubernetes ConfigMap

1. Data Type

Select the Data Type as Kubernetes ConfigMap, if you wish to use the ConfigMap created by Devtron.

2. Name

Provide a name to your configmap.

3. Use ConfigMap as

Here we are providing two options, one can select any of them as per your requirement

-Environment Variable as part of your configMap or you want to add Data Volume to your container using Config Map.

• Environment Variable

Select this option if you want to add Environment Variables as a part of configMap. You can provide Environment Variables in key-value pairs, which can be seen and accessed inside a pod.

• Data Volume

Select this option if you want to add a Data Volume to your container using the Config Map.

Key-value pairs that you provide here, are provided as a file to the mount path. Your application will read this file and collect the required data as configured.

4. Data

In the Data section, you provide your configmap in key-value pairs. You can provide one or more than one environment variable.

You can provide variables in two ways-

• YAML (raw data)

• GUI (more user friendly)

Once you have provided the config, You can click on any option-YAML or GUI to view the key and Value parameters of the ConfigMap.

Kubernetes ConfigMap using Environment Variable:

If you select Environment Variable in 3rd option, then you can provide your environment variables in key-value pairs in the Data section using YAML or GUI.

Data in YAML (please Check below screenshot)

Now, Click on Save ConfigMap to save your configmap configuration.

Kubernetes ConfigMap using Data Volume

Volume Mount Path

Provide the Volume Mount folder path in Volume Mount Path, a path where the data volume needs to be mounted, which will be accessible to the Containers running in a pod.

You can add Configuration data as in YAML or GUI format as explained above.

You can click on YAML or GUI to view the key and Value parameters of the ConfigMap that you have created.

You can click on Save ConfigMap to save the configMap.

Sub Path

For multiple files mount at the same location you need to check sub path bool field, it will use the file name (key) as sub path. Sub Path feature is not applicable in case of external configmap.

File Permission

File permission will be provide at the configmap level not on the each key of the configmap. It will take 3 digit standard permission for the file.

(B) Kubernetes External ConfigMap

You can select Kubernetes External ConfigMap in the data type field if you have created a ConfigMap using the kubectl command.

By default, the data type is set to Kubernetes ConfigMap.

Kubernetes External ConfigMap is created using the kubectl create configmap command. If you are using Kubernetes External ConfigMap, make sure you give the name of ConfigMap the same as the name that you have given using kubectl create Configmap <configmap-name> <data source> command, otherwise, it might result in an error during the built.

You have to ensure that the External ConfigMap exists and is available to the pod.

The config map is created.

Update ConfigMap

You can update your configmap anytime later but you cannot change the name of your configmap. If you want to change the name of the configmap then you have to create a new configmap. To update configmap, click on the configmap you have created make changes as required.

Click on Update Configmap to update your configmap.

Delete ConfigMap

You can delete your configmap. Click on your configmap and click on the delete sign to delete your configmap.

Secrets and configmaps both are used to store environment variables but there is one major difference between them: Configmap stores key-values in normal text format while secrets store them in base64 encrypted form. Devtron hides the data of secrets for the normal users and it is only visible to the users having edit permission.

Secret objects let you store and manage sensitive information, such as passwords, authentication tokens, and ssh keys. Embedding this information in secrets is safer and more flexible than putting it verbatim in a Pod definition or in a container image.

Configure Secret

Click on Add Secret to add a new secret.

Volume Mount Path

Specify the volume mount folder path in Volume Mount Path, a path where the data volume needs to be mounted. This volume will be accessible to the containers running in a pod.

Sub Path

For multiple files mount at the same location you need to check sub path bool field, it will use the file name (key) as sub path. Sub Path feature is not applicable in case of external configmap except AWS Secret Manager, AWS System Manager and Hashi Corp Vault, for these cases Name (Secret key) as sub path will be picked up automatically.

File Permission

File permission will be provide at the configmap level not on the each key of the configmap. it will take 3 digit standard permission for the file.

Click on Save Secret to save the secret.

You can see the Secret is added.

Update Secrets

You can update your secrets anytime later, but you cannot change the name of your secrets. If you want to change your name of secrets then you have to create a new secret.

To update secrets, click on the secret you wish to update.

Click on Update Secret to update your secret.

Delete Secret

You can delete your secret. Click on your secret and click on the delete sign to delete your secret.

Data Types

There are five Data types that you can use to save your secret.

• Kubernetes Secret: The secret that you create using Devtron.

• Kubernetes External Secret: The secret data of your application is fetched by Devtron externally. Then the Kubernetes External Secret is converted to Kubernetes Secret.

• AWS Secret Manager: The secret data of your application is fetched from AWS Secret Manager and then converted to Kubernetes Secret from AWS Secret.

• AWS System Manager: The secret data for your application is fetched from AWS System Secret Manager and all the secrets stored in AWS System Manager are converted to Kubernetes Secret.

• Hashi Corp Vault: The secret data for your application is fetched from Hashi Corp Vault and the secrets stored in Hashi Corp Vault are converted to Kubernetes Secret.

Note: The conversion of secrets from various data types to Kubernetes Secrets is done within Devtron and irrespective of the data type, after conversion, the Pods access secrets normally.

External Secrets

In some cases, it may be that you already have secrets for your application on some other sources and you want to use that on devtron. External secrets are fetched by devtron externally and then converted to kubernetes secrets.

Kubernetes External Secret

The secret that is already created and stored in the environment and being used by devtron externally is referred here as Kubernetes External Secret. For this option, devtron will not create any secret by itself but they can be used within the pods. Before adding secret from kubernetes external secret, please make sure that secret with the same name is present in the environment. To add secret from kubernetes external secret, follow the steps mentioned below:

1. Navigate to Secrets of the application.

2. Click on Add Secret to add a new secret.

3. Select Kubernetes External Secret from dropdown of Data type.

4. Provide a name to your secret. Devtron will search secret in the environment with the same name that you mention here.

AWS Secret Manager

Before adding any external secrets on devtron, kubernetes-external-secrets must be installed on the target cluster. Kubernetes External Secrets allows you to use external secret management systems (e.g., AWS Secrets Manager, Hashicorp Vault, etc) to securely add secrets in Kubernetes.

Installing kubernetes-external-secrets Using Chart

To install the chart with the release named my-release:

$ helm install my-release external-secrets/kubernetes-external-secrets

To install the chart with AWS IAM Roles for Service Accounts:

$ helm install my-release external-secrets/kubernetes-external-secrets --set securityContext.fsGroup=65534 --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"='arn:aws:iam::111111111111:role/ROLENAME'

Adding Secrets From AWS Secret Manager

To add secrets from AWS secret manager, navigate to Secrets of the application and follow the steps mentioned below :

1. Click on Add Secret to add a new secret.

1. Select AWS Secret Manager from dropdown of Data type.

2. Provide a name to your secret.

3. Select how you want to use the secret. You may leave it selected as environment variable and also you may leave Role ARN empty.

4. In Data section, you will have to provide data in key-value format.

All the required field to pass your data to fetch secrets on devtron are described below :

Adding Secrets in AWS Secret Manager

To add secrets in AWS secret manager, do the following steps :

1. Go to AWS secret manager console.

2. Click on Store a new secret.

3. Add and save your secret.

Application metrics can be enabled to see your application's metrics.

Standard Metrics

Devtron provides certain metrics (CPU and Memory utilization) for each application by default i.e. you do not need to enable “Application metrics”. However, prometheus needs to be present in the cluster and the endpoint of the same should be updated in Global Configurations --> Clusters & Environments section.

Advanced Metrics

There are certain advanced metrics (like Latency, Throughput, 4xx, 5xx, 2xx) which are only available when "Application metrics" is enabled from the Deployment Template. When you enable these advanced metrics, devtron attaches a envoy sidecar container to your main container which runs as a transparent proxy and passes each request through it to measure the advanced metrics.

Note: Since, all the requests are passed through envoy, any misconfiguration in envoy configs can bring your application down, so please test the configurations in a non-production environment extensively.

envoyproxy:
  image: envoyproxy/envoy:v1.14.1
  configMapName: ""
  resources:
    limits:
      cpu: "50m"
      memory: "50Mi"
    requests:
      cpu: "50m"
      memory: "50Mi"

CPU Usage Metrics

CPU usage is a utilization metric that shows the overall utilization of cpu by an application. It is available as both, aggregated or per pod.

Memory Usage Metrics

Memory usage is a utilization metric that shows the overall utilization of memory by an application. It is available as both, aggregated or per pod.

Throughput Metrics

This application metrics indicates the number of request processed by an application per minute.

Status Code Metrics

This metrics indicates the application’s response to client’s request with a specific status code i.e 1xx(Communicate transfer protocol leve information), 2xx(Client’s request was accepted successfully), 3xx(Client must take some additional action to complete their request), 4xx(Client side error) or 5xx(Server side error).

Latency Metrics

Latency metrics shows the latency for an application. Latency measures the delay between an action and a response.

99.9th percentile latency: The maximum latency, in seconds, for the fastest 99.9% of requests.

99th percentile latency: The maximum latency, in seconds, for the fastest 99% of requests.

95th percentile latency: The maximum latency, in seconds, for the fastest 95% of requests.

Note: We also support custom percentile input inside the dropdown .A latency measurement based on a single request is not meaningful.

Checklist for enabling Advanced Application metrics in Production

• Have adjusted resources to the envoy sidecar container, by default Devtron allocates 50m CPU and 50Mi Memory as both limits as well as requests. This should be enough for handling traffic upto 3000rpm per pod, if each replica of your pod is expected to handle more than 3000rpm, please adjust the resources accordingly.

• If you are not leveraging http2 / streaming protocols, make sure to set supportStreaming and useHTTP2 in ContainerPort as false.

• Use envoy image as "quay.io/devtron/envoy:v1.14.1" instead of default "envoyproxy/envoy:v1.14.1" if your cluster occasionally hit dockerhub pull rate limit or if you are running too many replicas/micro-services in a cluster.

• Enabled and tested extensively in non-production environment including load testing till highest rpm capacity per pod.

Users can run the test case using the Devtron dashboard or by including the test cases in the devtron.ci.yaml file in the source git repository. For reference, check: https://github.com/kumarnishant/getting-started-nodejs/blob/master/devtron-ci.yaml

The test cases given in the script will run before the test cases given in the devtron.ci.yaml

CI Pipeline can be created in three different ways, Continuous Integration, Linked CI Pipeline and Incoming Webhook.

Each of these methods have different use-cases which can be used according to the needs of the organization. Let’s begin with Continuous Integration.

A. Continuous Integration

Click on Continuous Integration, a prompt comes up in which we need to provide our custom configurations. Below is the description of some configurations which are required.

[Note] Options such as pipeline execution, stages and scan for vulnerabilities, will be visible after clicking on advanced options present in the bottom left corner.

I. Pipeline Name

Pipeline name is an auto-generated name which can also be renamed by clicking on Advanced options.

II. Pipeline Execution

You can select the method you want to execute the pipeline. By default the value is automatic. In this case it will get automatically triggered if any changes are made to the respective git repository. You can set it to manual if you want to trigger the pipeline manually.

III. Source Type

In source type, we can observe that we have three types of mechanisms which can be used for building your CI Pipeline. In the drop-down you can observe we have Branch Fixed, Pull Request and Tag Creation.

i) Branch Fixed

If you select the Branch Fixed as your source type for building CI Pipeline, then you need to provide the corresponding Branch Name.

Branch Name is the name of the corresponding branch (eg. main or master, or any other branch)

ii) Pull Request

[Note] It only works if Git Host is Github or Bitbucket Cloud as of now. In case you need support for any other Git Host, please create a github issue.

If you select the Pull Request option, you can configure the CI Pipeline using the generated PR. For this mechanism you need to configure a webhook for the repository added in the Git Material.

Prerequisites for Pull Request

If using GitHub - To use this mechanism, as stated above you need to create a webhook for the corresponding repository of your Git Provider. In Github to create a webhook for the repository -

1. Go to settings of that particular repository

2. Click on webhook section under options tab

3. In the Payload URL section, please copy paste the Webhook URL which can be found at Devtron Dashboard when you select source type as Pull Request as seen in above image.

4. Change content type to - application/json

5. Copy paste the Secret as well from the Dashboard when you select the source type as Pull Request

Now, scroll down and select the custom events for which you want to trigger the webhook to build CI Pipeline -

1. Check the radio button for Let me select individual events

2. Then, check the Branch or Tag Creation and Pull Request radio buttons under the individual events as mentioned in image below.

[Note] If you select Branch or Tag Creation, it will work for the Tag Creation mechanism as well.

After selecting the respective options, click on the generate the webhook button to create a webhook for your respective repository.

If using Bitbucket Cloud - If you are using Bitbucket cloud as your git provider, you need to create a webhook for that as we created for Github in the above section. Follow the steps to create webhook -

1. Go to Repository Settings on left sidebar of repository window

2. Click on Webhooks and then click on Add webhook as shown in the image.

1. Give any appropriate title as per your choice and then copy-paste the url which you can get from Devtron Dashboard when you select Pull Request as source type in case of Bitbucket Cloud as Git Host.

2. Check the Pull Request events for which you want to trigger the webhook and then save the configurations.

Filters

Now, coming back to the Pull Request mechanism, you can observe we have the option to add filters. In a single repository we have multiple PRs generated, so to have the exact PR for which you want to build the CI Pipeline, we have this feature of filters.

You can add a few filters which can be seen in the dropdown to sort the exact PR which you want to use for building the pipeline.

Below are the details of different filters which you can use as per your requirement. Please select any of the filters and pass the value in regex format as one has already given for example and then click on Create Pipeline.

Devtron uses regexp library, view regexp cheatsheet. You can test your custom regex from here.

iii) Tag Creation

The third option i.e, Tag Creation. In this mechanism you need to provide the tag name or author to specify the exact tag for which you want to build the CI Pipeline. To work with this feature as well, you need to configure the webhook for either Github or Bitbucket as we did in the previous mechanism i.e, Pull Request.

In this process as well you can find the option to filter the specific tags with certain filter parameters. Select the appropriate filter as per your requirement and pass the value in form of regex, one of the examples is already given.

Select the appropriate filter and pass the value in the form of regex and then click on Create Pipeline.

Advanced Options

When you click on the advanced options button which can be seen at the bottom-left of the screen, you can see some more configuration options which includes pipeline execution, stages and scan for vulnerabilities.

Stages:

There are 3 dropdowns given below:

• Pre-build

• Docker build

• Post-build

(a) Pre-build

This section is used for those steps which you want to execute before building the Docker image. To add a Pre-build stage, click on Add Stage and provide a name to your pre-stage and write your script as per your requirement. These stages will run in sequence before the docker image is built. Optionally, you can also provide the path of the directory where the output of the script will be stored locally.

You can add one or more than one stage in a CI Pipeline.

(b) Docker build

Though we have the option available in the Docker build configuration section to add arguments in key-value pairs for the docker build image. But one can also provide docker build arguments here as well. This is useful, in case you want to override them or want to add new arguments to build your docker image.

(c) Post-build

The post-build stage is similar to the pre-build stage. The difference between the post-build stage and the pre-build stage is that the post-build will run when your CI pipeline will be executed successfully.

Adding a post-build stage is similar to adding a pre-build stage. Click on Add Stage and provide a name to your post-stage. Here you can write your script as per your requirement, which will run in sequence after the docker image is built. You can also provide the path of the directory in which the output of the script will be stored in the Remote Directory column. And this is optional to fill because many times you run scripts that do not provide any output.

NOTE:

(a) You can provide pre-build and post-build stages via the Devtron tool’s console or can also provide these details by creating a file devtron-ci.yaml inside your repository. There is a pre-defined format to write this file. And we will run these stages using this YAML file. You can also provide some stages on the Devtron tool’s console and some stages in the devtron-ci.yaml file. But stages defined through the Devtron dashboard are first executed then the stages defined in the devtron-ci.yaml file.

(b) The total timeout for the execution of the CI pipeline is by default set as 3600 seconds. This default timeout is configurable according to the use-case. The timeout can be edited in the configmap of the orchestrator service in the env variable env:"DEFAULT_TIMEOUT" envDefault:"3600"

Scan for vulnerabilities

Scan for vulnerabilities adds a security feature to your application. If you enable this option, your code will be scanned for any vulnerabilities present in your code. And you will be informed about these vulnerabilities. For more details please check doc

You have provided all the details required to create a CI pipeline, now click on Create Pipeline.

Update CI Pipeline

You can also update any configuration of an already created CI Pipeline, except the pipeline name. The pipeline name can not be edited.

Click on your CI pipeline, to update your CI Pipeline. A window will be popped up with all the details of the current pipeline.

Make your changes and click on Update Pipeline at the bottom to update your Pipeline.

Delete CI Pipeline

You can only delete CI pipeline if you have no CD pipeline created in your workflow.

To delete a CI pipeline, go to the App Configurations and then click on Workflow editor

Click on Delete Pipeline at the bottom to delete the CD pipeline

Automated Test suite integration in the CI step using devtron-ci.yaml

Users can run the test case using the Devtron dashboard or by including the test cases in the devtron.ci.yaml file in the source git repository. For reference, check: https://github.com/kumarnishant/getting-started-nodejs/blob/master/devtron-ci.yaml

The test cases given in the script will run before the test cases given in the devtron.ci.yaml

B. Linked CI Pipeline

If one code is shared across multiple applications, Linked CI Pipeline can be used, and only one image will be built for multiple applications because if there is only one build, it is not advisable to create multiple CI Pipelines.

To create a Linked CI Pipeline, please follow the steps mentioned below :

1. Click on + New Build Pipeline button.

1. Select Linked CI Pipeline.

1. Select the application in which the source CI pipeline is present.

2. Select the source CI pipeline.

3. Provide a name for linked CI pipeline.

4. Click on Create Linked CI Pipeline button to create linked CI pipeline.

After creating a linked CI pipeline, you can create a CD pipeline. You cannot trigger build from linked CI pipeline, it can be triggered only from source CI pipeline. Initially you will not see any images to deploy in CD pipeline created from linked CI pipeline. Trigger build in source CI pipeline to see the images in CD pipeline of linked CI pipeline. After this, whenever you trigger buld in source CI pipeline, the build images will be listed in CD pipeline of linked CI pipeline too.

C. Incoming Webhook

You can use Devtron for deployments on Kubernetes while using your own CI tool such as Jenkins. External CI features can be used for cases where the CI tool is hosted outside the Devtron platform.

You can send the ‘Payload script’ to your CI tools such as Jenkins and Devtron will receive the build image every time the CI Service is triggered or you can use the Webhook URL which will build an image every time CI Service is triggered using Devtron Dashboard.

You will see all your environments associated with an application under the Environment Overrides section.

You can customize your Deployment template, ConfigMap, Secrets in Environment Overrides section to add separate customizations for different environments such as dev, test, integration, prod, etc.

Deployment template - Functionality

If you want to deploy an application in a non-production environment and then in production environment once testing is done in non-prod environment, then you do not need to create a new application for prod environment. Your existing pipeline(non-prod env) will work for both the environments with little customization in your deployment template under Environment overrides.

Example customization:

In a Non-production environment, you may have specified 100m CPU resources in the deployment template but in the Production environment you may want to have 500m CPU resources as the traffic on Pods will be higher than traffic on non-prod env.

Configuring the Deployment template inside Environment Overrides for a specific environment will not affect the other environments because Environment Overrides will configure deployment templates on environment basis. And at the time of deployment, it will always pick the overridden deployment template if any.

If there are no overrides specified for an environment in the Environment Overrides section, the deployment template will be the one you specified in the deployment template section of the app creation.

(Note: This example is meant only for a representational purpose. You can choose to add any customizations you want in your deployment templates in the Environment Overrides tab)

Any changes in the configuration will not be added to the template, instead, it will make a copy of the template and lets you customize it for each particular environment. And now this overridden template will be used only for the specified Environment.

This will save you the trouble to manually create deployment files separately for each environment. Instead, all you have to do is to change the required variables in the deployment template.

How to add Environment Overrides

In the Environment Overrides section, click on Allow Override and make changes to your Deployment template and click on Save to save your changes of the Deployment template.

ConfigMaps & Secrets

The same goes for ConfigMap and Secrets. You can also create an environment-specific configmap and Secrets inside the Environment override section.

If you want to configure your ConfigMap and secrets at the application level then you can provide them in ConfigMaps and Secrets, but if you want to have environment-specific ConfigMap and secrets then provide them under the Environment override Section. At the time of deployment, it will pick both of them and provide them inside your cluster.

Click on Update ConfigMap to update Configmaps.

Click on Update Secrets to update Secrets.

Once you are done creating your CI pipeline, you can move start building your CD pipeline. Devtron enables you to design your CD pipeline in a way that fully automates your deployments.

Creating CD Pipeline

Click on “+” sign on CI Pipeline to attach a CD Pipeline to it. A basic Create deployment modal will pop up.

This section expects two inputs:

• Select Environment

• Deployment Strategy

1. Select Environment

This section further including two inputs:

(a) Deploy to Environment

Select the environment where you want to deploy your application.

(b) Namespace

This field will be automatically populated with the Namespace corresponding to the Environment selected in the previous step.

Click on Create Pipeline to create a CD pipeline.

One can have a single CD pipeline or multiple CD pipelines connected to the same CI Pipeline. Each CD pipeline corresponds to only one environment, or in other words, any single environment of an application can have only one CD pipeline. So, the images created by the CI pipeline can be deployed into multiple environments through different CD pipelines originating from a single CI pipeline. If you already have one CD pipeline and want to add more, you can add them by clicking on the + sign and then choosing the environment in which you want to deploy your application. Once a new CD Pipeline is created for the environment of your choosing, you can move ahead and configure the CD pipeline as required. Your CD pipeline can be configured for the pre-deployment stage, the deployment stage, and the post-deployment stage. You can also select the deployment strategy of your choice. You can add your configurations as explained below:

To configure the advance CD option click on Advance Options at the bottom.

1. Pipeline Name

Pipeline name will be autogenerated.

2. Deploy to Environment

As we discussed above, Select the environment where you want to deploy your application. Once you select the environment, it will display the Namespace corresponding to your selected environment automatically.

Stages