Clusters & Environments

Introduction

Devtron allows you to connect and manage your existing Kubernetes clusters by adding them to its platform. Once a cluster is added, you can create different environments within it, making it possible to deploy your applications.

You can add any of the following cluster types:

• Kubernetes Cluster - If you have access to the cluster, use this option.

• Isolated Cluster - For airgapped-related use-cases, use this option.

Add Kubernetes Cluster

1. Go to Global Configurations → Clusters & Environments → Connect Cluster (button); a New Cluster modal window will appear.

1. Select Connect Cluster.

1. You can choose to add your Kubernetes cluster using either of the following methods:

1. Click Save Cluster and your cluster will be connected to Devtron.

Add Cluster Using Server URL & Bearer Token

1. To add a Kubernetes cluster on Devtron using Server URL and Bearer Token, provide the following information:

1. Complete the remaining steps (optional):

• Choose Connection Type

• Use Secure TLS Connection

• Configure Prometheus

• Assign a Category

Add Cluster Using Kubeconfig

In case you prefer to add clusters using kubeconfig, follow these steps:

1. Copy and paste your kubeconfig file into the editor. Alternatively, you may browse and select the file as well.

1. Click the Get Cluster button. This action will display the cluster details alongside the kubeconfig.

1. If your kubeconfig file lists multiple clusters, they will be displayed in the window. Use the checkboxes to select the desired cluster(s) and click Save.

1. Click the saved cluster, and complete the remaining steps (optional):

• Choose Connection Type

• Use Secure TLS Connection

• Configure Prometheus

• Assign a category

Assign Category to a Cluster

Devtron allows you to assign a category (for e.g. Prod, QA, Dev, or Stage) to your clusters. This enables category-based filtering in the UI, allowing you to determine whether an application is deployed to the Prod, QA, Dev, or Stage environment.

Before assigning a category, you must first add the category. To add a category, refer to the Adding a Category section to learn more.

To assign a category to a cluster, follow the steps below:

1. Select a category from the dropdown under Assign Category and click Update Cluster.

   
2. The selected category will be assigned to the cluster.

   

Choose Method of Connection

When adding a new cluster to Devtron, you must choose how Devtron will connect to it. There are three connection options available:

Direct Connection

Clusters with a directly accessible API server endpoint, either publicly or via private peering, can be added as Direct Connection clusters.

• Devtron connects directly without an intermediary.

• Recommended when the cluster is publicly accessible or has a direct network route from Devtron.

Via Proxy

For security reasons, some Kubernetes clusters are deployed behind a proxy. In this setup, Devtron routes all communication through the specified proxy URL.

• Use this option when network restrictions require traffic to go through a proxy server.

• Requires specifying a Proxy URL (e.g., http://proxy.example.org:3128).

• Limitation: Deployments via GitOps (ArgoCD) are not recommended for clusters connected via proxy.

Via SSH Tunnel

When a direct connection isn't possible, Devtron can connect to the Kubernetes cluster through an SSH tunnel, ensuring secure and encrypted communication.

• Requires:

  • SSH Server URL (e.g., http://proxy.example.org).

  • Username for authentication.

  • Authentication Method:

    • Password

    • SSH Private Key

    • Both Password & SSH Private Key

• Limitation: Deployments via GitOps (ArgoCD) are not recommended for clusters connected via SSH Tunnel.

Use Secure TLS Connection

For a secure cluster connection, you can opt for TLS connection, where you need to provide Certificate Authority Data, a TLS Key, and a TLS Certificate.

If your cluster is managed (e.g., EKS, AKS, GKE), you might need to download these certificates from your cloud provider’s dashboard or API.

Configure Prometheus (Enable Application Metrics)

If you want to see application metrics against the applications deployed in the cluster, Prometheus must be deployed in the cluster. Prometheus is a powerful tool to provide graphical insight into your application behavior.

Enable application metrics to configure Prometheus as shown below. In case it is not available, make sure to install the Monitoring (Grafana) integration from Devtron Stack Manager to configure Prometheus.

Provide the information in the following fields:

Click Save Cluster to save your cluster on Devtron.

Create Kubernetes Cluster

Prerequisites

To create an EKS cluster, you need:

• OpenTofu (tofu-controller) chart installed in your Devtron instance.

  Refer to Getting Started with OpenTofu for more information.

• FluxCD controller (flux2) chart installed in your Devtron instance

• Secret containing AWS credentials

Install OpenTofu

Follow the steps mentioned below to install OpenTofu:

1. Navigate to Global Configurations → Container/OCI Registry.

2. Refer to the following table and enter the information in the appropriate fields:

   Field

   Value

   Registry provider

   Other

   Registry type

   Public Registry

   Name

   tofu

   Registry URL

   ghcr.io

   List of repositories

   flux-iac/charts/tofu-controller

   
3. Click Save. The tofu-controller chart will be displayed in the Chart Store page.

4. Navigate to Chart Store and search for tofu-controller in the search box.

5. Select the chart and click Configure & Deploy. The following page will be displayed.

   
6. Enter the app name (e.g., tofu-controller) in the App Name field.

7. Select your project in the Project drop-down box.

8. Select the environment where you want to deploy the chart in the Deploy to Environment drop-down box.

1. Choose either Helm or GitOps if configured as the deployment method.

2. Click Deploy Chart. OpenTofu will be installed in your Devtron instance.

Now that OpenTofu is installed, you can install the FluxCD controller in your Devtron instance.

Install FluxCD Controller

Follow the steps mentioned below to install OpenTofu:

1. Navigate to Chart Store and search for flux2 in the search box.

   
2. Select the chart and click Deploy.

3. Enter the app name (e.g., tofu2) in the App Name field.

4. Select your project in the Project drop-down box.

5. Select the environment where you want to deploy the chart in the Deploy to Environment drop-down box.

1. Choose either Helm or GitOps if configured as the deployment method.

2. Click Deploy Chart. FluxCD controller will be installed in your Devtron instance.

Now that FluxCD controller is installed, the final prerequisite is to create a secret containing your AWS credentials.

Create a Secret

Follow the steps mentioned below to create a secret containing your AWS credentials:

1. Navigate to Resource Browser.

2. Click the default_cluster.

3. Click Create Resource.

4. Copy the YAML snippet given below and paste it in the Create Kubernetes Resource page.

   Copy

   apiVersion: v1
   data:
     AWS_ACCESS_KEY_ID: SDKDI382DKD0=
     AWS_SECRET_ACCESS_KEY: YVZsSIEOwcFRSMjlvM2xaUjSIE823J3PT0=
   kind: Secret
   metadata:
     name: tf-aws-creds
     namespace: your-namespace
   type: Opaque

1. Enter your AWS access key against the AWS_ACCESS_KEY_ID attribute and AWS secret key against the AWS_SECRET_ACCESS_KEY attribute. Refer to Create New Access Keys for more information.

2. Click Apply. The secret will be created.

Now that all the prerequisites are met, you can proceed to create a cluster from the Create Kubernetes Cluster page.

Steps

• Navigate to Global Configurations → Clusters & Environments → New Cluster → Create Kubernetes Cluster.

• Refer the following table (containing mandatory fields) and enter the details in the corresponding fields:

  Field

  Description

  Cluster Provider

  Select the type of cluster you'd like to create based on your requirement

  Name

  Enter the name of your Kubernetes cluster (e.g., eks-cluster-nonprod in the case of EKS and rancher-cluster-qa in the case of Rancher)

  Region

  Select the region where your cluster is hosted (e.g., us-east-1 in the case of EKS and ap-south-3 in the case of Rancher) Refer to View cluster details using the AWS Management Console for more information

  VPC CIDR

  Enter the VPC CIDR value. This value determines the number of pods, nodes, or services your cluster can host (e.g., 10.0.1.6/16)

  Authentication Mode

  Select the authentication mode you wish to perform for the cluster

  • API_AND_CONFIG_MAP - Select this if you want to use both the API and the ConfigMap to authenticate who can access the cluster. This option is recommended if you are migrating from the old aws-auth ConfigMap method (which is deprecated) to the new API method. Refer to Grant IAM users access to Kubernetes with EKS access entries for more information.

  • API - Select this if you want to manage access using a single API. This option is recommended as this is the best practice for EKS cluster creation. Refer to Manage User Access with API for more information.

  • CONFIG MAP - Select this if you want to rely on the original (but deprecated) way of authentication using aws-auth ConfigMap. This option is not recommended anymore. Refer to Grant IAM users access to Kubernetes. with a ConfigMap. for more information.

  Enable IRSA

  Turn on this IRSA toggle (IAM Roles for Service Accounts) if you want your application to securely connect to other AWS services using a service account

  Allow public access

  Turn on this toggle if you want to allow your EKS control plane endpoint to be accessed publicly from anywhere without the VPC. It is recommended to keep this toggle disabled

  Cluster Version

  Select your preferred Kubernetes cluster version. If you are running a live application in a production environment, it is recommended that you select a stable version instead of the latest version

  

• Refer the following table (containing optional fields) and enter the details in the corresponding fields:

  Field

  Description

  Team

  Select the team whose tag you want to attach to the cluster resources. For example, when you select qa-team, it means that the cluster resources (pods, ConfigMaps, etc.) created with this cluster are owned by the QA team

  Environment

  Select the environment. For example, when you select qa, it means that this cluster is a part of the QA environment

  Availability Zones

  Select availability zones (e.g., us-east-2b and ap-west-1a) if you prefer to distribute your worker nodes across multiple zones to make your cluster highly available. This means that even if one availability zone goes down (e.g., us-east-2b), the other zones (e.g., ap-west-1a) keep your cluster up and running

  Private access CIDRs

  Enter the private access CIDRs (IP addresses that are allowed to reach the API server). If you had turned off the Allow public access toggle, then your EKS control plane endpoint would be private. It then becomes crucial to enter the private access CIDRs so that the API server recognizes them and allows them to access the endpoint

• Click Create Cluster.

Add Isolated Cluster

For air-gapped Kubernetes clusters with restricted inbound and outbound traffic, Devtron enables seamless management using isolated clusters. While these are not actual clusters with API endpoints, they provide a convenient way to deploy applications in such environments.

1. On the New Cluster modal window, select Add Isolated Cluster.

1. Add a cluster name (e.g. banking-airgapped-cluster) and click Save Cluster.

You have successfully configured an isolated cluster.

Add Environment to a Cluster

After adding a cluster to Devtron (Kubernetes Cluster, Isolated Cluster, and a newly created cluster), initially it has no environments.

1. Select the Cluster to which you want to add an Environment and click Add Environment. Alternatively you can also hover over the cluster and click + icon (Add Environment icon); an Add Environment modal window appears.

1. Fill the following details within the Add Environment modal window.

1. Assign a Category to environment - Devtron allows you to assign a category (for e.g. Prod, QA, Dev, or Stage) to your environments. This enables category-based filtering in the UI, allowing you to determine whether an application is deployed to Prod, QA, Dev, or Stage environment. To assign a category to your environment, follow the steps below:

   1. Select a category from the dropdown under Assign Category and click Update.

   

   1. The selected category will be assigned to the environment.

   

Note: Before assigning a category, you must first add the category. To add a category, refer to Adding a Category section to learn more.

1. Add/Edit labels to namespace - You can attach labels to your specified namespace in the Kubernetes cluster. Using labels will help you filter and identify resources via CLI or other Kubernetes tools. Click here to know more about labels.

1. Click Save. Your new environment will be visible in your cluster as shown below.

Edit Environment

You can also make edits to an existing environment if needed.

1. Navigate to Environments tab.

2. Hover over the environment you wish to edit, and click the edit icon.

1. Edit the environment fields.

1. Click Update to save your changes.

Delete Environment

If an environment is no longer needed, you can delete it by following these steps:

1. Navigate to Environments tab.

2. Hover over the environment you wish to remove, and click the delete icon.

1. A confirmation dialog will appear. Click Delete to permanently delete the environment.

   

Add Category

Before assigning a category, you must first add the category. To add a category, follow the steps below:

1. Go to Global Configurations.

   
2. Select Clusters and Environments and click Manage Categories, a modal window will open.

   
3. Enter the name of the category in the CATEGORIES field and provide a description in the DESCRIPTION field.

1. If you wish to add more categories, click Add Category, a new row will appear, enter the name and description of the new category.

   
2. Click Update and your categories will be added.

   

Delete Category

To delete a category, follow the steps below:

1. Go to Global Configurations.

   
2. Select Clusters and Environments and click Manage Categories, a modal window will open.

   
3. Select the x icon next to the categories you want to delete.

   Note: You cannot delete a category if it is assigned to a cluster or environment.

1. Click Update to delete the categories.

Extras

Get Cluster Credentials

You can get the Server URL and Bearer Token by running the following command, depending on the cluster provider:

curl -O https://raw.githubusercontent.com/devtron-labs/utilities/main/kubeconfig-exporter/kubernetes_export_sa.sh && bash kubernetes_export_sa.sh cd-user  devtroncd

curl -O https://raw.githubusercontent.com/devtron-labs/utilities/main/kubeconfig-exporter/kubernetes_export_sa.sh && sed -i 's/kubectl/microk8s kubectl/g' \
kubernetes_export_sa.sh && bash kubernetes_export_sa.sh cd-user \
devtroncd

Benefits of Self-hosted URL

• Disaster Recovery:

  • You cannot edit the server URL of a cloud-specific provider. If you're using an EKS URL (e.g. *****.eu-west-1.elb.amazonaws.com), it will be a tedious task to add a new cluster and migrate all the services one by one.

  • But in case of using a self-hosted URL (e.g. clear.example.com), you can just point to the new cluster's server URL in DNS manager and update the new cluster token and sync all the deployments.

• Easy Cluster Migrations:

  • In case of managed Kubernetes clusters (like EKS, AKS, GKE etc) which is a cloud provider specific, migrating your cluster from one provider to another will result in waste of time and effort.

  • On the other hand, migration for a self-hosted URL is easy, as the URL belongs to a single hosted domain independent of the cloud provider.