Try Devtron Enterprise!
Start Free Trial
LogoLogo
WebsiteDevtron demoGithub RepoJoin Discord
main
  • Introduction
  • Getting Started
  • Install Devtron
    • Install Devtron with CI/CD
    • Install Devtron with CI/CD and GitOps (Argo CD)
    • Install Devtron without Integrations
    • Install Devtron on Minikube, Microk8s, K3s, Kind, Cloud VMs
    • Install Devtron on Airgapped Environment
    • Demo on Popular Cloud Providers
    • Backup for Disaster Recovery
    • Uninstall Devtron
    • FAQs
  • Install Devtron Enterprise Trial
  • Devtron Kubernetes Client
  • Production Infra Recommendations
  • Configurations
    • Installation Configurations
    • Override Configurations
    • Ingress Setup
  • Global Configurations
    • Host URL
    • GitOps
    • Projects
    • Clusters & Environments
    • Git Accounts
    • Container/OCI Registry
    • Chart Repositories
    • Deployment Charts
    • Authorization
      • SSO Login Services
        • Google
        • GitHub
        • GitLab
        • Microsoft
        • LDAP
        • OIDC
          • Keycloak
          • Okta
        • OpenShift
      • User Permissions
      • Permission Groups
      • API Tokens
    • Notifications
    • Deployment Window
    • Approval Policy
    • External Links
    • Catalog Framework
    • Scoped Variables
    • Plugin Policy
    • Pull Image Digest
    • Tags Policy
    • Filter Condition
    • Lock Deployment Configuration
    • Image Promotion Policy
    • Build Infra
  • Devtron Upgrade
    • Update Devtron from Devtron UI
    • Upgrade to 1.5.0
    • 0.6.x-0.7.x
    • 0.5.x-0.6.x
    • 0.4.x-0.5.x
    • 0.4.x-0.4.x
    • 0.3.x-0.4.x
    • 0.3.x-0.3.x
    • 0.2.x-0.3.x
  • Usage
    • Applications
      • Create a New Application
      • Clone an Existing Application
      • Deploy a Sample Application
      • App Configuration
        • Git Repository
        • Build Configuration
        • Base Deployment Template
          • Deployment
          • Rollout Deployment
          • Job and Cronjob
          • StatefulSets
        • GitOps Configuration
        • Workflow Editor
          • CI Pipeline
            • Pre-Build/Post-Build Stages
            • Override Build Configuration
          • CD Pipeline
        • ConfigMaps
        • Secrets
          • External Secret Operator (ESO)
            • AWS Secrets Manager
            • Google Secrets Manager
            • HashiCorp Vault
        • Environment Overrides
        • Deleting Application
      • Build and Deploy
        • Triggering CI
        • Triggering CD
        • Rollback Deployment
        • Applying Labels to Images
      • App Details
        • Debugging Deployment And Monitoring
        • Using Ephemeral Containers
        • Application Metrics
        • Deployment Visibility & Actions
      • Application Overview
    • Jobs
      • Create a new job
      • Configurations
      • Workflow Editor
      • Trigger Job
      • Overview
    • Application Groups
    • Software Distribution Hub
      • Tenants
      • Release Hub
    • Resource Browser
      • Overview Page
      • Discover and Manage Resources
      • Nodes and Operations
      • Pod Management and Debugging
      • Using Devtron Intelligence
      • Cluster Terminal
      • Add Monitoring Dashboards/Graphs
      • Run Kubectl Commands Locally
    • Resource Watcher
    • Charts
      • Charts Overview
      • Deploy & Observe
      • Examples
        • Deploying Mysql Helm Chart
        • Deploying MongoDB Helm Chart
      • Chart Group
    • Security
      • Security Scans
      • Security Policies
    • Bulk Edit
    • Integrations
      • Build and Deploy (CI/CD)
      • GitOps (Argo CD)
      • Vulnerability Scanning
        • Clair
        • Trivy
      • Notifications
      • Monitoring (Grafana)
    • Pipeline Plugins
      • Create Your Plugin
      • Our Plugins
        • Ansible Runner
        • Bitbucket Runner Trigger
        • Codacy
        • Code-Scan
        • Copacetic
        • Container Image Exporter
        • Copy Container Image
        • Cosign
        • CraneCopy
        • Dependency track - Maven & Gradle
        • Dependency track - NodeJS
        • Dependency track - Python
        • Devtron CD Trigger
        • Devtron CI Trigger
        • Devtron Job Trigger
        • DockerSlim
        • EKS Create Cluster
        • GCS Create Bucket
        • GitHub Pull Request Updater
        • GKE Provisioner
        • GoLang-migrate
        • Jenkins
        • Jira Issue Validator
        • Jira Issue Updater
        • K6 Load Testing
        • Pull images from container repository
        • Semgrep
        • SonarQube
        • SonarQube v1.1.0
        • Terraform CLI
        • Vulnerability Scanning
  • Resources
    • Glossary
    • Troubleshooting
    • Use Cases
      • Devtron Generic Helm Chart To Run CronJob Or One Time Job
      • Connect SpringBoot with Mysql Database
      • Connect Expressjs With Mongodb Database
      • Connect Django With Mysql Database
      • Pull Helm Charts from OCI Registry
    • Telemetry Overview
    • Devtron on Graviton
    • Release Notes
Powered by GitBook
On this page
  • Introduction
  • Creating a Watcher
  • Basic Details
  • Namespaces to Watch
  • Intercept Change in Resources
  • Execute Runbook
  • Viewing Intercepted Changes
  • Details
  • Change in Resource
  • Job Execution Log
  • Use Cases
  • Live Stream Traffic Surge
  • Pod Health Monitoring

Was this helpful?

Export as PDF
  1. Usage

Resource Watcher

PreviousRun Kubectl Commands LocallyNextCharts

Last updated 3 days ago

Was this helpful?

Introduction

An incident response if delayed can impact businesses, revenue, and waste valuable engineering time. Devtron's Resource Watcher enables you to perform automated actions upon the occurrence of events:

  • Create Event - Occurs when a new Kubernetes resource is created, for e.g., a new pod spun up to handle increased traffic.

  • Update Event - Occurs when an existing Kubernetes resource is modified, for e.g., deployment configuration tweaked to increase the replica count.

  • Delete Event - Occurs when an existing Kubernetes resource is deleted, for e.g., deletion of an orphaned pod.

You can make the Resource Watcher listen to the above events and accordingly run a webhook or run a job you wish to get done, for e.g., increasing memory, executing a script, raising Jira ticket, emailing your stakeholders, sending Slack notifications, and many more. Since manual intervention is absent, the timely response of this auto-remediation system improves your operational efficiency.


Creating a Watcher

Who Can Perform This Action?

Users need to have super-admin permission to create a watcher.

This page allows you to create a watcher to track events and run a job. It also shows the existing list of watchers (if any).

  1. Click + Create Watcher.

    Figure 1: Watchers - Page
  2. Creating a watcher consists of 4 parts, fill all the sections one by one:

    • Basic Details

    • Namespaces to Watch

    • Intercept Change in Resources

    • Execute Runbook

    Figure 2: Create Watcher - Window

Basic Details

Here, you can give a name and description to your watcher.

Namespaces to Watch

Here, you can select the namespaces whose Kubernetes resource you wish to monitor for changes.

  • You can watch the namespace(s) across All Clusters (existing and future).

  • Or you can watch namespace(s) of Specific Clusters.

In both the above options, if you choose 'Specific Namespaces', you can further decide whether to track the namespaces you enter (by clicking 'Include selections') or to track the namespaces except the ones you enter (by clicking 'Exclude selections').

Intercept Change in Resources

Here, you can select the exact Kubernetes resource(s) you wish to track for changes (in the namespace(s) you selected in the previous step).

  • You can choose the resource from the Resource kind(s) to watch dropdown. Enter the Group/Version/Kind (GVK) if it's a custom resource definition (CRD), for e.g., install.istio.io/v1apha1/IstioOperator

  • Choose the event type your watcher should listen to: Created, Updated, Deleted.

    Event Type
    Description

    Created

    Triggers the watcher when your Kubernetes resource is created

    Updated

    Triggers the watcher when your existing Kubernetes resource is modified

    Deleted

    Triggers the watcher when your existing Kubernetes resource is deleted

  • Enter a CEL expression to catch a specific change in the resource's manifest.

  • If resource is created - Use 'DEVTRON_FINAL_MANIFEST'

  • If resource is updated - Both 'DEVTRON_INITIAL_MANIFEST' and 'DEVTRON_FINAL_MANIFEST' can exist

  • If resource is deleted - Use 'DEVTRON_INITIAL_MANIFEST'

Example: DEVTRON_FINAL_MANIFEST.status.currentReplicas == DEVTRON_FINAL_MANIFEST.spec.maxReplicas

Execute Runbook

Trigger Devtron Job

The Trigger Devtron Job option allows you to choose a Devtron job pipeline that triggers a job (e.g., executing a script, emailing your stakeholders, etc.) whenever the watcher intercepts any changes.

Follow the below steps to trigger Devtron job:

  1. Select the Trigger Devtron Job option.

  2. Choose your preferred Devtron job pipeline from the Run Devtron Job pipeline drop-down box. The pipelines configured while creating a job are displayed as options in the Run Devtron Job pipeline drop-down box. Unless a job pipeline is selected, the watcher will not intercept matching resource changes, even if defined conditions are met.

  3. Select your preferred environment for the job to run in the Execute job in environment drop-down box.

  4. Enter the runtime parameters (Key, Type, Value) in the Runtime Parameters section.

    When a job is executed, its container can access the initial and final resource manifest through the following special environment variables:

    • To access the initial resource manifest use: DEVTRON_INITIAL_MANIFEST

    • To access the final resource manifest use: DEVTRON_FINAL_MANIFEST

  5. Click Create Watcher.

The watcher is now ready to intercept changes to selected resources and execute the configured job.

Trigger Webhook

The Trigger Webhook option allows you to configure a Webhook URL along with the payload (data) to be sent whenever the webhook is triggered. For example, to receive notifications in Slack, you can provide the Slack webhook URL and define the payload accordingly.

Follow the below steps to trigger webhook:

  1. Select the Trigger Webhook option.

  2. Enter your complete webhook URL, including https://, in the Webhook URL field. Your configured payload will be delivered through this webhook.

  3. Enter the relevant header key-value pairs that are necessary for authentication or to include additional metadata for the receiving endpoint in the Header Key and Value fields accordingly.

  4. Configure the payload in the Payload (Data to be shared through Webhook) field.

    The payload is the actual content delivered to the webhook endpoint when the webhook is triggered. It contains the relavant information about changes in intercepted resources. The payload data must be entered in valid JSON format. Other formats such as YAML, or plain text are not supported. You can also customize the payload to include any resource-specific values that are useful for your integration. You can pass the properties of resource manifest in the webhook payload using the following keys:

    • To access the initial resource manifest use: DEVTRON_INITIAL_MANIFEST

    • To access the final resource manifest use: DEVTRON_FINAL_MANIFEST

      The above keys return values as stringified JSON.

  5. Click Create Watcher.

The watcher is now ready to intercept changes to selected resources and trigger the webhook.


Viewing Intercepted Changes

Who Can Perform This Action?

Users need to have super-admin permission to view intercepted changes.

Details

This page allows you to view the changes to Kubernetes resources that you have selected for tracking changes.

It comes with the following items to help you locate the resource, where the event has been intercepted:

  • Searchbox

  • Cluster filter

  • Namespace filter

  • Action filter (event type, i.e., Created, Updated, Deleted)

  • Watcher filter (to check the intercepted changes of a specific watcher)

You get the following details in the results shown on the page.

Field
Description

Describes the type of change to the Kubernetes resource along with a link to its manifest

Shows the cluster and namespace where the tracked Kubernetes resource belongs to

Intercepted By

Shows the name of the watcher that intercepted the change

Intercepted At

Shows the date and time when the event occurred

Shows the status of the execution of job, e.g., In Progress, Succeeded, Failed

Links to the job log, i.e, the Run history page of the job

Change in Resource

You can check the changes in manifest by clicking View Manifest in Change In Resource column.

Job Execution Log

You can check the logs of the job executed when the Resource Watcher intercepts any change by clicking logs.


Use Cases

Live Stream Traffic Surge

A live streaming sports application experiences a surge in viewers during a major game. The Horizontal Pod Autoscaler (HPA) might not be able to handle the unexpected traffic if it's capped at a low max replica count.

  1. Create a watcher named 'Live Stream Scaling Alert'.

  2. Monitor updates to HPA resource in the application's namespace.

  3. When currentReplicas count reaches maxReplicas, trigger a job that contains the script to increase the replica count.

Pod Health Monitoring

A stock trading application constantly updates stock prices for its traders. If the pods become unhealthy, traders might see incorrect stock prices leading to bad investments.

  1. Create a watcher named 'Pod Health Monitor'.

  2. Track the pod workload of your application, if DEVTRON_FINAL_MANIFEST.status.phase != 'Running', trigger a job that sends an Email/Slack alert with pod details.

Figure 3: Adding Name and Description of Watcher
Figure 4: Choosing Namespaces of all Clusters
Figure 5: Choosing Namespaces of Specific Clusters
Figure 6: Picking Resources to Track
Figure 7: Trigger Devtron Job
Figure 8: Intercepted Changes
Figure 9: Trigger Webhook
Figure 10: Intercepted Changes
Figure 11: Intercepted Changes - Page

Figure 12a: Created Resource Manifest - Final Manifest
Figure 12b: Updated Resource - Initial and Final Manifest
Figure 12c: Deleted Resource - Initial Manifest
Figure 13: Job Progress
Change In Resource
Cluster/Namespace
Job Execution
Logs