🔔 Alert Management

Alertmanager Configuration

Configure Alertmanager for intelligent alert routing, grouping, and notifications. Learn to set up multiple notification channels and manage alert lifecycle effectively.

Notification Channels

Email Notifications

Send alerts via email to team members

Configure SMTP settings to send detailed alert notifications to email addresses.

smtpHost:smtp.gmail.com:587
smtpAuth:true
smtpUser:alerts@company.com
templates:HTML and text templates

Slack Integration

Send alerts to Slack channels

Integrate with Slack for real-time team notifications and collaboration.

webhookUrl:https://hooks.slack.com/services/...
channel:#alerts
username:Alertmanager
templates:Slack-specific formatting

PagerDuty Integration

Escalate critical alerts to on-call engineers

Route critical alerts to PagerDuty for immediate incident response.

integrationKey:pagerduty-integration-key
severity:critical, warning
autoResolve:true
escalation:On-call rotation

Webhook Endpoints

Send alerts to custom webhook endpoints

Configure custom webhooks to integrate with internal systems and APIs.

url:https://api.company.com/alerts
method:POST
headers:Authorization, Content-Type
payload:JSON formatted alerts

Alert Routing Configuration

Routing Rules

Configure how alerts are routed to different teams and notification channels based on severity and labels.

Critical Alerts

critical-team
Matchers:severity=critical
Group By:alertname, cluster
Group Wait:10s
Group Interval:5m
Repeat Interval:1h

Warning Alerts

warning-team
Matchers:severity=warning
Group By:alertname
Group Wait:30s
Group Interval:10m
Repeat Interval:4h

Info Alerts

info-team
Matchers:severity=info
Group By:alertname
Group Wait:1m
Group Interval:30m
Repeat Interval:12h

Key Features

Alert Grouping

Group related alerts to reduce notification noise

Combine multiple alerts into single notifications based on labels and time windows.

Inhibition Rules

Suppress lower-priority alerts when higher-priority ones are firing

Prevent alert spam by automatically suppressing related alerts.

Silencing

Temporarily silence specific alerts or alert patterns

Manually or automatically silence alerts during maintenance windows.

Rich Templates

Customize notification content with templates

Create custom notification templates for different channels and teams.

Sample Configuration

Alertmanager Configuration File

A complete Alertmanager configuration example with routing and notification setup.

global:
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: 'alerts@company.com'

route:
  group_by: ['alertname']
  group_wait: '10s'
  group_interval: '10s'
  repeat_interval: '1h'
  receiver: 'web.hook'
  routes:
  - match:
      severity: critical
    receiver: 'critical-team'
    group_wait: '5s'
    repeat_interval: '30m'
  - match:
      severity: warning
    receiver: 'warning-team'
    group_wait: '30s'
    repeat_interval: '2h'

receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://127.0.0.1:5001/'
    
- name: 'critical-team'
  email_configs:
  - to: 'critical@company.com'
    subject: 'CRITICAL: {{ .GroupLabels.alertname }}'
    body: |
      {{ range .Alerts }}
      Alert: {{ .Annotations.summary }}
      Description: {{ .Annotations.description }}
      {{ end }}
      
- name: 'warning-team'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/...'
    channel: '#alerts'
    title: 'Warning Alert'
    text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'

Access Alertmanager

Quick Setup Guide

Access the Alertmanager UI to view and manage alerts.

1. Port Forward to Alertmanager

$ kubectl port-forward svc/prometheus-kube-prometheus-alertmanager 9093:9093 -n monitoring

2. Access Alertmanager UI

Open your browser and navigate to http://localhost:9093

3. Key Features Available

View Active Alerts
Configure Silences
Manage Groups

Ready to Monitor External Apps?

Now that you have a complete alerting system, learn how to monitor third-party applications and services.

Monitor Third-party AppsCustom Applications