Alermanager prometheus send notify to slack chat

Slack Setup
Create new channel

Go to the Incoming Webhooks page in the App Directory, selecr our new channel and click “Add Incoming WebHooks”

Then copy Webhook URL

https://hooks.slack.com/services/

Alertmanager setup Slack
Configure alertmanager

global:
#templates:
#- '/prometheus-data/alertmanager/template/*.tmpl'
route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 30s
  repeat_interval: 8h
  receiver: WH
  routes:
  - match_re:
      service: ^(prm|prometheus|p)$
    receiver: WH
    routes:
    - match:
        severity: critical
      receiver: WH
  - match:
      service: files
    receiver: WH
    routes:
    - match:
        severity: critical
      receiver: WH
inhibit_rules:
- source_match:
    severity: 'critical'
  target_match:
    severity: 'warning'
  equal: ['alertname', 'cluster', 'service']
receivers:
 - name: 'WH'
   slack_configs:
   - channel: services-alerts
     send_resolved: true
     api_url: https://hooks.slack.com/services/T824GJ22323232Q5/B9WfsfadfsfQ9R9LH/TEu5fsdfdsf343Nmdu
     title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] Monitoring Event Notification'
     text: >-
       {{ range .Alerts }}
           *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
         *Description:* {{ .Annotations.description }}
         *Graph:* <{{ .GeneratorURL }}|:chart_with_upwards_trend:> *Runbook:* <{{ .Annotations.runbook }}|:spiral_note_pad:>
         *Details:*
         {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
         {{ end }}
       {{ end }}

Configure alert(example).

  - alert: Нода недоступна для мониторинга
    expr: up{job="node"} == 0
    for: 20s
    labels:
      severity: critical
    annotations:
      summary: "{{ $labels.instance }} Нода недоступна для мониторинга"
      description: "{{$labels.instance}}: Нода недоступна для мониторинга"
      runbook: "https://mywiki.com/ElasticacheCPUUtilisation"

Reload configs

curl -X POST http://IP_SERVICES:9090/-/reload ; curl -s -XPOST http://IP_SERVICES:9115/-/reload ; curl -s -XPOST http://IP_SERVICES:9093/-/reload

Profit!!!

Добавить комментарий