当前位置：网站首页>AlertManager告警的单独使用及prometheus配置告警规则使用

AlertManager告警的单独使用及prometheus配置告警规则使用

2022-06-23 03:52:00 【韧小钊】

目录

AlertManager

官方资料
在这里插入图片描述

下载部署

  • 下载地址
    在这里插入图片描述
  • 解压查看
    在这里插入图片描述
  • 可以规范下目录（批量创建常见的bin、conf、data结构）
[[email protected] ~]$ mkdir -p alertmanager/{bin,conf,data,logs,templates}
[[email protected] ~]$ ll alertmanager
总用量 0
drwxrwxr-x. 2 prometheus prometheus 57 5月  11 17:12 bin
drwxrwxr-x. 2 prometheus prometheus 47 5月  13 10:27 conf
drwxrwxr-x. 3 prometheus prometheus 81 5月  16 04:45 data
drwxrwxr-x. 2 prometheus prometheus  6 5月  11 15:23 logs
drwxrwxr-x. 2 prometheus prometheus  6 5月  11 15:23 templates
[[email protected] ~]$
  • 移动解压目录文件到相应目录
    执行文件移到bin目录，配置文件移到conf目录下
[[email protected] ~]$ tar xf alertmanager-0.24.0.linux-amd64.tar.gz 
[[email protected] ~]$  mkdir -p alertmanager/{bin,conf,data,logs,templates}
[[email protected] ~]$ cp alertmanager-0.24.0.linux-amd64/alertmanager alertmanager/bin/
[[email protected] ~]$ cp alertmanager-0.24.0.linux-amd64/amtool alertmanager/bin/
[[email protected] ~]$ cp alertmanager-0.24.0.linux-amd64/alertmanager.yml alertmanager/conf/
  • conf/alertmanager.yml文件内容
## Alertmanager 配置文件
global:
  resolve_timeout: 5s
  # smtp配置
  smtp_from: "[email protected]"
  smtp_smarthost: 'smtp.qq.com:25'
  smtp_auth_username: "[email protected]"
  smtp_auth_password: "ovosgonmfliabbbf"
  smtp_require_tls: true
 
# 路由分组
route:
  receiver: email
  group_wait: 3s # 在组内等待所配置的时间，如果同组内，30秒内出现相同报警，在一个组内出现。
  group_interval: 1m # 如果组内内容不变化，合并为一条警报信息，5m后发送。
  repeat_interval: 5m # 发送报警间隔，如果指定时间内没有修复，则重新发送报警。
  group_by: [alertname]  # 报警分组

 
# 接收器指定发送人以及发送渠道
receivers:
# ops分组的定义
- name: email
  email_configs:
  - to: '[email protected]'
    send_resolved: true
    headers:
      subject: "测试告警"
      from: "alertmanager"
      to: "test"

 
# 抑制器配置
inhibit_rules: # 抑制规则
  - source_match: # 源标签警报触发时抑制含有目标标签的警报，在当前警报匹配 status: 'High'
      status: 'High'  
    target_match:
      status: 'Warning' # 
    equal: ['alertname','operations', 'instance'] # 确保这个配置下的标签内容相同才会抑制，也就是说警报中必须有这三个标签值才会被抑制。
  • 启动alertmanager
    后台启动（也可以配置成系统服务启动）
nohup /home/prometheus/alertmanager/bin/alertmanager --storage.path=/home/prometheus/alertmanager/data/ --config.file=/home/prometheus/alertmanager/conf/alertmanager.yml &

在这里插入图片描述

  • 查看日志，默认9093监听端口
    在这里插入图片描述

prometheus 配置告警规则

创建告警规则文件

  • alert.yml （自定义名称，类型为yml）
[[email protected] data]$ pwd
/home/prometheus/alertmanager/data
[[email protected] data]$ cat alert.yml 
groups:
    - name: Node_Metrics
      rules:
       - alert: Node_Filesystem_Below_20MB
         expr: sum(node_filesystem_avail_bytes)by(instance)/1024/1024 < 20
         for: 2m
         labels:
           team: node
         annotations:
          summary: "{
   {$labels.instance}}: Filesystem Pressure"
          description: "{
   {$labels.instance}}: Filesystem is below 20MB"
       - alert: FLUME_CHANNEL_ChannelCapacity_1W
         expr: FLUME_CHANNEL_ChannelCapacity > 10000
         for: 2m
         labels:
           team: flume
         annotations:
           summary: "{
   {$labels.instance}}: low Capacity"
           description: "{
   {$labels.instance}}: low Capacity is above 10000 (current value is: {
   { $value }}"
[[email protected] data]$

加载告警规则文件

  • prometheus.yml
    在这里插入图片描述
# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
           - 192.168.56.10:9093
rule_files:
    - "/home/prometheus/alertmanager/data/alert.yml"

验证查看

prometheus查看地址：

  • http://192.168.56.10:9090/alerts
  • http://192.168.56.10:9090/rules

Alertmanager查看地址：

  • http://192.168.56.10:9093/#/alerts

inactive-未触发告警规则，pending-触发告警规则，待发送，firing-发送告警
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述

验证告警

修改告警规则

  • 改成大于20，测试告警
    在这里插入图片描述
  • 重启prometheus,重新加载规则

验证告警

  • 查看页面
    在这里插入图片描述
  • 等待状态变为firing后，查看是否收到告警邮件
    在这里插入图片描述
    在这里插入图片描述

dolphinscheduler配置AlertManager告警

AlertManager API

AlertManager API

dolphinscheduler HTTP告警代码修改

在这里插入图片描述

HttpSender源码

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.dolphinscheduler.plugin.alert.http;

import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import com.fasterxml.jackson.databind.node.ObjectNode;
import org.apache.dolphinscheduler.alert.api.AlertResult;
import org.apache.dolphinscheduler.spi.utils.JSONUtils;
import org.apache.dolphinscheduler.spi.utils.StringUtils;
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpRequestBase;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.util.EntityUtils;
import org.slf4j.Logger;

import java.util.HashMap;
import java.util.Map;

public final class HttpSender {
    private static final Logger log = org.slf4j.LoggerFactory.getLogger(HttpSender.class);
    private static final String URL_SPLICE_CHAR = "?";
    /**
     * request type post
     */
    private static final String REQUEST_TYPE_POST = "POST";
    /**
     * request type get
     */
    private static final String REQUEST_TYPE_GET = "GET";
    private static final String DEFAULT_CHARSET = "utf-8";
    private final String headerParams;
    private final String bodyParams;
    private final String contentField;
    private final String requestType;
    private String url;
    private HttpRequestBase httpRequest;

    public HttpSender(Map<String, String> paramsMap) {

        url = paramsMap.get(HttpAlertConstants.URL);
        headerParams = paramsMap.get(HttpAlertConstants.HEADER_PARAMS);
        bodyParams = paramsMap.get(HttpAlertConstants.BODY_PARAMS);
        contentField = paramsMap.get(HttpAlertConstants.CONTENT_FIELD);
        requestType = paramsMap.get(HttpAlertConstants.REQUEST_TYPE);
    }

    public AlertResult send(String msg) {

        AlertResult alertResult = new AlertResult();

        createHttpRequest(msg);

        if (httpRequest == null) {
            alertResult.setStatus("false");
            alertResult.setMessage("Request types are not supported");
            return alertResult;
        }

        try {
            CloseableHttpClient httpClient = HttpClientBuilder.create().build();
            CloseableHttpResponse response = httpClient.execute(httpRequest);
            HttpEntity entity = response.getEntity();
            String resp = EntityUtils.toString(entity, DEFAULT_CHARSET);
            alertResult.setStatus("true");
            alertResult.setMessage(resp);
        } catch (Exception e) {
            log.error("send http alert msg  exception : {}", e.getMessage());
            alertResult.setStatus("false");
            alertResult.setMessage("send http request  alert fail.");
        }

        return alertResult;
    }

    private void createHttpRequest(String msg) {
        if (REQUEST_TYPE_POST.equals(requestType)) {
            httpRequest = new HttpPost(url);
            setHeader();
            //POST request add param in request body
            setMsgInRequestBody(msg);
        } else if (REQUEST_TYPE_GET.equals(requestType)) {
            //GET request add param in url
            setMsgInUrl(msg);
            httpRequest = new HttpGet(url);
            setHeader();
        }
    }

    /**
     * add msg param in url
     */
    private void setMsgInUrl(String msg) {

        if (StringUtils.isNotBlank(contentField)) {
            String type = "&";
            //check splice char is & or ?
            if (!url.contains(URL_SPLICE_CHAR)) {
                type = URL_SPLICE_CHAR;
            }
            url = String.format("%s%s%s=%s", url, type, contentField, msg);
        }
    }

    /**
     * set header params
     */
    private void setHeader() {

        if (httpRequest == null) {
            return;
        }

        HashMap<String, Object> map = JSONUtils.parseObject(headerParams, HashMap.class);
        for (Map.Entry<String, Object> entry : map.entrySet()) {
            httpRequest.setHeader(entry.getKey(), String.valueOf(entry.getValue()));
        }
    }

    /**
     * set body params
     */
    private void setMsgInRequestBody(String msg) {
        ObjectNode objectNode = JSONUtils.parseObject(bodyParams);
        //set msg content field
        objectNode.put(contentField, msg);
        try {
        	//rxz 20220517 modify
        	msg = int2Str(msg);
            StringEntity entity = new StringEntity(msg, ContentType.APPLICATION_JSON);
            log.info("send http alert msg: {}",msg);
            ((HttpPost) httpRequest).setEntity(entity);
        } catch (Exception e) {
            log.error("send http alert msg  exception : {}", e.getMessage());
        }
    }
    
    private String int2Str(String msg) {
    	String sendMsg = "[{";
    	JSONArray array = JSONObject.parseArray(msg);
		for(Object arr:array) {
			JSONObject jsonObject = (JSONObject)arr;
			for (Map.Entry<String, Object> entry: jsonObject.entrySet()) {
	            Object o = entry.getValue();
	            if(o instanceof Integer ||o instanceof Long) {
	            	entry.setValue(entry.getValue().toString());
	            }
	        }
			sendMsg+=("\"labels\":"+jsonObject.toString()+",");
		}
    	return sendMsg.substring(0, sendMsg.length()-1)+"}]";
    }
}

测试验证

告警配置

  • 告警实例配置
http://192.168.56.10:9093/api/v1/alerts
POST
{"Content-Type":"application/json"}
{}
content

在这里插入图片描述

  • 告警组配置
    在这里插入图片描述

执行测试

  • 工作流启动

测试结果

  • 查看日志
    在这里插入图片描述
  • 邮件查看
    在这里插入图片描述

告警内容

[{
	"labels": {
		"owner": "admin",
		"taskHost": "192.168.56.10:1234",
		"processDefinitionCode": "5327475818624",
		"taskStartTime": "2022-05-16 09:30:11",
		"taskType": "PROCEDURE",
		"taskState": "FAILURE",
		"taskCode": "5327458833408",
		"processId": "62822",
		"processName": "存储过程pgsql-6-20220516093011807",
		"logPath": "/home/dolphinscheduler/app/dolphinscheduler/logs/5327475818624_6/62822/483135.log",
		"taskName": "存储过程pgsql",
		"projectName": "dolphin",
		"projectId": "30",
		"taskEndTime": "2022-05-16 09:30:23"
	}
}]

测试api2

在这里插入图片描述
在这里插入图片描述
在这里插入图片描述

API2测试结果

也是可以的，格式没有区别，只是url从v1换成了v2
在这里插入图片描述

其它

  • View In Alertmanager
    进入 Alertmanager页面，看不到刚刚发送的记录，看现象，第三方触发的告警不会逗留，只有触发告警规则的记录才会一直在这，直到不满足告警规则。怎么把alert信息存到数据库？待研究
    在这里插入图片描述

dolphinscheduler 2.0.5 告警组件-HTTP

告警组件-HTTP试用及改造

原网站

版权声明
本文为[韧小钊]所创，转载请带上原文链接，感谢
https://renxiaozhao.blog.csdn.net/article/details/124800905

边栏推荐

猜你喜欢

随机推荐