Kubernetes deploys a secret pit where thanos ruler sends repeated alarms

1 summary ：

1.1 Environmental Science

thanos ruler and alertmanager All deployed in kubernetes colony , The version information is as follows ：
a、kubernetes colony ：v1.18.5
b、thanos ruler: v0.11.0
c、alertmanager: v0.20.0

thanos ruler Of yaml File info ：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/name: thanos-rule
  name: thanos-rule
  namespace: monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: thanos-rule
  serviceName: thanos-rules
  template:
    metadata:
      labels:
        app.kubernetes.io/name: thanos-rule
    spec:
      containers:
      - image: registry.cn-shenzhen.aliyuncs.com/gzlj/thanos-reloader:v0.1
        imagePullPolicy: Always
        name: reloader
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - args:
        - rule
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --rule-file=/etc/thanos/rules/*rules.yaml
        - --data-dir=/var/thanos/rule
        - --label=rule_replica="$(NAME)"
        # Please note that --alert.label-drop This line of record , Value is band ""
        - --alert.label-drop="rule_replica"
        - --query=dnssrv+_http._tcp.thanos-query.monitoring.svc.cluster.local
        - --alertmanagers.url=http://alertmanager-main.monitoring.svc.cluster.local:9093
        env:
        - name: NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        image: quay.mirrors.ustc.edu.cn/thanos/thanos:v0.11.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 24
          httpGet:
            path: /-/healthy
            port: 10902
            scheme: HTTP
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        name: thanos-rule
        ports:
        - containerPort: 10901
          name: grpc
          protocol: TCP
        - containerPort: 10902
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 18
          httpGet:
            path: /-/ready
            port: 10902
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        volumeMounts:
        - mountPath: /var/thanos/rule
          name: data
        - mountPath: /etc/thanos/rules
          name: thanos-rules
      restartPolicy: Always
      serviceAccount: thanos-rules
      serviceAccountName: thanos-rules
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: thanos-rules
        name: thanos-rules
      - emptyDir: {}
        name: data

The key screenshots are as follows

1.2 The phenomenon

alertmanager Repeated alarm received , The only difference between the two repeated alarms is the custom tag rule_replica The values are different , As shown in the figure ：

2 Solution

Tried to change to thanos ruler The mirror version of （v0.15.0）, But the phenomenon remains .
About to give up , I put thanos ruler Start command parameters --alert.label-drop="rule_replica" become --alert.label-drop=rule_replica, That is, just remove the double quotation marks ,alertmanager The phenomenon of repeatedly receiving alarms is solved .

3 The phenomenon after solution

thanos ruler Label the alarm information rule_replica Throw away , Then send the alarm to alertmanager, here alertmanager There is only one alarm message in the , Instead of the previous two .