Prometheus operator configures promethesrule alarm rules

PrometheusRule

---

Used for configuration Prometheus Of Rule Rules file , Include recording rules and alerting, Can be automatically Prometheus load .

To configure PrometheusRule

---

Now we know how to customize one  ServiceMonitor  Object , But what if you need to customize an alarm rule ？ Let's go check Prometheus Dashboard Of Alert There are many alarm rules at the bottom of the page , This series of rules actually come from the project  GitHub - kubernetes-monitoring/kubernetes-mixin: A set of Grafana dashboards and Prometheus alerts for Kubernetes., We all go through Prometheus Operator Installation configuration .

But where did these alarm messages come from ？ How should they inform us ？ We know that we can use custom methods in Prometheus Specified in the configuration file of AlertManager Instance and alarm rules file , Now let's go through Operator Deployed ？ We can do it in Prometheus Dashboard Of Config At the bottom of the page, see about AlertManager Configuration of ：

alerting:
  alert_relabel_configs:
    - separator: ;
      regex: prometheus_replica
      replacement:  *1
      action: labeldrop
  alertmanagers:
    - follow_redirects: true
      enable_http2: true
      scheme: http
      path_prefix: /
      timeout: 10s
      api_version: v2
      relabel_configs:
        - source_labels: [__meta_kubernetes_service_name]
          separator: ;
          regex: alertmanager-main
          replacement:  *1
          action: keep
        - source_labels: [__meta_kubernetes_endpoint_port_name]
          separator: ;
          regex: web
          replacement:  *1
          action: keep
      kubernetes_sd_configs:
        - role: endpoints
          kubeconfig_file: ""
          follow_redirects: true
          enable_http2: true
          namespaces:
            own_namespace: false
            names:
              - monitoring
rule_files:
  - /etc/prometheus/rules/prometheus-k8s-rulefiles-0/*.yaml

above  alertmanagers  We can see that the configuration of is through role by  endpoints  Of kubernetes Obtained by the automatic discovery mechanism of , The matching service name is  alertmanager-main, The port name is  web  Of Service service , We can check  alertmanager-main  This Service：

 * kubectl describe svc alertmanager-main -n monitoring
Name:                     alertmanager-main
Namespace:                monitoring
Labels:                   app.kubernetes.io/component=alert-router
                          app.kubernetes.io/instance=main
                          app.kubernetes.io/name=alertmanager
                          app.kubernetes.io/part-of=kube-prometheus
                          app.kubernetes.io/version=0.24.0
Annotations:              <none>
Selector:                 app.kubernetes.io/component=alert-router,app.kubernetes.io/instance=main,app.kubernetes.io/name=alertmanager,app.kubernetes.io/part-of=kube-prometheus
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.109.67.21
IPs:                      10.109.67.21
Port:                     web  9093/TCP
TargetPort:               web/TCP
NodePort:                 web  32033/TCP
Endpoints:                10.244.1.193:9093,10.244.2.208:9093,10.244.2.210:9093
Port:                     reloader-web  8080/TCP
TargetPort:               reloader-web/TCP
NodePort:                 reloader-web  30181/TCP
Endpoints:                10.244.1.193:8080,10.244.2.208:8080,10.244.2.210:8080
Session Affinity:         ClientIP
External Traffic Policy:  Cluster
Events:                   <none>

You can see that the service name is  alertmanager-main,Port The name of the definition is also  web, According to the above rules , therefore Prometheus and AlertManager The components are correctly associated .

The corresponding alarm rule file is located in ：/etc/prometheus/rules/prometheus-k8s-rulefiles-0/  All the YAML file . We can enter Prometheus Of Pod Verify whether there is any under the directory YAML file ：

 * kubectl exec -it prometheus-k8s-0 /bin/sh -n monitoringkubectl exec -it prometheus-k8s-0 -n monitoring -- /bin/sh
/prometheus  * ls /etc/prometheus/rules/prometheus-k8s-rulefiles-0/
monitoring-alertmanager-main-rules-79543974-2f8e-4c5f-9d23-2c349c38ff1d.yaml
monitoring-grafana-rules-8fc5e057-099e-4546-b6bd-d8fb1107c24d.yaml
monitoring-kube-prometheus-rules-79b18777-2df4-4e43-84a8-193053400842.yaml
monitoring-kube-state-metrics-rules-8341740e-f2b7-48e9-82c2-bd6b979f1da2.yaml
monitoring-kubernetes-monitoring-rules-4b169784-b211-4449-922f-52fb2efd839c.yaml
monitoring-node-exporter-rules-b5f0f4d3-aa18-4e7d-836f-ef0a8fda7569.yaml
monitoring-prometheus-k8s-prometheus-rules-9560ae4f-764c-4ba4-9a37-2fedb56773c7.yaml
monitoring-prometheus-operator-rules-7d3a1645-efe3-4214-b825-c77c39ceb0d4.yaml
/prometheus  * cat /etc/prometheus/rules/prometheus-k8s-rulefiles-0/monitoring-kube-prometheus-rules-79b18777-2df4-4e43-84a8-193053400842.yaml
groups:
- name: general.rules
  rules:
  - alert: TargetDown
    annotations:
      description: '{
   { printf "%.4g"  *value }}% of the {
   {  *labels.job }}/{
   {  *labels.service
        }} targets in {
   {  *labels.namespace }} namespace are down.'
      runbook_url: https://runbooks.prometheus-operator.dev/runbooks/general/targetdown
      summary: One or more targets are unreachable.
    expr: 100 * (count(up == 0) BY (job, namespace, service) / count(up) BY (job,
      namespace, service)) > 10
    for: 10m
    labels:
      severity: warning
......

This YAML The file is actually the one we created earlier  PrometheusRule  What the file contains ：

 * cat kubePrometheus-prometheusRule.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-prometheus
    app.kubernetes.io/part-of: kube-prometheus
    prometheus: k8s
    role: alert-rules
  name: kube-prometheus-rules
  namespace: monitoring
spec:
  groups:
  - name: general.rules
    rules:
    - alert: TargetDown
      annotations:
        description: '{
   { printf "%.4g"  *value }}% of the {
   {  *labels.job }}/{
   {  *labels.service
          }} targets in {
   {  *labels.namespace }} namespace are down.'
        runbook_url: https://runbooks.prometheus-operator.dev/runbooks/general/targetdown
        summary: One or more targets are unreachable.
      expr: 100 * (count(up == 0) BY (job, namespace, service) / count(up) BY (job,
        namespace, service)) > 10
      for: 10m
      labels:
        severity: warning
......

We have  PrometheusRule  Of name by  kube-prometheus-rules,namespace by  monitoring, We can guess that we created a PrometheusRule After the resource object , It will automatically be on top of  prometheus-k8s-rulefiles-0  Under the directory, generate a corresponding  <namespace>-<name>-<xxx-id>.yaml  file , So if we need to customize an alarm option in the future , I just need to define one  PrometheusRule  The resource object can be .

As for why Prometheus Be able to recognize this PrometheusRule What about resource objects ？ This requires us to check the prometheus This resource object , There is a very important attribute  ruleSelector, To match rule Regular filters , We have no filter here , So it can match all , Assume that the matching is required to have  prometheus=k8s  and  role=alert-rules  Labeled  PrometheusRule  Resource objects , You can add the following configuration ：

ruleSelector:
  matchLabels:
    prometheus: k8s
    role: alert-rules

So we want to customize an alarm rule , Just create one that can be prometheus Object matching  PrometheusRule  Object can , For example, now we add a etcd Whether the alarm is available , We know etcd If more than half of the nodes in the whole cluster are available, the cluster is available , So we judge if it is not available etcd If the quantity exceeds half, the alarm will be triggered , create a file  prometheus-etcdRules.yaml：

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name: etcd-rules
  namespace: monitoring
spec:
  groups:
    - name: etcd
      rules:
        - alert: EtcdClusterUnavailable
          annotations:
            summary: etcd cluster small
            description: If one more etcd peer goes down the cluster will be unavailable
          expr: |
            count(up{job="etcd"} == 0) > (count(up{job="etcd"}) / 2 - 1)
          for: 3m
          labels:
            severity: critical

After creation , I'll check in the container later rules Folder ：

 * kubectl apply -f https://p8s.io/docs/operator/manifests/prometheus-etcdRules.yaml
prometheusrule.monitoring.coreos.com/etcd-rules created
 * kubectl exec -it prometheus-k8s-0 /bin/sh -n monitoring
Defaulting container name to prometheus.
Use 'kubectl describe pod/prometheus-k8s-0 -n monitoring' to see all of the containers in this pod.
/prometheus  * ls /etc/prometheus/rules/prometheus-k8s-rulefiles-0/
monitoring-etcd-rules.yaml            monitoring-prometheus-k8s-rules.yaml

You can see what we created rule The file has been injected into the corresponding  rulefiles  Under the folder , Prove that our above assumption is correct . Then go again. Prometheus Dashboard Of Alert At the bottom of the page, you can see the newly created alarm rules above ：