Detailed summary of reasons why alertmanager fails to send alarm messages at specified intervals / irregularly

Great simplicity , Up like water

Use prometheus+alertmanager Children's shoes that monitor the development of alarm schemes have more or less similar situations , The problem with the greatest probability of occurrence is that the alarm messages are not sent according to their own set sending frequency ？ Why send two at a time ？ Why do I always send them repeatedly ？ Why sending alarm messages is irregular ？...

There are many comprehensive factors that cause this problem , More complicated , The following has been practiced for a long time , Summarize the correct posture and share with you .

Let's start with a basic but real influencing factor with the lowest probability ： The Internet .

Whether it's prometheus----->alertmanager, still alertmanager-----> Message receiver service , Occasionally there will be network delays , But the general network is not so bad , If you really think it is a network problem , Sure curl Of the following components api, See if the interaction is slow .

In addition, keep in mind when verifying your changes , After changing any profile parameters reload Hot load and let the component use the latest value , Otherwise, it may be a correct method after a long time of verification, but it is still an incorrect phenomenon .

curl -XPOST localhost:9090/-/reload
curl -XPOST localhost:9093/-/reload

The following is a detailed explanation of the reasons for various situations starting from several phenomena .（ The following scenario solutions apply to all message channels, including mail