Promtool Check

Promtool  yes  Prometheus  An operation and maintenance tool provided , Used to correct  Prometheus  Inspection , Before, there was only the simplest function to check the configuration file and rule file , Now there are more new functions , Including query indicators 、debug  service 、 Check the database and so on . This time we're based on  2.37.0  The version is combed , If the version is lower than this version , Please confirm whether the functions to be used have been added before using , In case of version ratio  2.37.0  high , Please check whether the relevant functions are updated .

see  Promtool  The following information can be seen in the help information of the tool ：


[[email protected] ~]# ./promtool --help
usage: promtool [<flags>] <command> [<args> ...]

Tooling for the Prometheus monitoring system.

Flags:
 -h, --help Show context-sensitive help (also try --help-long and
 --help-man).
 --version Show application version.
 --enable-feature= ... Comma separated feature names to enable (only PromQL
 related). See
 https://prometheus.io/docs/prometheus/latest/feature_flags/
 for the options and more details.

Commands:
 help [<command>...]
 Show help.

 check service-discovery [<flags>] <config-file> <job>
 Perform service discovery for the given job name and report the results,
 including relabeling.

 check config [<flags>] <config-files>...
 Check if the config files are valid or not.

 check web-config <web-config-files>...
 Check if the web config files are valid or not.

 check rules [<flags>] <rule-files>...
 Check if the rule files are valid or not.

 check metrics
 Pass Prometheus metrics over stdin to lint them for consistency and
 correctness.

 examples:

 $ cat metrics.prom | promtool check metrics

 $ curl -s http://localhost:9090/metrics | promtool check metrics

 query instant [<flags>] <server> <expr>
 Run instant query.

 query range [<flags>] <server> <expr>
 Run range query.

 query series --match=MATCH [<flags>] <server>
 Run series query.

 query labels [<flags>] <server> <name>
 Run labels query.

 debug pprof <server>
 Fetch profiling debug information.

 debug metrics <server>
 Fetch metrics debug information.

 debug all <server>
 Fetch all debug information.

 test rules <test-rule-file>...
 Unit tests for rules.

 tsdb bench write [<flags>] [<file>]
 Run a write performance benchmark.

 tsdb analyze [<flags>] [<db path>] [<block id>]
 Analyze churn, label pair cardinality and compaction efficiency.

 tsdb list [<flags>] [<db path>]
 List tsdb blocks.

 tsdb dump [<flags>] [<db path>]
 Dump samples from a TSDB.

 tsdb create-blocks-from openmetrics <input file> [<output directory>]
 Import samples from OpenMetrics input and produce TSDB blocks. Please refer to
 the storage docs for more details.

 tsdb create-blocks-from rules --start=START [<flags>] <rule-files>...
 Create blocks of data for new recording rules.

[[email protected] ~]#


essential information

This tool has some basic information to use , Some parameters are basically general .

First   Use  

--help

  You can see help information , You can also use  

--help-long

  See the help information of the extended version for parameters

Use  

--version

  You can see the tool version information 、 Build branches and  commit ID、 Build time 、Go  Version of 、 Run the information of the platform .

[[email protected] ~]# ./promtool --version
promtool, version 2.37.0 (branch: HEAD, revision: b41e0750abf5cc18d8233161560731de05199330)
 build user: [email protected]
 build date: 20220714-15:13:18
 go version: go1.18.4
 platform: linux/amd64


Use  Promtool  You can also verify the functions of some new features , Use  

--enable-feature=

  Parameters to turn on , Multiple parameters are separated by commas . Relevant features need reference  https://prometheus.io/docs/prometheus/latest/feature_flags/  link .

Use  Promtool  Inspection

Check the service and find

Use  Promtool  You can check the service discovery configuration in the configuration file , Prevent the program from starting due to configuration errors , Or the content cannot be found normally after startup .

The specific command parameters are

./promtool check service-discovery [<flags>] <config-file> <job>


You need to specify  Prometheus  Configuration file for , And the services to be checked  job  name .Promtool  The inspection result will be returned after the inspection , And will include  relable  The content of .

The experimental results are as follows , This command returns all the information that can be found , Even if  job  As long as a specific name is found  pod  that will do .

./promtool check service-discovery /erdong/prometheus-pushgateway/conf/prometheus.yml pushgateway-k8s 
[
 {

 ......

 "discoveredLabels": {
 "__address__": "172.31.91.38:9091",
 "__meta_kubernetes_namespace": "appops",
 "__meta_kubernetes_pod_annotation_cattle_io_timestamp": "2022-07-22T09:40:36Z",
 "__meta_kubernetes_pod_annotation_cni_projectcalico_org_podIP": "172.31.91.38/32",
 "__meta_kubernetes_pod_annotation_field_cattle_io_ports": "[[{\"containerPort\":9091,\"dnsName\":\"pushgateway-nodeport\",\"hostPort\":0,\"kind\":\"NodePort\",\"name\":\"pushgateway\",\"protocol\":\"TCP\",\"sourcePort\":9091}]]",
 "__meta_kubernetes_pod_annotationpresent_cattle_io_timestamp": "true",
 "__meta_kubernetes_pod_annotationpresent_cni_projectcalico_org_podIP": "true",
 "__meta_kubernetes_pod_annotationpresent_field_cattle_io_ports": "true",
 "__meta_kubernetes_pod_container_init": "false",
 "__meta_kubernetes_pod_container_name": "pushgateway",
 "__meta_kubernetes_pod_container_port_name": "pushgateway",
 "__meta_kubernetes_pod_container_port_number": "9091",
 "__meta_kubernetes_pod_container_port_protocol": "TCP",
 "__meta_kubernetes_pod_controller_kind": "ReplicaSet",
 "__meta_kubernetes_pod_controller_name": "pushgateway-6d695f75cf",
 "__meta_kubernetes_pod_host_ip": "172.30.7.229",
 "__meta_kubernetes_pod_ip": "172.31.91.38",
 "__meta_kubernetes_pod_label_pod_template_hash": "6d695f75cf",
 "__meta_kubernetes_pod_label_workload_user_cattle_io_workloadselector": "pushgateway",
 "__meta_kubernetes_pod_labelpresent_pod_template_hash": "true",
 "__meta_kubernetes_pod_labelpresent_workload_user_cattle_io_workloadselector": "true",
 "__meta_kubernetes_pod_name": "pushgateway-flink-6d695f75cf-9jgdk",
 "__meta_kubernetes_pod_node_name": "erdong-test-7-229",
 "__meta_kubernetes_pod_phase": "Running",
 "__meta_kubernetes_pod_ready": "true",
 "__meta_kubernetes_pod_uid": "7b9662d8-1831-4440-9bc5-9523dbda0c12",
 "__metrics_path__": "/metrics",
 "__scheme__": "http",
 "__scrape_interval__": "20s",
 "__scrape_timeout__": "10s",
 "job": "pushgateway-k8s"
 },

 ......
 "labels": {}
 }
] 


Check the configuration file

Use  Promtool  You can check  Prometheus  Whether the startup configuration file of is correct . The parameters are as follows ：

./promtool check config [<flags>] <config-files>...


Let's execute the order and take a look ：

./promtool check config /erdong/prometheus-pushgateway/conf/prometheus.yml
Checking /erdong/prometheus-pushgateway/conf/prometheus.yml
 SUCCESS: /erdong/prometheus-pushgateway/conf/prometheus.yml is valid prometheus config file syntax


This is the normal execution .

Check  web-config

Use  Promtool  You can check  web-config  The configuration file

./promtool check web-config <web-config-files>...


Check the alarm rules and recording rules

Use  Promtool  You can check the alarm rules and record rule files

./promtool check rules [<flags>] <rule-files>...


Inspection index

Use  Promtool  You can check the indicators

./promtool check metrics

 Pass Prometheus metrics over stdin to lint them for consistency and
 correctness.

 examples:

 $ cat metrics.prom | promtool check metrics

 $ curl -s http://localhost:9090/metrics | promtool check metrics