# Awesome Prometheus alerting rules

(WIP)

## Todo

- Write full alert rules in yml files
- Make a small website with form for each rule, to build custom alerts (criticity, thresolds, instance...)

## Queries

### Prometheus internal

- `up == 0` // killed exporters

### node-exporter

Memory:

- `(node_memory_MemFree{} + node_memory_Cached{} + node_memory_Buffers{}) / node_memory_MemTotal{} * 100 < 5`

Network:

- `sum by (instance) (irate(node_network_transmit_bytes{}[2m])) / 1024 / 1024 > 100`
- `sum by (instance) (irate(node_network_receive_bytes{}[2m])) / 1024 / 1024 > 100`

Disk:

- `sum by (instance) (irate(node_disk_bytes_read{}[2m])) / 1024 / 1024 > 50`
- `sum by (instance) (irate(node_disk_bytes_written{}[2m])) / 1024 / 1024 > 50`
- `node_filesystem_free{mountpoint ="/rootfs"} / node_filesystem_size{mountpoint ="/rootfs"} * 100 < 10` // gb
- `node_filesystem_files_free{mountpoint ="/rootfs"} / node_filesystem_files{mountpoint ="/rootfs"} * 100` // inodes
- `rate(node_disk_read_time_ms{}[1m]) / rate(node_disk_reads_completed{}[1m]) > 100` // too much latency
- `rate(node_disk_write_time_ms{}[1m]) / rate(node_disk_writes_completed{}[1m]) > 100` // too much latency

CPU:

- `avg by (instance) (sum by (cpu) (rate(node_cpu{mode!="idle"}[2m]))) * 100 > 75`  // load
- `rate(node_context_switches{}[5m]) > 1000` // nbr context switch per second

### cAdvisor

- `time() - container_last_seen{} > 60` // get killed container

### Nginx

- `rate(nginx_http_requests_total{status=~"^4.."}[1m]) > 10`  // get 4xx http requests
- `rate(nginx_http_requests_total{status=~"^5.."}[1m]) > 10`  // get 5xx http requests

### Rabbitmq (kbudde/rabbitmq-exporter)

- `rabbitmq_up{} == 0`
- `rabbitmq_running{} >= 2`   // cluster
- `rabbitmq_partitions{} > 0` // cluster got partition :-(
- `rabbitmq_node_mem_used{} / rabbitmq_node_mem_limit{} * 100 > 90`   // too much ram used
- `rabbitmq_connectionsTotal{} > 1000`

- `rabbitmq_queue_messages_unacknowledged{queue="my-queue"} > 5`
- `rabbitmq_queue_messages_ready{queue="my-queue"} > 1000` // more consumers needed
- `time() - rabbitmq_queue_head_message_timestamp{queue="my-queue"} > 60` // takes more than 1min to consume messages
- `rabbitmq_queue_consumers{} == 0`  // no consumer on queue
- `rate(rabbitmq_exchange_messages_published_in_total{exchange="my-exchange"}[1m]) < 5`  // no activity on exchange

### PostgreSQL (wrouesnel/postgres_exporter)

- `pg_up{} == 0`
- `pg_replication_lag{} > 10` // more than 10s lag between master and slave
- `time() - pg_stat_user_tables_last_autovacuum{} > 60 * 60 * 24` // did not vaccum for 1 day
- `time() - pg_stat_user_tables_last_autoanalyze{} > 60 * 60 * 24` // did not analyse for 1 day
- `sum by (datname) (pg_stat_activity_count{datname!~"template.*|postgres"}) > 100` // too many connections
- `sum by (datname) (pg_stat_activity_count{datname!~"template.*|postgres"}) < 5` // connections number too small
- `rate(pg_stat_database_deadlocks{pg_stat_database_de}[1m]) > 0`

### Redis (oliver006/redis_exporter)

- `redis_up{} == 0`
- `time() - redis_rdb_last_save_timestamp_seconds{} > 60 * 60 * 24` // did not backup for 1 day
- `redis_memory_used_bytes{} / redis_total_system_memory_bytes{} * 100 > 90`
- `redis_connected_slaves{}`
- `delta(redis_connected_slaves{}[1m]) < 0`  // slaved killed
- `redis_connected_clients{} > 100` // too many connections
- `redis_connected_clients{} < 5` // connections number too small
- `increase(redis_rejected_connections_total{}[1m]) > 0` // rejected connections

### MySQL

### Elasticsearch

### MongoDB

### Apache

### HaProxy

### Traefik

### PHP-FPM

### Kubernetes

### Nomad

### Consul

### Etcd

### Zookeeper

### Linkerd

### Istio

### Blackbox