XeLabs/awesome-prometheus-alerts
2
0
Fork 0
mirror of https://github.com/samber/awesome-prometheus-alerts.git synced 2026-06-26 19:37:27 +08:00
Code Issues Projects Releases Packages Wiki Activity

Update rules.yml

Browse source 
Added reasonable time periods for each query to avoid false positives and in some cased give the system a short window to try to solve the issue.
Also changed the severity level of authentication alerts from critical to info which seems more appropriate
This commit is contained in:
Ali 2024-05-13 11:27:44 +03:30 committed by GitHub
parent 551a0a129e
commit 97646e5d14
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 13 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

17
_data/rules.yml
View file

@ -1310,10 +1310,12 @@ groups:
description: Memory usage is critically high, over 90%. description: Memory usage is critically high, over 90%.
query: "ClickHouseAsyncMetrics_CGroupMemoryUsed / ClickHouseAsyncMetrics_CGroupMemoryTotal * 100 > 90" query: "ClickHouseAsyncMetrics_CGroupMemoryUsed / ClickHouseAsyncMetrics_CGroupMemoryTotal * 100 > 90"
severity: critical severity: critical
for: 5m
- name: ClickHouse Memory Usage Warning - name: ClickHouse Memory Usage Warning
description: Memory usage is over 80%. description: Memory usage is over 80%.
query: "ClickHouseAsyncMetrics_CGroupMemoryUsed / ClickHouseAsyncMetrics_CGroupMemoryTotal * 100 > 80" query: "ClickHouseAsyncMetrics_CGroupMemoryUsed / ClickHouseAsyncMetrics_CGroupMemoryTotal * 100 > 80"
severity: warning severity: warning
for: 5m
- name: ClickHouse Disk Space Low on Default - name: ClickHouse Disk Space Low on Default
description: Disk space on default is below 20%. description: Disk space on default is below 20%.
query: "ClickHouseAsyncMetrics_DiskAvailable_default / (ClickHouseAsyncMetrics_DiskAvailable_default + ClickHouseAsyncMetrics_DiskUsed_default) * 100 < 20" query: "ClickHouseAsyncMetrics_DiskAvailable_default / (ClickHouseAsyncMetrics_DiskAvailable_default + ClickHouseAsyncMetrics_DiskUsed_default) * 100 < 20"
@ -1330,42 +1332,49 @@ groups:
description: Critical replica errors detected, either all replicas are stale or lost. description: Critical replica errors detected, either all replicas are stale or lost.
query: "ClickHouseErrorMetric_ALL_REPLICAS_ARE_STALE == 1 or ClickHouseErrorMetric_ALL_REPLICAS_LOST == 1" query: "ClickHouseErrorMetric_ALL_REPLICAS_ARE_STALE == 1 or ClickHouseErrorMetric_ALL_REPLICAS_LOST == 1"
severity: critical severity: critical
for: 3m
- name: ClickHouse No Available Replicas - name: ClickHouse No Available Replicas
description: No available replicas in ClickHouse. description: No available replicas in ClickHouse.
query: "ClickHouseErrorMetric_NO_AVAILABLE_REPLICA == 1" query: "ClickHouseErrorMetric_NO_AVAILABLE_REPLICA == 1"
severity: critical severity: critical
for: 3m
- name: ClickHouse No Live Replicas - name: ClickHouse No Live Replicas
description: There are too few live replicas available, risking data loss and service disruption. description: There are too few live replicas available, risking data loss and service disruption.
query: "ClickHouseErrorMetric_TOO_FEW_LIVE_REPLICAS == 1" query: "ClickHouseErrorMetric_TOO_FEW_LIVE_REPLICAS == 1"
severity: critical severity: critical
for: 3m
- name: ClickHouse High Network Traffic - name: ClickHouse High Network Traffic
description: Network traffic is unusually high, may affect cluster performance. description: Network traffic is unusually high, may affect cluster performance.
query: "ClickHouseMetrics_NetworkSend > 1000 or ClickHouseMetrics_NetworkReceive > 1000" query: "ClickHouseMetrics_NetworkSend > 250 or ClickHouseMetrics_NetworkReceive > 250"
severity: warning severity: warning
for: 3m
comments: | comments: |
Please replace the threshold with an appropriate value Please replace the threshold with an appropriate value
- name: ClickHouse High TCP Connections - name: ClickHouse High TCP Connections
description: High number of TCP connections, indicating heavy client or inter-cluster communication. description: High number of TCP connections, indicating heavy client or inter-cluster communication.
query: "ClickHouseMetrics_TCPConnection > 1500" query: "ClickHouseMetrics_TCPConnection > 400"
severity: warning severity: warning
for: 3m
comments: | comments: |
Please replace the threshold with an appropriate value Please replace the threshold with an appropriate value
- name: ClickHouse Interserver Connection Issues - name: ClickHouse Interserver Connection Issues
description: An increase in interserver connections may indicate replication or distributed query handling issues. description: An increase in interserver connections may indicate replication or distributed query handling issues.
query: "increase(ClickHouseMetrics_InterserverConnection[5m]) > 0" query: "increase(ClickHouseMetrics_InterserverConnection[5m]) > 0"
severity: warning severity: warning
for: 1m
- name: ClickHouse ZooKeeper Connection Issues - name: ClickHouse ZooKeeper Connection Issues
description: ClickHouse is experiencing issues with ZooKeeper connections, which may affect cluster state and coordination. description: ClickHouse is experiencing issues with ZooKeeper connections, which may affect cluster state and coordination.
query: "avg(ClickHouseMetrics_ZooKeeperSession) != 1" query: "avg(ClickHouseMetrics_ZooKeeperSession) != 1"
severity: warning severity: warning
for: 1m
- name: ClickHouse Authentication Failures - name: ClickHouse Authentication Failures
description: Authentication failures detected, indicating potential security issues or misconfiguration. description: Authentication failures detected, indicating potential security issues or misconfiguration.
query: "increase(ClickHouseErrorMetric_AUTHENTICATION_FAILED[5m]) > 0" query: "increase(ClickHouseErrorMetric_AUTHENTICATION_FAILED[5m]) > 0"
severity: critical severity: info
- name: ClickHouse Access Denied Errors - name: ClickHouse Access Denied Errors
description: Access denied errors have been logged, which could indicate permission issues or unauthorized access attempts. description: Access denied errors have been logged, which could indicate permission issues or unauthorized access attempts.
query: "increase(ClickHouseErrorMetric_RESOURCE_ACCESS_DENIED[5m]) > 0" query: "increase(ClickHouseErrorMetric_RESOURCE_ACCESS_DENIED[5m]) > 0"
severity: critical severity: info
- name: Zookeeper - name: Zookeeper