From 8cb72ad32e59fbd721577a8b8e86448325c3e61a Mon Sep 17 00:00:00 2001 From: Jan Gosmann Date: Wed, 25 Jan 2023 19:52:50 +0100 Subject: [PATCH] Make ElasticsearchNoNewDocuments alert more robust Use `elasticsearch_indices_indexing_index_total` instead of `elasticsearch_indices_docs` because `elasticsearch_indices_docs` might not update without an index refresh [1]. Refreshes happen every second by default, *but* only if there have been search requests within the last 30 seconds [2]. If there are no search requests for a sufficiently long duration, the alert based on `elasticsearch_indices_docs` will fire mistakenly. Apart from that, `elasticsearch_indices_docs` has the gauge metric type (while `elasticsearch_indices_indexing_index_total` is of the counter type) and the `increase` function is not intended to be used with gauges. Drops in the document count would be treated as a reset to 0, thus showing an increase by all remaining documents. [1]: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html#index-stats-api-path-params [2]: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html --- _data/rules.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data/rules.yml b/_data/rules.yml index 893d3ab..124c9c4 100644 --- a/_data/rules.yml +++ b/_data/rules.yml @@ -1054,7 +1054,7 @@ groups: for: 15m - name: Elasticsearch no new documents description: No new documents for 10 min! - query: 'increase(elasticsearch_indices_docs{es_data_node="true"}[10m]) < 1' + query: 'increase(elasticsearch_indices_indexing_index_total{es_data_node="true"}[10m]) < 1' severity: warning - name: Cassandra