Document new alerting rule format.
@@ -11,31 +11,36 @@ to an external service. Whenever the alert expression results in one or more
vector elements at a given point in time, the alert counts as active for these
elements' label sets.
-Alerting rules are configured in Prometheus in the same way as [recording
-rules](recording_rules.md).
-
### Defining alerting rules
-Alerting rules are defined in the following syntax:
-
- ALERT <alert name>
- IF <expression>
- [ FOR <duration> ]
- [ LABELS <label set> ]
- [ ANNOTATIONS <label set> ]
-
-The alert name must be a valid metric name.
+Alerting rules are configured in Prometheus in the same way as [recording
+rules](recording_rules.md).
-The optional `FOR` clause causes Prometheus to wait for a certain duration
+An example rules file with an alert would be:
+
+```yaml
+groups:
+- name: example
+ rules:
+ - alert: HighErrorRate
+ expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5
+ for: 10m
+ labels:
+ severity: page
+ annotations:
+ summary: High request latency
+```
+
+The optional `for` clause causes Prometheus to wait for a certain duration
between first encountering a new expression output vector element (like an
instance with a high HTTP error rate) and counting an alert as firing for this
element. Elements that are active, but not firing yet, are in pending state.
-The `LABELS` clause allows specifying a set of additional labels to be attached
+The `labels` clause allows specifying a set of additional labels to be attached
to the alert. Any existing conflicting labels will be overwritten. The label
values can be templated.
-The `ANNOTATIONS` clause specifies another set of labels that are not
+The `annotations` clause specifies another set of labels that are not
identifying for an alert instance. They are used to store longer additional
information such as alert descriptions or runbook links. The annotation values
can be templated.
@@ -53,24 +58,29 @@ and `$value` holds the evaluated value of an alert instance.
Examples:
+```yaml
+groups:
+- name: example
+ rules:
+
# Alert for any instance that is unreachable for >5 minutes.
- ALERT InstanceDown
- IF up == 0
- FOR 5m
- LABELS { severity = "page" }
- ANNOTATIONS {
- summary = "Instance {{ $labels.instance }} down",
- description = "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.",
- }
-
- # Alert for any instance that have a median request latency >1s.
- ALERT APIHighRequestLatency
- IF api_http_request_latencies_second{quantile="0.5"} > 1
- FOR 1m
- ANNOTATIONS {
- summary = "High request latency on {{ $labels.instance }}",
- description = "{{ $labels.instance }} has a median request latency above 1s (current value: {{ $value }}s)",
- }
+ - alert: InstanceDown
+ expr: up == 0
+ for: 5m
+ labels:
+ severity: page
+ annotations:
+ summary: "Instance {{ $labels.instance }} down"
+ description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."
+
+ # Alert for any instance that has a median request latency >1s.
+ - alert: APIHighRequestLatency
+ expr: api_http_request_latencies_second{quantile="0.5"} > 1
+ for: 10m
+ annotations:
+ summary: "High request latency on {{ $labels.instance }}"
+ description: "{{ $labels.instance }} has a median request latency above 1s (current value: {{ $value }}s)"
+```
### Inspecting alerts during runtime
