Grafana templates allow generating custom alert messages based on the alert details. However by default the firing alert and resolved messages are the same, typically taking from the summary annotation. This can make for resolved alerts looking at first glance like they’re still a problem.
To make it easier to distinguish between firing and resolved alerts you can use a custom annotation that only shows for resolved alerts.
Let’s say we have an alert that fires on over 80% CPU, the annotations could look like this.
The “Summary” is the message to send when firing.
The custom “Resolution” message is to be sent when the alert is resolved.
The message template might look like this. If the alert has a resolution annotation, use that when resolved, otherwise fallback to summary.
{{- if .Alerts.Firing }}
{{- len .Alerts.Firing }} Firing
{{- range .Alerts.Firing }}
🔥 {{ .Annotations.summary }}
{{- end }}
{{- end }}
{{ if .Alerts.Resolved }}
{{- len .Alerts.Resolved }} Resolved
{{- range .Alerts.Resolved }}
👍 {{ if .Annotations.Resolution }}{{- .Annotations.Resolution }}{{else}}{{ .Annotations.summary }}{{ end }}
{{- end }}
{{- end }}Given the above, 2 firing alerts and 1 resolved we would get the following message.
2 Firing
🔥 Core 0,0 is above 80%.
🔥 Core 0,1 is above 80%.
1 Resolved
👍 Core 0,2 has reduced to normal.