[hazdl]

Analysis PlatformAP-1372

Summary

• Add an alert for high P95 workflow-template runtime.
• Add an alert for high P95 workflow queue latency.
• Add an alert for high P95 workflow queue duration.
• Include workflow template, namespace, and queue labels in alert descriptions where applicable.
• Align alert labels with existing warning alerts.

Validation

• Helm template rendering: Rendered the monitoring Helm chart successfully.
• Prometheus rule validation: Validated the extracted alert rules using promtool check rules.
• Alert thresholds and metric units: Reviewed the alert expressions, thresholds, and metric units against the relevant metric definitions.

[hazdl]

The metric is reported in seconds, so the existing threshold of 60 equates to 60 seconds. I've updated the threshold to 3600 (60 minutes) and adjusted the alert description accordingly.

[hazdl]

Hi @JamesDoingStuff , could you please take another look at this PR when you get a chance? I’ve added a comment addressing the issue, but the PR is still blocked. Let me know if anything else is needed from my side. Thanks!

[JamesDoingStuff]

It looks like these alerts got a bit mixed up with the scraping ones when you rebased - please could you go through and make sure they're correct? Also, the Confirm the metric unit is seconds. lines can probably be done away with now

[hazdl]

@JamesDoingStuff Try now please.

[JamesDoingStuff]

This is a bit pedantic on my part but could we have a bit of consistency with the commit scope please (the bit in brackets)? Three different scopes for one feature is a bit confusing. I'd suggest just monitoring