Alert templates

The name is misleading. The thing called an "alert template" is actually a multi-channel notification config — one row that can fan out to email, Slack, Teams, Discord, webhook, and SMS simultaneously, with a per-channel enabled toggle and per-channel body template.

The same table started life serving the asset alert engine, then quietly got reused by three more subsystems. This article unpacks the whole model so you know what you're looking at when the picker turns up in unexpected places.

Asset-alert templates are edited on the Alert delivery tab of Settings → Notifications (the standalone Alerts → Templates page still works and shows the same rows). Every org ships with a Default alerts template marked is_default — all channels disabled until you configure it.

What's actually in a template

One template row holds:

• Name — what consumers refer to it by
• Kind — asset_alert, ticket_notification, or custom (added 2026-05-22 to facet the editor)
• Email channel — recipients (list), subject template, body template, optional from-override
• Webhook channel — URL, custom headers, body template
• SMS channel — recipients, body template (SMS dispatch is stubbed today; the config persists for the future hook)
• Slack channel — incoming webhook URL, channel override, body template
• Teams channel — Workflows webhook URL, body template
• Discord channel — incoming webhook URL, body template

For Slack, Teams and Discord, leaving the webhook URL blank makes the template use the org-wide default webhook configured on the Channels tab of Settings → Notifications — set the URL once, point every template at it. The generic webhook channel has no org default; it always needs its own URL.

Teams webhooks changed in 2026. Microsoft permanently retired the classic Office 365 Connector incoming webhooks (webhook.office.com) in May 2026, and the first-generation Workflows URLs (logic.azure.com) were replaced in late 2025. The supported setup is a Teams Workflow: in the channel, click ... → Workflows → choose the "Send webhook alerts to a channel" template → confirm the team and channel → save → copy the webhook link (it lives under api.powerplatform.com). The alert card posts as the Flow bot — the bot name and icon can't be customised, and card action buttons don't render. OpsMerge rejects retired webhook.office.com URLs with an error telling you to recreate the workflow.

Every channel section in the editor has a Send test button (on saved templates) that pushes a synthetic test alert through the channel's saved configuration — the fastest way to confirm a webhook URL or recipient list actually works. The channel's enable toggle and severity filter are ignored for tests.

• is_default — at most one per org. Used as the org-wide fallback when nothing more specific resolves.
• system_key — set for the four PSA system templates. Null for everything else.

Every channel has its own enable toggle and its own minimum severity ("Send for" in the editor: all severities / warnings and above / errors only). A channel skips alerts below its threshold, so one template can email everything while only posting errors to Slack — no need for a second template. The default is all severities, matching the old behaviour.

The four consumers

Consumer	What fires it	How it picks a template
Alert engine (Monitoring → Alerts)	Check fails / agent goes offline / task fails	Resolves via the agent → site → client → org-default cascade
Monitoring workflows with create_alert action	check_failure / agent_status / event log / schedule / webhook_inbound	Inserts an alert row; the alert engine then resolves a template the normal way
PSA ticket rules with send_email action	ticket.created, ticket.commented, ticket.sla_warning, ticket.scheduled_stale, ticket.scheduled_customer_silence, ticket.updated	Looks up by name or system_key (kind = ticket_notification or custom)
PSA system notifications (the four [System] rows)	Auto-ack inbound email / public reply / assignee comment / watcher comment	Bound by system_key to a system-owned rule

The kind column (added 2026-05-22) is the facet that keeps these from leaking into each other's pickers. Pre-facet, an agent install dropdown could legitimately show ticket notification templates — which was the prompt to fix the IA.

Three template syntaxes (this is the awkward bit)

Asset-alert templates, ticket-notification templates and monitoring-workflow action params each render with a different variable set. They don't speak each other's language.

Asset-alert templates use Go's text/template syntax — double-brace, dot-prefix, PascalCase names supplied by the notification dispatcher:

Disk space low on {{.AgentHostname}} — {{.CheckName}} reported {{.Message}}

The full variable list for asset-alert kind: {{.AlertType}}, {{.Severity}}, {{.Status}}, {{.Message}}, {{.AgentHostname}}, {{.AgentOS}}, {{.AgentID}}, {{.AgentURL}}, {{.CheckName}}, {{.CheckType}}, {{.TaskName}}, {{.ConsecutiveFailures}}, {{.AlertID}}, {{.Timestamp}}. The editor's Template variables panel lists exactly these. A misspelt variable fails the whole send — missing keys are errors, not blanks — so copy from the list rather than guessing.

{{.AgentURL}} is a deep link to the affected device's page in the portal (your primary custom domain when one is active, else the platform URL; empty for alerts with no device). The built-in layouts already use it: the hostname in default emails and Slack/Teams/Discord cards is clickable, and the default webhook JSON carries agent.url.

Leaving a subject or body blank gives you the built-in layout for that channel (branded HTML for email, a structured JSON payload for webhooks, a rich alert card for Slack/Teams/Discord). A custom Slack/Teams/Discord body must render to the channel's JSON payload shape — Block Kit for Slack — not prose.

Monitoring-workflow action params (ticket titles and webhook payloads authored in the Monitoring → Workflows editor) are a different context again: flat tokens like {agent_name} / {{agent_name}}, documented in Workflows. They work in workflow actions only, not in alert templates.

Ticket-notification templates use Go's text/template and html/template syntax — double-brace, dot-prefix, with html kind auto-escaping attacker-controlled variables:

<p>Hi {{.requester.first_name}},</p>
<p>Your ticket <strong>#{{.ticket.number}}</strong> is now {{.ticket.status}}.</p>

Common variables for ticket_notification kind: {{.ticket.number}}, {{.ticket.subject}}, {{.ticket.priority}}, {{.ticket.status}}, {{.ticket.url}}, {{.client.name}}, {{.requester.first_name}}, {{.requester.last_name}}, {{.requester.email}}, {{.comment.body}}, {{.assignee.name}}, {{.watchers}}.

Copy-pasting from one to the other will leave literal {var} or {{.var}} strings in the rendered output. If you see that, it's a syntax mismatch, not a missing variable.

Resolution: how asset-alert templates get picked

When a check fails on agent X at site Y for client Z, the alert engine walks a cascade to decide which template to fire:

1. agents.alert_template_id      (most specific — set per agent)
2. sites.alert_template_id       (covers every agent at the site)
3. clients.alert_template_id     (covers every site/agent under the client)
4. alert_templates.is_default    (org-wide fallback)
5. nothing — no notification

The first match wins. This means you can have one template for production servers (HTML email plus Slack to #alerts), a different one for development environments (silent alert row, no notification), and only override at the agent level for the few endpoints that need different routing.

Four ways to assign an asset-alert template

Install token. When you generate an install token at Clients → site → Generate Install Token, you can optionally pick a template. The new agent registers with alert_template_id already set on its row. Bypasses the site/client cascade. Useful when you're deploying a batch of servers that need their own routing from day one.

Site editor. Clients → site → Alert Template field. Applies to every agent at that site that doesn't have its own.

Client editor. Clients → client → Alert Template field. Applies to every site and agent under that client that doesn't have a more specific override.

Org default. On the templates page, mark one template is_default = true. There can only be one. Used when nothing higher in the cascade resolves.

In practice most MSPs set one org-default ("everything goes to the noc inbox plus #alerts") and add per-client overrides only when a client demands something different.

Resolution: how ticket-notification templates get picked

No cascade. Ticket rules and system notifications reference templates by name or system_key. There's no "default ticket template" — every consumer is explicit about which template it wants.

For system notifications, the binding is fixed: auto_ack_email_ticket always uses the row with system_key = 'auto_ack_email_ticket', and so on.

For custom rules, the rule's send_email action carries a template_key param. You pick a template by name when authoring the rule. Renames break the binding; if you rename a template, update any rules that reference it.

Where each kind appears in the UI

Kind	Editor location	Picker locations
asset_alert	Alerts → Templates (/alerts/templates)	Install token dropdown, site Alert Template field, client Alert Template field
ticket_notification	PSA → Workflows → System Notifications (the four system rows) and PSA → Workflows → Notification Templates (custom ones)	PSA rule editor's send_email action
custom	PSA → Workflows → Notification Templates	PSA rule editor's send_email action

Templates with system_key set get a lock badge — you can edit subject and body freely, but delete is disabled.

Worked example: cascading templates for one MSP

Say you run an MSP with three clients. Most of them want failing-server alerts to land in your central NOC inbox. One client, Acme, has its own SRE team and wants their own Slack workspace pinged on their alerts. A third client, BoringCo, has dev hosts they don't want notifications about at all.

1. Create "NOC default" template at Alerts → Templates. Email channel only. Recipients = [email protected]. Mark is_default = true.
2. Create "Acme SRE" template. Email channel = Acme's ops list, plus Slack channel pointing at their #opsmerge-alerts. Set is_default = false.
3. Assign "Acme SRE" to the Acme client at Clients → Acme → Alert Template. Every Acme site and agent inherits.
4. Create "Silent" template. No channels enabled (just a placeholder so the cascade resolves to something).
5. Assign "Silent" to BoringCo's "dev" site at Clients → BoringCo → Dev → Alert Template. Production hosts at BoringCo still inherit the org default.

Now:

• A failing check on bigco-dc-01 falls through to the org default → NOC inbox gets emailed.
• A failing check on acme-prod-02 resolves at the client level → email to Acme's ops, plus Slack to their channel.
• A failing check on boringco-dev-04 resolves at the site level → silent.
• Override acme-prod-02 directly if you need to (e.g., a dedicated escalation template for a critical box).

Worked example: chaining alert template to ticket creation

The two engines work nicely in series:

1. Monitoring workflow at Monitoring → Workflows: check_failure with check_type = disk, action = create_ticket priority high, title template "Disk full on {agent_name}".
2. PSA ticket rule at PSA → Workflows → Ticket Rules: trigger ticket.created, condition category = infrastructure, action send_email with template_key = "Infra escalation".
3. Create the "Infra escalation" template at PSA → Workflows → Notification Templates with kind = ticket_notification, subject {{.ticket_ref}} {{.ticket_subject}} [#{{.ticket_number}}], body referencing {{.ticket_url}}. The {{.ticket_ref}} token is the human reference (e.g. INC-1042); [#{{.ticket_number}}] is the email-threading anchor and must stay in the subject for customer replies to thread back onto the ticket.

A failing disk now opens a ticket and emails the infra team via the ticket rule, with full template control on both halves.

Common patterns

"Everything goes to one inbox, with Slack on critical"

One template, both channels enabled, is_default = true. Set the email channel's "Send for" to all severities and the Slack channel's to errors only — warnings and info still reach the inbox, but the channel only lights up when something is actually on fire. The built-in Slack card colour-codes by severity too.

"Different routing per environment tag"

Asset-alert templates resolve via the agent/site/client cascade. There's no "tag = prod" condition. To split routing by environment, set the alert_template_id on agents in the prod tag explicitly (or on a site that contains only prod servers).

"Send to a webhook instead of email"

One template, email channel disabled, webhook channel enabled with the URL. SSRF protection blocks RFC 1918 / loopback / link-local destinations.

"Customer-facing template doesn't leak internal details"

Build two templates: an internal one (NOC inbox plus Slack) and a customer one (their ops list, terse phrasing, no {{.Message}} or {{.CheckName}} leaks). Assign the customer one at the client level so it cascades. For PSA replies use ticket-notification kind so it goes via the rules engine with full template-variable escaping.

Common issues

Variable shows literal {agent_name} in the rendered email. Wrong variable family. {agent_name} belongs to monitoring-workflow action params, not alert templates. Asset-alert templates want {{.AgentHostname}}; ticket-notification templates want {{.requester.first_name}}-style dotted names. Literal braces in the output means the renderer didn't recognise the token.

Alert email/webhook stopped sending after editing the template. Almost always a misspelt variable — asset-alert templates fail the whole send on an unknown variable rather than rendering a blank. Check the editor's Template variables panel and correct the name.

Picker shows ticket-notification templates in the install token dropdown. Fixed 2026-05-22 (commit d876b4b1). The picker now filters to kind = asset_alert. If you're on an older build, the four system rows leak into the picker because the kind facet wasn't yet enforced on that consumer.

An agent isn't sending notifications even though the agent has alert_template_id set. Check the template's enabled toggles — if email is enabled but email_recipients is empty, no email goes out. Webhook needs a non-blocked URL. Slack/Discord need a real incoming webhook URL and Teams needs a Workflows webhook URL — on the template or as the org default on the Channels tab. Use the per-channel Send test button to confirm.

Teams notifications stopped working around May 2026. Your webhook is probably a retired Office 365 Connector URL (webhook.office.com) — Microsoft disabled these permanently. Recreate it as a Teams Workflow ("Send webhook alerts to a channel" template) and paste the new URL.

Org-default and per-agent both set — which wins? The cascade. Per-agent wins over per-site wins over per-client wins over org-default. If you've set both, the more specific one is what fires.

Deleted a template that was referenced by an agent/site/client. The alert_template_id foreign key uses ON DELETE SET NULL. Existing assignments get nulled out, and the cascade falls through to the next level. No alerts get sent to a dead template.

Migration: from one mixed pile to a kind facet

Until 2026-05-22 the alert_templates table had no kind column. The four consumers all looked at the same flat list. That's why pickers showed ticket notification templates alongside asset alert templates — the database didn't have a way to distinguish them.

The migration 000260_alert_templates_workflows_kind added the column and backfilled existing rows:

• Rows with system_key set → ticket_notification
• Rows referenced by the alert engine hierarchy (agents/sites/clients) or marked is_default → asset_alert
• Everything else → custom

You can change the kind on a template after creation via the editor. The cascade and rule lookup don't care about kind directly; they only look at which rows are pointed at, by name, or by system_key. Kind is purely a UI organisation hint.

Next

• Workflows — the monitoring-side rules engine that consumes asset-alert templates
• PSA Workflows — the ticket-side rules engine that consumes ticket-notification templates
• Notifications — conceptual overview of PSA notifications