Alert templates

The name is misleading. The thing called an "alert template" is actually a multi-channel notification config — one row that can fan out to email, Slack, Teams, Discord, webhook, and SMS simultaneously, with a per-channel enabled toggle and per-channel body template.

The same table started life serving the asset alert engine, then quietly got reused by three more subsystems. This article unpacks the whole model so you know what you're looking at when the picker turns up in unexpected places.

What's actually in a template

One template row holds:

• Name — what consumers refer to it by
• Kind — asset_alert, ticket_notification, or custom (added 2026-05-22 to facet the editor)
• Email channel — recipients (list), subject template, body template, optional from-override
• Webhook channel — URL, custom headers, body template
• SMS channel — recipients, body template (SMS dispatch is stubbed today; the config persists for the future hook)
• Slack channel — incoming webhook URL, channel override, body template
• Teams channel — incoming webhook URL, body template
• Discord channel — incoming webhook URL, body template
• is_default — at most one per org. Used as the org-wide fallback when nothing more specific resolves.
• system_key — set for the four PSA system templates. Null for everything else.

Every channel has its own enable toggle. You can build a single template that sends a short Slack message and a long HTML email simultaneously, while ignoring the other four channels.

The four consumers

Consumer	What fires it	How it picks a template
Alert engine (Monitoring → Alerts)	Check fails / agent goes offline / task fails	Resolves via the agent → site → client → org-default cascade
Monitoring workflows with create_alert action	check_failure / agent_status / event log / schedule / webhook_inbound	Inserts an alert row; the alert engine then resolves a template the normal way
PSA ticket rules with send_email action	ticket.created, ticket.commented, ticket.sla_warning, ticket.scheduled_stale, ticket.scheduled_customer_silence, ticket.updated	Looks up by name or system_key (kind = ticket_notification or custom)
PSA system notifications (the four [System] rows)	Auto-ack inbound email / public reply / assignee comment / watcher comment	Bound by system_key to a system-owned rule

The kind column (added 2026-05-22) is the facet that keeps these from leaking into each other's pickers. Pre-facet, an agent install dropdown could legitimately show ticket notification templates — which was the prompt to fix the IA.

Two template syntaxes (this is the awkward bit)

The asset-alert side and the PSA-ticket side render with different engines and different placeholder syntax. Both are valid; they just don't speak each other's language.

Asset-alert templates use single-brace, snake-case names resolved by the alert engine's own renderer:

Disk space low on {agent_name} — {check_name} reported {message}

Common variables for asset-alert kind: {agent_name}, {client_name}, {site_name}, {check_name}, {check_type}, {severity}, {message}, {alert_url}.

Ticket-notification templates use Go's text/template and html/template syntax — double-brace, dot-prefix, with html kind auto-escaping attacker-controlled variables:

<p>Hi {{.requester.first_name}},</p>
<p>Your ticket <strong>#{{.ticket.number}}</strong> is now {{.ticket.status}}.</p>

Common variables for ticket_notification kind: {{.ticket.number}}, {{.ticket.subject}}, {{.ticket.priority}}, {{.ticket.status}}, {{.ticket.url}}, {{.client.name}}, {{.requester.first_name}}, {{.requester.last_name}}, {{.requester.email}}, {{.comment.body}}, {{.assignee.name}}, {{.watchers}}.

Copy-pasting from one to the other will leave literal {var} or {{.var}} strings in the rendered output. If you see that, it's a syntax mismatch, not a missing variable.

Resolution: how asset-alert templates get picked

When a check fails on agent X at site Y for client Z, the alert engine walks a cascade to decide which template to fire:

1. agents.alert_template_id      (most specific — set per agent)
2. sites.alert_template_id       (covers every agent at the site)
3. clients.alert_template_id     (covers every site/agent under the client)
4. alert_templates.is_default    (org-wide fallback)
5. nothing — no notification

The first match wins. This means you can have one template for production servers (HTML email plus Slack to #alerts), a different one for development environments (silent alert row, no notification), and only override at the agent level for the few endpoints that need different routing.

Four ways to assign an asset-alert template

Install token. When you generate an install token at Clients → site → Generate Install Token, you can optionally pick a template. The new agent registers with alert_template_id already set on its row. Bypasses the site/client cascade. Useful when you're deploying a batch of servers that need their own routing from day one.

Site editor. Clients → site → Alert Template field. Applies to every agent at that site that doesn't have its own.

Client editor. Clients → client → Alert Template field. Applies to every site and agent under that client that doesn't have a more specific override.

Org default. On the templates page, mark one template is_default = true. There can only be one. Used when nothing higher in the cascade resolves.

In practice most MSPs set one org-default ("everything goes to the noc inbox plus #alerts") and add per-client overrides only when a client demands something different.

Resolution: how ticket-notification templates get picked

No cascade. Ticket rules and system notifications reference templates by name or system_key. There's no "default ticket template" — every consumer is explicit about which template it wants.

For system notifications, the binding is fixed: auto_ack_email_ticket always uses the row with system_key = 'auto_ack_email_ticket', and so on.

For custom rules, the rule's send_email action carries a template_key param. You pick a template by name when authoring the rule. Renames break the binding; if you rename a template, update any rules that reference it.

Where each kind appears in the UI

Kind	Editor location	Picker locations
asset_alert	Alerts → Templates (/alerts/templates)	Install token dropdown, site Alert Template field, client Alert Template field
ticket_notification	PSA → Workflows → System Notifications (the four system rows) and PSA → Workflows → Notification Templates (custom ones)	PSA rule editor's send_email action
custom	PSA → Workflows → Notification Templates	PSA rule editor's send_email action

Templates with system_key set get a lock badge — you can edit subject and body freely, but delete is disabled.

Worked example: cascading templates for one MSP

Say you run an MSP with three clients. Most of them want failing-server alerts to land in your central NOC inbox. One client, Acme, has its own SRE team and wants their own Slack workspace pinged on their alerts. A third client, BoringCo, has dev hosts they don't want notifications about at all.

1. Create "NOC default" template at Alerts → Templates. Email channel only. Recipients = [email protected]. Mark is_default = true.
2. Create "Acme SRE" template. Email channel = Acme's ops list, plus Slack channel pointing at their #opsmerge-alerts. Set is_default = false.
3. Assign "Acme SRE" to the Acme client at Clients → Acme → Alert Template. Every Acme site and agent inherits.
4. Create "Silent" template. No channels enabled (just a placeholder so the cascade resolves to something).
5. Assign "Silent" to BoringCo's "dev" site at Clients → BoringCo → Dev → Alert Template. Production hosts at BoringCo still inherit the org default.

Now:

• A failing check on bigco-dc-01 falls through to the org default → NOC inbox gets emailed.
• A failing check on acme-prod-02 resolves at the client level → email to Acme's ops, plus Slack to their channel.
• A failing check on boringco-dev-04 resolves at the site level → silent.
• Override acme-prod-02 directly if you need to (e.g., a dedicated escalation template for a critical box).

Worked example: chaining alert template to ticket creation

The two engines work nicely in series:

1. Monitoring workflow at Monitoring → Workflows: check_failure with check_type = disk, action = create_ticket priority high, title template "Disk full on {agent_name}".
2. PSA ticket rule at PSA → Workflows → Ticket Rules: trigger ticket.created, condition category = infrastructure, action send_email with template_key = "Infra escalation".
3. Create the "Infra escalation" template at PSA → Workflows → Notification Templates with kind = ticket_notification, subject {{.ticket_ref}} {{.ticket_subject}} [#{{.ticket_number}}], body referencing {{.ticket_url}}. The {{.ticket_ref}} token is the human reference (e.g. INC-1042); [#{{.ticket_number}}] is the email-threading anchor and must stay in the subject for customer replies to thread back onto the ticket.

A failing disk now opens a ticket and emails the infra team via the ticket rule, with full template control on both halves.

Common patterns

"Everything goes to one inbox, with Slack on critical"

One template, both channels enabled, is_default = true. The Slack body uses {severity} so only critical/error rows look visually loud; warning-level rows still go to email + Slack but with neutral phrasing.

"Different routing per environment tag"

Asset-alert templates resolve via the agent/site/client cascade. There's no "tag = prod" condition. To split routing by environment, set the alert_template_id on agents in the prod tag explicitly (or on a site that contains only prod servers).

"Send to a webhook instead of email"

One template, email channel disabled, webhook channel enabled with the URL. SSRF protection blocks RFC 1918 / loopback / link-local destinations.

"Customer-facing template doesn't leak internal details"

Build two templates: an internal one (NOC inbox plus Slack) and a customer one (their ops list, terse phrasing, no {message} or {check_name} leaks). Assign the customer one at the client level so it cascades. For PSA replies use ticket-notification kind so it goes via the rules engine with full template-variable escaping.

Common issues

Variable shows literal {agent_name} in the rendered email. Wrong syntax for the kind. Asset-alert templates use single-brace; ticket-notification templates use Go {{.var}} syntax. If you're seeing literal text, you've used the wrong family.

Picker shows ticket-notification templates in the install token dropdown. Fixed 2026-05-22 (commit d876b4b1). The picker now filters to kind = asset_alert. If you're on an older build, the four system rows leak into the picker because the kind facet wasn't yet enforced on that consumer.

An agent isn't sending notifications even though the agent has alert_template_id set. Check the template's enabled toggles — if email is enabled but email_recipients is empty, no email goes out. Webhook needs a non-blocked URL. Slack/Teams/Discord need real incoming webhook URLs, not their organisational dashboard URLs.

Org-default and per-agent both set — which wins? The cascade. Per-agent wins over per-site wins over per-client wins over org-default. If you've set both, the more specific one is what fires.

Deleted a template that was referenced by an agent/site/client. The alert_template_id foreign key uses ON DELETE SET NULL. Existing assignments get nulled out, and the cascade falls through to the next level. No alerts get sent to a dead template.

Migration: from one mixed pile to a kind facet

Until 2026-05-22 the alert_templates table had no kind column. The four consumers all looked at the same flat list. That's why pickers showed ticket notification templates alongside asset alert templates — the database didn't have a way to distinguish them.

The migration 000260_alert_templates_workflows_kind added the column and backfilled existing rows:

• Rows with system_key set → ticket_notification
• Rows referenced by the alert engine hierarchy (agents/sites/clients) or marked is_default → asset_alert
• Everything else → custom

You can change the kind on a template after creation via the editor. The cascade and rule lookup don't care about kind directly; they only look at which rows are pointed at, by name, or by system_key. Kind is purely a UI organisation hint.

Next

• Workflows — the monitoring-side rules engine that consumes asset-alert templates
• PSA Workflows — the ticket-side rules engine that consumes ticket-notification templates
• Notifications — conceptual overview of PSA notifications