Writing Prometheus Alerting Rules With LLMs in October 2026: Check Every Metric Name, Unit-Test With promtool, and Alert on Symptoms

Ask a large language model for "an alert when the API error rate is too high" and you will get a tidy Prometheus rule in seconds. It will look right. It might even be right. But alerting rules fail in quiet ways: a metric name that doesn't exist in your setup never fires, a missing for clause pages someone at 3 a.m. for a two-second blip, and a label the model invented means the alert routes nowhere. None of these produce an error. They just produce silence or noise.

This guide walks through a practical workflow for using LLMs to draft Prometheus alerting rules without shipping those failures. The short version: check every metric and label against your real Prometheus, validate and unit-test every rule with promtool, and design alerts around user-facing symptoms with deliberate thresholds and durations.

Why LLM-written alert rules go wrong

Models have seen a lot of PromQL, mostly from documentation, blog posts, and popular exporters. That gives them good instincts for syntax and a strong tendency to fill gaps with plausible guesses. The common failure modes are predictable:

Invented or outdated metric names. The model writes http_requests_total when your service exports myapp_http_server_requests_seconds_count, or uses a node_exporter name that was renamed years ago.
Invented labels. It filters on status="500" when your label is code, or groups by service when your targets use job.
Counter mistakes. It compares a raw counter to a threshold (errors_total > 100), which fires forever once the counter passes 100, or applies rate() to a gauge.
No duration. It omits for:, so a single bad evaluation fires the alert immediately.
Silent on missing data. If the metric disappears because the exporter died, the expression returns nothing and the alert never fires.
Thresholds pulled from thin air. "CPU above 80%" is a classic. It is rarely tied to anything a user actually feels.

The fix is not to stop using the model. It is to treat its output as a draft and run it through the same checks you'd apply to a colleague's pull request, with a few extra ones because the model can't see your environment.

Step 1: Give the model your real metrics, not a description

The single biggest improvement comes from pasting real data into the prompt. Before asking for a rule, pull the actual series names and labels from your Prometheus server. The HTTP API makes this easy:

# List all metric names that match a prefix
curl -s 'http://prometheus:9090/api/v1/label/__name__/values' \
  | jq -r '.data[]' | grep '^myapp_http'

# Show the label sets for one metric
curl -s -G 'http://prometheus:9090/api/v1/series' \
  --data-urlencode 'match[]=myapp_http_server_requests_seconds_count' \
  | jq '.data[0:5]'

Paste a handful of those series into your prompt along with the metric type (counter, gauge, histogram, summary) and your scrape interval. A good prompt looks something like this:

Write a Prometheus alerting rule.
Goal: page when more than 5% of requests to the checkout service
return 5xx errors for 10 minutes.
Metric (counter): myapp_http_server_requests_seconds_count
Example series:
  {job="checkout", code="200", method="POST", route="/api/order"}
  {job="checkout", code="503", method="POST", route="/api/order"}
Scrape interval: 30s
Use only these metric names and labels. Include for:, a severity
label, and summary/description annotations. Output YAML only.

The instruction "use only these metric names and labels" matters. It won't stop every guess, but it gives you something concrete to check the output against.

Step 2: Read the PromQL before you validate it

Here is the kind of rule a model might return for that prompt:

groups:
  - name: checkout-slo
    rules:
      - alert: CheckoutHighErrorRate
        expr: |
          sum(rate(myapp_http_server_requests_seconds_count{job="checkout", code=~"5.."}[5m]))
            /
          sum(rate(myapp_http_server_requests_seconds_count{job="checkout"}[5m]))
            > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Checkout 5xx error ratio above 5%"
          description: "{{ $value | humanizePercentage }} of checkout requests are failing."

Before running any tool, read it with a checklist:

Every metric name and label appears in your pasted series. Here, job and code are real; nothing was invented.
Counters are wrapped in rate() or increase(). They are.
The range window is long enough. A common rule of thumb is at least four times the scrape interval so rate() always has enough samples. With 30-second scrapes, [5m] is comfortable; [1m] would be fragile.
Aggregation keeps the labels you need for routing. sum(...) with no by clause drops every label, including job. If your Alertmanager routes on job or team, use sum by (job) (...) on both sides of the division.
The for duration matches the goal. The prompt said 10 minutes, and the rule says 10 minutes.

Point 4 is the kind of thing models get wrong constantly, and it is invisible until an alert lands in the wrong channel. A safer version of the expression:

sum by (job) (rate(myapp_http_server_requests_seconds_count{job="checkout", code=~"5.."}[5m]))
  /
sum by (job) (rate(myapp_http_server_requests_seconds_count{job="checkout"}[5m]))
  > 0.05

Step 3: Validate syntax with promtool

promtool ships with Prometheus and catches structural mistakes: bad YAML, invalid PromQL, duplicate rule names, and malformed templates in annotations. Run it on every generated file:

promtool check rules checkout-alerts.yml

A clean run reports how many rules it found. Anything else is a hard stop. Wire this into CI so a rule file can't merge without passing. A minimal step looks like:

docker run --rm -v "$PWD/rules:/rules" --entrypoint promtool \
  prom/prometheus check rules /rules/*.yml

Pin the image to the same Prometheus version you run in production, so the check uses the same PromQL parser your server does.

Step 4: Unit-test the rule with promtool test rules

Syntax checks don't tell you whether the alert fires when it should. promtool test rules does. You describe synthetic input series and assert which alerts should be firing at specific times. This is also a task models are genuinely good at drafting, as long as you check the expectations yourself.

The test below assumes the corrected sum by (job) expression from Step 2, which is why it expects a job label on the firing alert.

# checkout-alerts.test.yml
rule_files:
  - checkout-alerts.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      # 100 requests per minute, 10 of them 5xx (10% errors)
      - series: 'myapp_http_server_requests_seconds_count{job="checkout", code="200"}'
        values: '0+90x30'
      - series: 'myapp_http_server_requests_seconds_count{job="checkout", code="503"}'
        values: '0+10x30'
    alert_rule_test:
      # Not yet firing: the condition has not held for 10 minutes
      - eval_time: 5m
        alertname: CheckoutHighErrorRate
        exp_alerts: []
      # Firing after the for: duration has passed
      - eval_time: 20m
        alertname: CheckoutHighErrorRate
        exp_alerts:
          - exp_labels:
              severity: page
              job: checkout

The notation 0+90x30 means "start at 0 and add 90 each interval, 30 times," which simulates a steadily increasing counter. Run it with:

promtool test rules checkout-alerts.test.yml

Write at least three cases for every alert:

Healthy traffic where the alert must not fire.
A short spike shorter than the for duration, where it must not fire.
A sustained problem where it must fire, with the labels your routing depends on.

If the model writes the tests, deliberately break the rule (change > 0.05 to > 0.5) and confirm the "must fire" test fails. A test that passes no matter what the rule says is worse than no test, because it gives false confidence.

Step 5: Handle missing data on purpose

If the checkout service stops reporting entirely, both sides of the error ratio return nothing and the alert stays quiet. That's often exactly when you most need to know. Add a separate rule for absent data:

      - alert: CheckoutMetricsMissing
        expr: absent(up{job="checkout"} == 1)
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "No healthy checkout targets are being scraped"

Ask the model explicitly for an absence rule. Most won't add one unprompted. And check the logic: absent(up{job="checkout"} == 1) fires both when the targets disappear from service discovery and when every target is down, which is usually what you want.

Step 6: Alert on symptoms, and pick thresholds you can defend

Models default to cause-based alerts because those dominate example rule collections: high CPU, high memory, disk 80% full. Some of these are useful, especially disk space that will genuinely run out. But paging on causes produces a lot of alerts that don't correspond to any user impact.

A better split:

Page on symptoms users feel: error ratio, latency at a high percentile, failed checkouts, queue age growing without bound.
Ticket or dashboard for causes: CPU saturation, memory pressure, a single pod restarting.
Page on predicted exhaustion instead of fixed percentages, for example predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4 * 3600) < 0, which fires when the trend says the disk fills within four hours.

For latency, a histogram-based rule is a good example to ask for and then verify:

histogram_quantile(
  0.99,
  sum by (job, le) (rate(myapp_http_server_requests_seconds_bucket{job="checkout"}[5m]))
) > 1.5

Check two things here. First, the le label must survive the aggregation, or histogram_quantile returns nothing. Second, the threshold (1.5 seconds) should come from your own service-level objective or from looking at the last few weeks of real data in the Prometheus graph view, not from the model. Ask it to explain where a number came from; if the answer is "a common default," replace it.

Step 7: Make annotations useful at 3 a.m.

The person who receives the alert needs to know what's broken, how bad it is, and where to start. Ask the model for annotations that include the current value and a runbook link, then make sure the link is real:

annotations:
  summary: "Checkout p99 latency above 1.5s"
  description: "p99 is {{ $value | humanizeDuration }} for job {{ $labels.job }}."
  runbook_url: "https://wiki.example.com/runbooks/checkout-latency"

promtool check rules will catch template syntax errors in annotations, but not a runbook URL that points to a page nobody wrote. Models will happily invent those, so click every link before merging.

A short review checklist

Keep this next to your editor when reviewing LLM-generated alerting rules:

Every metric and label exists in your Prometheus (check with the series API).
Counters use rate() or increase(); gauges don't.
Range windows are at least four scrape intervals.
Aggregations keep routing labels and, for histograms, le.
Every alert has a for duration that matches the intent.
An absence rule covers the case where the data stops.
promtool check rules passes in CI, pinned to your production version.
promtool test rules covers healthy, short spike, and sustained failure cases, and you've proven the tests can fail.
Thresholds come from your SLOs or real data, not model defaults.
Annotations include the current value and a runbook link that actually exists.

The bottom line

LLMs are a fast way to get from "I need an alert for this" to a reasonable first draft, and they're especially helpful at writing the tedious parts: unit test fixtures, annotation templates, and boilerplate rule groups. What they can't do is see your metrics, your routing, or your users. Feed them real series data, validate with promtool, test with synthetic inputs that you know should and shouldn't fire, and keep the threshold decisions for yourself. That's how you get alerts that wake people up only when something is genuinely wrong.

Search This Blog

Technology On the Net