71 lines
2.3 KiB
Markdown
71 lines
2.3 KiB
Markdown
# Prometheus Alerting
|
|
|
|
The repository includes a ready-to-use example rule group in
|
|
[`examples/pve-exporter.rules.yml`](../examples/pve-exporter.rules.yml). It
|
|
covers node, guest, storage, subscription, physical disk, ZFS, SDN, clock-skew,
|
|
and memory-overcommit conditions.
|
|
|
|
## Enable the rule file
|
|
|
|
Make the rule file available to Prometheus and reference it from
|
|
`prometheus.yml`:
|
|
|
|
```yaml
|
|
rule_files:
|
|
- /etc/prometheus/rules/pve-exporter.rules.yml
|
|
|
|
scrape_configs:
|
|
- job_name: pve-exporter
|
|
static_configs:
|
|
- targets: [pve-exporter:9090]
|
|
```
|
|
|
|
For a containerized Prometheus deployment, mount the example at the configured
|
|
path:
|
|
|
|
```yaml
|
|
services:
|
|
prometheus:
|
|
image: prom/prometheus:latest
|
|
volumes:
|
|
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
|
- ./examples/pve-exporter.rules.yml:/etc/prometheus/rules/pve-exporter.rules.yml:ro
|
|
```
|
|
|
|
Validate the file before reloading Prometheus:
|
|
|
|
```sh
|
|
promtool check rules examples/pve-exporter.rules.yml
|
|
```
|
|
|
|
## Collector requirements
|
|
|
|
Prometheus evaluates a rule only when its referenced time series exist. Enable
|
|
the matching exporter collectors:
|
|
|
|
| Alerts | Required collector setting |
|
|
| --- | --- |
|
|
| Node filesystem, state, CPU, clock skew, memory overcommit | `metrics.nodeStatus` |
|
|
| Guest CPU and memory overcommit | `metrics.qemu` and/or `metrics.lxc` |
|
|
| Storage state and capacity | `metrics.storage` |
|
|
| Subscription expiration | `metrics.subscription` |
|
|
| Physical disk health and wear | `metrics.disk` |
|
|
| ZFS state and errors | `metrics.zfs` |
|
|
| SDN state | `metrics.sdn` |
|
|
|
|
The default thresholds and `for` durations are operational examples. Review
|
|
them against the size, workload, redundancy, and maintenance policy of your
|
|
environment before enabling notifications.
|
|
|
|
Three rules require deployment-specific review:
|
|
|
|
- `PveUnexpectedStandaloneMode` should be removed or disabled when standalone
|
|
PVE operation is expected.
|
|
- `PveNodeSubscriptionMissing` is informational and should be removed when
|
|
unsubscribed nodes are intentional.
|
|
- `PveExporterDown` expects the Prometheus scrape job to be named
|
|
`pve-exporter`; adjust its `job` matcher if a different name is used.
|
|
|
|
Every rule uses the `cluster` label. Standalone installations are supported and
|
|
use the label value `Standalone node - <node>`.
|