lostakj/pve-exporter
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 8 Wiki Activity

Align metric names with Prometheus naming conventions
Build Docker image on push / docker (push) Successful in 21s
Details
Build and push Docker image on tag / docker (push) Successful in 22s
Details

Browse Source 
Use base units, export cumulative values as _total counters, update docs, dashboards and rules.
This commit is contained in:
lostakj
2026-08-24 19:13:05 +02:00
parent 9b076d4353
commit 92e54418bc
16 changed files with 541 additions and 284 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+15 -7
View File
@@ -58,15 +58,18 @@ type PveMetricsCollector interface {
```
Successful execution time is recorded in
`pve_metrics_collection_latency_ms`. A collector error is logged and the
`pve_metrics_collection_duration_seconds`. A collector error is logged and the
manager continues with the next collector.
## TTL metrics
Most project metrics use `TTLGaugeVec`, a wrapper around Prometheus `GaugeVec`.
Every label set records its last update. `TTLRegistry` checks registered metrics
every five seconds and removes series that have not been updated for five
minutes.
Gauges use `TTLGaugeVec`, a wrapper around Prometheus `GaugeVec`. Cumulative
values use `TTLCounterVec`, which exports the absolute value reported by the PVE
API as a Prometheus counter (`GaugeVec` cannot be used because a counter name
must carry a `_total` suffix, and `CounterVec` cannot be used because it only
supports `Inc`/`Add`). Every label set records its last update. `TTLRegistry`
checks registered metrics every five seconds and removes series that have not
been updated for five minutes.
This prevents deleted guests or storage resources from remaining indefinitely,
while allowing short API failures to preserve the last known values.
@@ -75,14 +78,19 @@ while allowing short API failures to preserve the last known values.
1. Add typed response models and API methods under `proxmox`.
2. Add a collector implementing `PveMetricsCollector` under `metrics`.
3. Create metrics with `NewTTLGaugeVec`, choose stable labels, and register each
metric with the shared `TTLRegistry`.
3. Create metrics with `NewTTLGaugeVec` or, for cumulative values, with
`NewTTLCounterVec`, choose stable labels, and register each metric with the
shared `TTLRegistry`.
4. Add a boolean switch to `PveMetricsConfiguration` and `config.example.yaml`.
5. Register the collector in `NewPveMetricsManager`.
6. Add parsing, state-mapping, and topology tests as appropriate.
7. Document the switch, labels, units, and state values in
[configuration.md](configuration.md) and [metrics.md](metrics.md).
Metric names must follow the Prometheus naming conventions; they are verified by
`TestPveMetricNamesFollowPrometheusConventions`, which lints every registered
metric with `promlint`.
Avoid labels containing changing messages, timestamps, or other unbounded
values. For recursive resources such as ZFS, include a stable path so repeated
component names remain distinguishable.
docs/configuration.md
+1 -1
View File
@@ -101,7 +101,7 @@ All switches are booleans. Omitted switches default to `false`.
| `subscription` | Node subscription information | `pve_node_subscription_` |
| `sdn` | Software-defined network state | `pve_sdn_` |
The exporter-level `pve_metrics_collection_latency_ms` summary is always
The exporter-level `pve_metrics_collection_duration_seconds` summary is always
registered.
## Validation notes
docs/metrics.md
+103 -43
View File
@@ -3,6 +3,12 @@
All exporter-specific metric names start with `pve_`. The Go Prometheus client
also exposes its standard `go_*`, `process_*`, and HTTP handler metrics.
Metric names follow the [Prometheus naming conventions](https://prometheus.io/docs/practices/naming/):
base units only (seconds, bytes, ratios) and a `_total` suffix on every metric
exported as a counter. Counters hold the absolute value reported by the PVE API
and reset when the guest or node restarts, so query them with `rate()` or
`increase()`. All remaining metrics are gauges.
## Common labels and lifecycle
Most resource metrics use these labels:
@@ -22,7 +28,7 @@ remain visible for up to five minutes after it stops.
| Metric | Labels | Description |
| --- | --- | --- |
| `pve_metrics_collection_latency_ms` | `collector` | Summary of successful collector execution time in milliseconds. |
| `pve_metrics_collection_duration_seconds` | `collector` | Summary of successful collector execution time in seconds. |
## Cluster state
@@ -42,9 +48,9 @@ and `node`.
| Metric | Description |
| --- | --- |
| `pve_node_state` | Node online state: `1` online, `0` offline. |
| `pve_node_uptime` | Node uptime in seconds. |
| `pve_node_cpu_count` | Number of logical CPUs. |
| `pve_node_cpu_usage` | CPU usage reported by PVE. |
| `pve_node_uptime_seconds` | Node uptime in seconds. |
| `pve_node_cpus` | Number of logical CPUs. |
| `pve_node_cpu_usage_ratio` | CPU usage reported by PVE as a ratio between `0` and `1`. |
| `pve_node_memory_total_bytes` | Total memory in bytes. |
| `pve_node_memory_used_bytes` | Used memory in bytes. |
| `pve_node_memory_free_bytes` | Free memory in bytes. |
@@ -57,10 +63,10 @@ and `node`.
| `pve_node_rootfs_used_bytes` | Root filesystem used bytes. |
| `pve_node_rootfs_total_bytes` | Root filesystem total bytes. |
| `pve_node_rootfs_avail_bytes` | Root filesystem bytes available to unprivileged processes. |
| `pve_node_time` | Node UTC Unix timestamp. |
| `pve_node_localtime` | Node local Unix timestamp. |
| `pve_node_cpuinfo` | Constant `1` carrying `flags`, `cores`, `model`, `sockets`, `cpus`, and `hvm` labels. |
| `pve_node_systeminfo` | Constant `1` carrying `kversion`, `pveversion`, `machine`, `sysname`, and `release` labels. |
| `pve_node_time_seconds` | Node UTC Unix timestamp. |
| `pve_node_localtime_seconds` | Node local Unix timestamp. |
| `pve_node_cpu_info` | Constant `1` carrying `flags`, `cores`, `model`, `sockets`, `cpus`, and `hvm` labels. |
| `pve_node_system_info` | Constant `1` carrying `kversion`, `pveversion`, `machine`, `sysname`, and `release` labels. |
## Physical disks
@@ -70,7 +76,7 @@ Enabled with `metrics.disk`. Labels are `cluster`, `node`, `wwn`, `type`,
| Metric | Description |
| --- | --- |
| `pve_node_disk_healthy` | SMART health state: `1` for `OK` or `PASSED`, otherwise `0`. |
| `pve_node_disk_wearout` | Device wearout percentage when supplied by the PVE API. |
| `pve_node_disk_wearout_percent` | Device wearout percentage when supplied by the PVE API. |
| `pve_node_disk_size_bytes` | Physical disk size in bytes. |
## ZFS
@@ -86,9 +92,9 @@ hierarchy, and `leaf` is `true` for a device entry.
| Metric | Description |
| --- | --- |
| `pve_node_zfs_state` | Numeric component state: `0=UNKNOWN`, `1=ONLINE`, `2=DEGRADED`, `3=FAULTED`, `4=OFFLINE`, `5=REMOVED`, `6=UNAVAIL`. |
| `pve_node_zfs_read_errors` | ZFS read error count reported for the component. |
| `pve_node_zfs_write_errors` | ZFS write error count reported for the component. |
| `pve_node_zfs_checksum_errors` | ZFS checksum error count reported for the component. |
| `pve_node_zfs_read_errors_total` | ZFS read error count reported for the component. |
| `pve_node_zfs_write_errors_total` | ZFS write error count reported for the component. |
| `pve_node_zfs_checksum_errors_total` | ZFS checksum error count reported for the component. |
ZFS section entries that do not contain a counter do not produce a false zero
series. A counter explicitly returned as zero is exported normally.
@@ -100,9 +106,9 @@ pve_node_zfs_state != 1
```
```promql
(pve_node_zfs_read_errors > 0)
or (pve_node_zfs_write_errors > 0)
or (pve_node_zfs_checksum_errors > 0)
(pve_node_zfs_read_errors_total > 0)
or (pve_node_zfs_write_errors_total > 0)
or (pve_node_zfs_checksum_errors_total > 0)
```
## PVE storage
@@ -126,8 +132,8 @@ Enabled with `metrics.subscription`. The common labels are `cluster` and
| --- | --- |
| `pve_node_subscription_info` | Constant `1` carrying subscription product and server identifiers. |
| `pve_node_subscription_status` | `0` for API status `notfound`, `1` for `active`, `2` for expired or otherwise unusable states such as `invalid`, `suspended`, and `new`. |
| `pve_node_subscription_regdate` | Subscription registration date as a Unix timestamp, when available. |
| `pve_node_subscription_nextduedate` | Next due date as a Unix timestamp, when available. |
| `pve_node_subscription_registration_timestamp_seconds` | Subscription registration date as a Unix timestamp, when available. |
| `pve_node_subscription_next_due_timestamp_seconds` | Next due date as a Unix timestamp, when available. |
| `pve_node_subscription_sockets` | Number of covered sockets reported by PVE. |
The PVE API cannot distinguish a node that never had a subscription from one
@@ -155,16 +161,16 @@ Enabled with `metrics.lxc`. Templates are skipped. Labels are `cluster`,
| Metric | Description |
| --- | --- |
| `pve_ct_state` | `1` running, `0` stopped. |
| `pve_ct_uptime` | Uptime in seconds; updated only while running. |
| `pve_ct_cpu_count` | Configured CPU count. |
| `pve_ct_cpu_usage` | CPU usage reported by PVE; updated only while running. |
| `pve_ct_mem_total_bytes` | Configured memory limit in bytes. |
| `pve_ct_mem_used_bytes` | Used memory in bytes; updated only while running. |
| `pve_ct_network_in_bytes` | Received bytes since container start; updated only while running. |
| `pve_ct_network_out_bytes` | Transmitted bytes since container start; updated only while running. |
| `pve_ct_disk_rd_bytes` | Disk bytes read; updated only while running. |
| `pve_ct_disk_wr_bytes` | Disk bytes written; updated only while running. |
| `pve_ct_disk_usage_bytes` | Used root disk bytes; updated only while running. |
| `pve_ct_uptime_seconds` | Uptime in seconds; updated only while running. |
| `pve_ct_cpus` | Configured CPU count. |
| `pve_ct_cpu_usage_ratio` | CPU usage reported by PVE as a ratio between `0` and `1`; updated only while running. |
| `pve_ct_memory_total_bytes` | Configured memory limit in bytes. |
| `pve_ct_memory_used_bytes` | Used memory in bytes; updated only while running. |
| `pve_ct_network_receive_bytes_total` | Received bytes since container start; updated only while running. |
| `pve_ct_network_transmit_bytes_total` | Transmitted bytes since container start; updated only while running. |
| `pve_ct_disk_read_bytes_total` | Disk bytes read; updated only while running. |
| `pve_ct_disk_write_bytes_total` | Disk bytes written; updated only while running. |
| `pve_ct_disk_used_bytes` | Used root disk bytes; updated only while running. |
| `pve_ct_disk_size_bytes` | Configured root disk size in bytes. |
| `pve_ct_swap_used_bytes` | Used swap in bytes; updated only while running. |
@@ -176,23 +182,77 @@ Enabled with `metrics.qemu`. Templates are skipped. Base labels are `cluster`,
| Metric | Extra label | Description |
| --- | --- | --- |
| `pve_vm_state` | none | `1` running, `0` stopped. |
| `pve_vm_uptime` | none | Uptime in seconds; updated only while running. |
| `pve_vm_uptime_seconds` | none | Uptime in seconds; updated only while running. |
| `pve_vm_agent` | none | QEMU guest agent state reported by PVE; updated only while running. |
| `pve_vm_cpu_count` | none | Configured CPU count. |
| `pve_vm_cpu_usage` | none | CPU usage reported by PVE; updated only while running. |
| `pve_vm_mem_total_bytes` | none | Configured maximum memory in bytes. |
| `pve_vm_mem_used_bytes` | none | Used memory in bytes; updated only while running. |
| `pve_vm_disk_usage_bytes` | none | Root disk usage reported by PVE. |
| `pve_vm_cpus` | none | Configured CPU count. |
| `pve_vm_cpu_usage_ratio` | none | CPU usage reported by PVE as a ratio between `0` and `1`; updated only while running. |
| `pve_vm_memory_total_bytes` | none | Configured maximum memory in bytes. |
| `pve_vm_memory_used_bytes` | none | Used memory in bytes; updated only while running. |
| `pve_vm_disk_used_bytes` | none | Root disk usage reported by PVE. |
| `pve_vm_disk_size_bytes` | none | Configured maximum root disk size in bytes. |
| `pve_vm_network_in_bytes` | `interface` | Bytes received by a virtual interface. |
| `pve_vm_network_out_bytes` | `interface` | Bytes transmitted by a virtual interface. |
| `pve_vm_disk_rd_operations` | `device` | Successful read operations. |
| `pve_vm_disk_wr_operations` | `device` | Successful write operations. |
| `pve_vm_disk_rd_bytes` | `device` | Bytes read from the block device. |
| `pve_vm_disk_wr_bytes` | `device` | Bytes written to the block device. |
| `pve_vm_disk_failed_rd_ops` | `device` | Failed read operations. |
| `pve_vm_disk_failed_wr_ops` | `device` | Failed write operations. |
| `pve_vm_disk_rd_time_total_ns` | `device` | Total block-device read time in nanoseconds. |
| `pve_vm_disk_wr_time_total_ns` | `device` | Total block-device write time in nanoseconds. |
| `pve_vm_network_receive_bytes_total` | `interface` | Bytes received by a virtual interface. |
| `pve_vm_network_transmit_bytes_total` | `interface` | Bytes transmitted by a virtual interface. |
| `pve_vm_disk_read_operations_total` | `device` | Successful read operations. |
| `pve_vm_disk_write_operations_total` | `device` | Successful write operations. |
| `pve_vm_disk_read_bytes_total` | `device` | Bytes read from the block device. |
| `pve_vm_disk_write_bytes_total` | `device` | Bytes written to the block device. |
| `pve_vm_disk_failed_read_operations_total` | `device` | Failed read operations. |
| `pve_vm_disk_failed_write_operations_total` | `device` | Failed write operations. |
| `pve_vm_disk_read_time_seconds_total` | `device` | Total block-device read time in seconds. |
| `pve_vm_disk_write_time_seconds_total` | `device` | Total block-device write time in seconds. |
Interface and block-device metrics are collected only for running VMs.
## Migration from earlier releases
All metric names were aligned with the Prometheus naming conventions. The table
below maps the previous names to the current ones. Dashboards, recording rules,
and alerting rules built against the old names have to be updated; the example
dashboards and rules in `examples/` already use the new names.
| Previous name | Current name | Note |
| --- | --- | --- |
| `pve_ct_cpu_usage` | `pve_ct_cpu_usage_ratio` | Values unchanged, they are already a ratio between `0` and `1`. |
| `pve_ct_cpu_count` | `pve_ct_cpus` | Renamed only. |
| `pve_ct_disk_rd_bytes` | `pve_ct_disk_read_bytes_total` | Type changed to counter. |
| `pve_ct_disk_usage_bytes` | `pve_ct_disk_used_bytes` | Renamed only. |
| `pve_ct_disk_wr_bytes` | `pve_ct_disk_write_bytes_total` | Type changed to counter. |
| `pve_ct_mem_total_bytes` | `pve_ct_memory_total_bytes` | Renamed only. |
| `pve_ct_mem_used_bytes` | `pve_ct_memory_used_bytes` | Renamed only. |
| `pve_ct_network_in_bytes` | `pve_ct_network_receive_bytes_total` | Type changed to counter. |
| `pve_ct_network_out_bytes` | `pve_ct_network_transmit_bytes_total` | Type changed to counter. |
| `pve_ct_uptime` | `pve_ct_uptime_seconds` | Renamed only. |
| `pve_metrics_collection_latency_ms` | `pve_metrics_collection_duration_seconds` | Values converted from milliseconds to seconds. |
| `pve_node_cpuinfo` | `pve_node_cpu_info` | Renamed only. |
| `pve_node_cpu_usage` | `pve_node_cpu_usage_ratio` | Values unchanged, they are already a ratio between `0` and `1`. |
| `pve_node_cpu_count` | `pve_node_cpus` | Renamed only. |
| `pve_node_disk_wearout` | `pve_node_disk_wearout_percent` | Renamed only. |
| `pve_node_localtime` | `pve_node_localtime_seconds` | Renamed only. |
| `pve_node_subscription_nextduedate` | `pve_node_subscription_next_due_timestamp_seconds` | Renamed only. |
| `pve_node_subscription_regdate` | `pve_node_subscription_registration_timestamp_seconds` | Renamed only. |
| `pve_node_systeminfo` | `pve_node_system_info` | Renamed only. |
| `pve_node_time` | `pve_node_time_seconds` | Renamed only. |
| `pve_node_uptime` | `pve_node_uptime_seconds` | Renamed only. |
| `pve_node_zfs_checksum_errors` | `pve_node_zfs_checksum_errors_total` | Type changed to counter. |
| `pve_node_zfs_read_errors` | `pve_node_zfs_read_errors_total` | Type changed to counter. |
| `pve_node_zfs_write_errors` | `pve_node_zfs_write_errors_total` | Type changed to counter. |
| `pve_vm_cpu_usage` | `pve_vm_cpu_usage_ratio` | Values unchanged, they are already a ratio between `0` and `1`. |
| `pve_vm_cpu_count` | `pve_vm_cpus` | Renamed only. |
| `pve_vm_disk_failed_rd_ops` | `pve_vm_disk_failed_read_operations_total` | Type changed to counter. |
| `pve_vm_disk_failed_wr_ops` | `pve_vm_disk_failed_write_operations_total` | Type changed to counter. |
| `pve_vm_disk_rd_bytes` | `pve_vm_disk_read_bytes_total` | Type changed to counter. |
| `pve_vm_disk_rd_operations` | `pve_vm_disk_read_operations_total` | Type changed to counter. |
| `pve_vm_disk_rd_time_total_ns` | `pve_vm_disk_read_time_seconds_total` | Values converted from nanoseconds to seconds, type changed to counter. |
| `pve_vm_disk_usage_bytes` | `pve_vm_disk_used_bytes` | Renamed only. |
| `pve_vm_disk_wr_bytes` | `pve_vm_disk_write_bytes_total` | Type changed to counter. |
| `pve_vm_disk_wr_operations` | `pve_vm_disk_write_operations_total` | Type changed to counter. |
| `pve_vm_disk_wr_time_total_ns` | `pve_vm_disk_write_time_seconds_total` | Values converted from nanoseconds to seconds, type changed to counter. |
| `pve_vm_mem_total_bytes` | `pve_vm_memory_total_bytes` | Renamed only. |
| `pve_vm_mem_used_bytes` | `pve_vm_memory_used_bytes` | Renamed only. |
| `pve_vm_network_in_bytes` | `pve_vm_network_receive_bytes_total` | Type changed to counter. |
| `pve_vm_network_out_bytes` | `pve_vm_network_transmit_bytes_total` | Type changed to counter. |
| `pve_vm_uptime` | `pve_vm_uptime_seconds` | Renamed only. |
Cumulative metrics are now exported as Prometheus counters instead of gauges,
which is what `rate()` and `increase()` expect and what removes the
"metric might not be a counter" hint in Grafana.