199 lines
8.9 KiB
Markdown
199 lines
8.9 KiB
Markdown
# Metrics Reference
|
|
|
|
All exporter-specific metric names start with `pve_`. The Go Prometheus client
|
|
also exposes its standard `go_*`, `process_*`, and HTTP handler metrics.
|
|
|
|
## Common labels and lifecycle
|
|
|
|
Most resource metrics use these labels:
|
|
|
|
| Label | Description |
|
|
| --- | --- |
|
|
| `cluster` | PVE cluster name. A standalone installation uses `Standalone node - <node>`. |
|
|
| `node` | Proxmox node name. |
|
|
| `vmid` | QEMU VM or LXC container ID. |
|
|
| `name` | Guest name. |
|
|
|
|
Metric label sets are retained for five minutes after their last update and are
|
|
then removed. Consequently, metrics available only while a guest is running can
|
|
remain visible for up to five minutes after it stops.
|
|
|
|
## Exporter
|
|
|
|
| Metric | Labels | Description |
|
|
| --- | --- | --- |
|
|
| `pve_metrics_collection_latency_ms` | `collector` | Summary of successful collector execution time in milliseconds. |
|
|
|
|
## Cluster state
|
|
|
|
Enabled with `metrics.clusterState`. Labels: `cluster`.
|
|
|
|
| Metric | Description |
|
|
| --- | --- |
|
|
| `pve_cluster_mode` | `1` when PVE is configured as a cluster, `0` for a standalone node. A configured one-node cluster still reports `1`. |
|
|
| `pve_cluster_nodes` | Number of nodes reported by the cluster status record. Standalone mode reports `0`. |
|
|
| `pve_cluster_quorate` | Cluster quorum state: `1` quorate, `0` not quorate or standalone. |
|
|
|
|
## Node status
|
|
|
|
Enabled with `metrics.nodeStatus`. Unless noted otherwise, labels are `cluster`
|
|
and `node`.
|
|
|
|
| Metric | Description |
|
|
| --- | --- |
|
|
| `pve_node_state` | Node online state: `1` online, `0` offline. |
|
|
| `pve_node_uptime` | Node uptime in seconds. |
|
|
| `pve_node_cpu_count` | Number of logical CPUs. |
|
|
| `pve_node_cpu_usage` | CPU usage reported by PVE. |
|
|
| `pve_node_memory_total_bytes` | Total memory in bytes. |
|
|
| `pve_node_memory_used_bytes` | Used memory in bytes. |
|
|
| `pve_node_memory_free_bytes` | Free memory in bytes. |
|
|
| `pve_node_ksm_bytes` | Memory shared by Kernel Same-page Merging in bytes. |
|
|
| `pve_node_cgroup_mode` | PVE cgroup mode. |
|
|
| `pve_node_load1` | One-minute load average. |
|
|
| `pve_node_load5` | Five-minute load average. |
|
|
| `pve_node_load15` | Fifteen-minute load average. |
|
|
| `pve_node_rootfs_free_bytes` | Root filesystem free bytes. |
|
|
| `pve_node_rootfs_used_bytes` | Root filesystem used bytes. |
|
|
| `pve_node_rootfs_total_bytes` | Root filesystem total bytes. |
|
|
| `pve_node_rootfs_avail_bytes` | Root filesystem bytes available to unprivileged processes. |
|
|
| `pve_node_time` | Node UTC Unix timestamp. |
|
|
| `pve_node_localtime` | Node local Unix timestamp. |
|
|
| `pve_node_cpuinfo` | Constant `1` carrying `flags`, `cores`, `model`, `sockets`, `cpus`, and `hvm` labels. |
|
|
| `pve_node_systeminfo` | Constant `1` carrying `kversion`, `pveversion`, `machine`, `sysname`, and `release` labels. |
|
|
|
|
## Physical disks
|
|
|
|
Enabled with `metrics.disk`. Labels are `cluster`, `node`, `wwn`, `type`,
|
|
`model`, `serial`, `vendor`, `used`, and `osd_id`.
|
|
|
|
| Metric | Description |
|
|
| --- | --- |
|
|
| `pve_node_disk_healthy` | SMART health state: `1` for `OK` or `PASSED`, otherwise `0`. |
|
|
| `pve_node_disk_wearout` | Device wearout percentage when supplied by the PVE API. |
|
|
| `pve_node_disk_size_bytes` | Physical disk size in bytes. |
|
|
|
|
## ZFS
|
|
|
|
Enabled with `metrics.zfs`. The exporter discovers all pools on every node,
|
|
retrieves each pool detail, and recursively flattens pool, vdev, section, cache,
|
|
and leaf-device entries.
|
|
|
|
Labels are `cluster`, `node`, `pool`, `component`, `path`, and `leaf`.
|
|
`component` is the current topology entry, `path` identifies its complete
|
|
hierarchy, and `leaf` is `true` for a device entry.
|
|
|
|
| Metric | Description |
|
|
| --- | --- |
|
|
| `pve_node_zfs_state` | Numeric component state: `0=UNKNOWN`, `1=ONLINE`, `2=DEGRADED`, `3=FAULTED`, `4=OFFLINE`, `5=REMOVED`, `6=UNAVAIL`. |
|
|
| `pve_node_zfs_read_errors` | ZFS read error count reported for the component. |
|
|
| `pve_node_zfs_write_errors` | ZFS write error count reported for the component. |
|
|
| `pve_node_zfs_checksum_errors` | ZFS checksum error count reported for the component. |
|
|
|
|
ZFS section entries that do not contain a counter do not produce a false zero
|
|
series. A counter explicitly returned as zero is exported normally.
|
|
|
|
Example alert expressions:
|
|
|
|
```promql
|
|
pve_node_zfs_state != 1
|
|
```
|
|
|
|
```promql
|
|
(pve_node_zfs_read_errors > 0)
|
|
or (pve_node_zfs_write_errors > 0)
|
|
or (pve_node_zfs_checksum_errors > 0)
|
|
```
|
|
|
|
## PVE storage
|
|
|
|
Enabled with `metrics.storage`. Disabled storages are skipped. Labels are
|
|
`cluster`, `node`, `storage`, `type`, `content`, and `shared`.
|
|
|
|
| Metric | Description |
|
|
| --- | --- |
|
|
| `pve_storage_up` | Storage active state: `1` active, `0` inactive. |
|
|
| `pve_storage_total_bytes` | Total storage capacity in bytes. |
|
|
| `pve_storage_avail_bytes` | Available storage capacity in bytes. |
|
|
| `pve_storage_used_bytes` | Used storage capacity in bytes. |
|
|
|
|
## Node subscriptions
|
|
|
|
Enabled with `metrics.subscription`. The common labels are `cluster` and
|
|
`node`; `pve_node_subscription_info` also has `productname` and `serverid`.
|
|
|
|
| Metric | Description |
|
|
| --- | --- |
|
|
| `pve_node_subscription_info` | Constant `1` carrying subscription product and server identifiers. |
|
|
| `pve_node_subscription_status` | `0` for API status `notfound`, `1` for `active`, `2` for expired or otherwise unusable states such as `invalid`, `suspended`, and `new`. |
|
|
| `pve_node_subscription_regdate` | Subscription registration date as a Unix timestamp, when available. |
|
|
| `pve_node_subscription_nextduedate` | Next due date as a Unix timestamp, when available. |
|
|
| `pve_node_subscription_sockets` | Number of covered sockets reported by PVE. |
|
|
|
|
The PVE API cannot distinguish a node that never had a subscription from one
|
|
whose key was removed; both appear as `notfound` and therefore use value `0`.
|
|
|
|
## SDN
|
|
|
|
Enabled with `metrics.sdn`. Labels are `cluster`, `node`, `sdn`, and `sdn_id`.
|
|
|
|
| Metric | Description |
|
|
| --- | --- |
|
|
| `pve_sdn_state` | `1` when the SDN resource status is `ok`, otherwise `0`. |
|
|
|
|
PVE 8 reports SDN zones as `sdn` cluster resources, PVE 9 reports them as
|
|
`network` resources (zones and fabrics). Both are exported as `pve_sdn_state`;
|
|
the `sdn` label holds the zone or fabric name and `sdn_id` the resource ID,
|
|
which differs between the two formats (`sdn/<node>/<zone>` versus
|
|
`network/<node>/<network-type>/<name>`).
|
|
|
|
## LXC containers
|
|
|
|
Enabled with `metrics.lxc`. Templates are skipped. Labels are `cluster`,
|
|
`node`, `vmid`, and `name`.
|
|
|
|
| Metric | Description |
|
|
| --- | --- |
|
|
| `pve_ct_state` | `1` running, `0` stopped. |
|
|
| `pve_ct_uptime` | Uptime in seconds; updated only while running. |
|
|
| `pve_ct_cpu_count` | Configured CPU count. |
|
|
| `pve_ct_cpu_usage` | CPU usage reported by PVE; updated only while running. |
|
|
| `pve_ct_mem_total_bytes` | Configured memory limit in bytes. |
|
|
| `pve_ct_mem_used_bytes` | Used memory in bytes; updated only while running. |
|
|
| `pve_ct_network_in_bytes` | Received bytes since container start; updated only while running. |
|
|
| `pve_ct_network_out_bytes` | Transmitted bytes since container start; updated only while running. |
|
|
| `pve_ct_disk_rd_bytes` | Disk bytes read; updated only while running. |
|
|
| `pve_ct_disk_wr_bytes` | Disk bytes written; updated only while running. |
|
|
| `pve_ct_disk_usage_bytes` | Used root disk bytes; updated only while running. |
|
|
| `pve_ct_disk_size_bytes` | Configured root disk size in bytes. |
|
|
| `pve_ct_swap_used_bytes` | Used swap in bytes; updated only while running. |
|
|
|
|
## QEMU virtual machines
|
|
|
|
Enabled with `metrics.qemu`. Templates are skipped. Base labels are `cluster`,
|
|
`node`, `vmid`, and `name`.
|
|
|
|
| Metric | Extra label | Description |
|
|
| --- | --- | --- |
|
|
| `pve_vm_state` | none | `1` running, `0` stopped. |
|
|
| `pve_vm_uptime` | none | Uptime in seconds; updated only while running. |
|
|
| `pve_vm_agent` | none | QEMU guest agent state reported by PVE; updated only while running. |
|
|
| `pve_vm_cpu_count` | none | Configured CPU count. |
|
|
| `pve_vm_cpu_usage` | none | CPU usage reported by PVE; updated only while running. |
|
|
| `pve_vm_mem_total_bytes` | none | Configured maximum memory in bytes. |
|
|
| `pve_vm_mem_used_bytes` | none | Used memory in bytes; updated only while running. |
|
|
| `pve_vm_disk_usage_bytes` | none | Root disk usage reported by PVE. |
|
|
| `pve_vm_disk_size_bytes` | none | Configured maximum root disk size in bytes. |
|
|
| `pve_vm_network_in_bytes` | `interface` | Bytes received by a virtual interface. |
|
|
| `pve_vm_network_out_bytes` | `interface` | Bytes transmitted by a virtual interface. |
|
|
| `pve_vm_disk_rd_operations` | `device` | Successful read operations. |
|
|
| `pve_vm_disk_wr_operations` | `device` | Successful write operations. |
|
|
| `pve_vm_disk_rd_bytes` | `device` | Bytes read from the block device. |
|
|
| `pve_vm_disk_wr_bytes` | `device` | Bytes written to the block device. |
|
|
| `pve_vm_disk_failed_rd_ops` | `device` | Failed read operations. |
|
|
| `pve_vm_disk_failed_wr_ops` | `device` | Failed write operations. |
|
|
| `pve_vm_disk_rd_time_total_ns` | `device` | Total block-device read time in nanoseconds. |
|
|
| `pve_vm_disk_wr_time_total_ns` | `device` | Total block-device write time in nanoseconds. |
|
|
|
|
Interface and block-device metrics are collected only for running VMs.
|