lostakj/pve-exporter
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 8 Wiki Activity

Updated docs
Build Docker image on push / docker (push) Successful in 23s
Details

Browse Source
This commit is contained in:
lostakj
2026-08-18 01:13:27 +02:00
parent 2d634470d9
commit b69e10d52f
7 changed files with 654 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/README.md
+25
View File
@@ -0,0 +1,25 @@
# PVE Exporter Documentation
PVE Exporter periodically collects monitoring data from the Proxmox VE HTTP API
and stores the latest values in the Prometheus registry exposed at `/metrics`.
## Guides
- [Configuration reference](configuration.md) describes every YAML setting,
API token permissions, and collector switch.
- [Metrics reference](metrics.md) lists exported metrics, labels, units, and
numeric state mappings.
- [Operations and troubleshooting](operations.md) covers native and container
deployment, endpoint failover, logging, security, and common errors.
- [Architecture and adding collectors](architecture.md) explains the collection
flow and how the main packages fit together.
## Important behavior
- One exporter instance is intended for one standalone node or one PVE cluster.
- Multiple configured API hosts provide redundant access to that same cluster.
- Collection runs every `proxmox.interval` seconds independently of Prometheus
scrapes.
- Collector label sets that stop receiving updates are removed after five
minutes.
- The only application HTTP endpoint is `/metrics`.
docs/architecture.md
+88
View File
@@ -0,0 +1,88 @@
# Architecture and Adding Collectors
## Collection flow
```text
config.yaml
|
v
Application -> PveApiClient -> Proxmox VE API host(s)
| |
| +-> liveness checks, round-robin selection, short API cache
v
PveMetricsManager -> enabled collectors -> Prometheus registry -> /metrics
|
+-> TTL cleanup for stale label sets
```
## Packages
| Package | Responsibility |
| --- | --- |
| `application` | Loads configuration, creates dependencies, registers `/metrics`, and starts the HTTP server. |
| `configuration` | YAML models and startup validation. |
| `proxmox` | HTTP client, Proxmox-specific API methods, response models, and numeric state conversion. |
| `metrics` | Collector implementations, periodic scheduling, Prometheus metric definitions, and TTL cleanup. |
| `utils` | Shared formatting helpers. |
## API client
`PveApiClient` wraps the generic `ApiClient` with typed methods such as cluster
status, node status, guest status, storage, disk, and ZFS calls.
The generic client:
- normalizes API hosts so a trailing slash is optional;
- checks every host at startup and every five seconds;
- selects reachable hosts in round-robin order;
- caches a successful method/path response briefly to avoid duplicate API calls;
- uses API-token authorization and a ten-second HTTP timeout;
- accepts self-signed certificates by disabling TLS verification.
Configured hosts must represent the same cluster because cached results are
shared by method and path rather than host.
## Metrics manager
`PveMetricsManager` creates only the collectors enabled in configuration. It
runs one full collection immediately, then repeats it every
`proxmox.interval` seconds. Collectors execute sequentially.
Each collector implements:
```go
type PveMetricsCollector interface {
CollectMetrics() error
GetName() string
}
```
Successful execution time is recorded in
`pve_metrics_collection_latency_ms`. A collector error is logged and the
manager continues with the next collector.
## TTL metrics
Most project metrics use `TTLGaugeVec`, a wrapper around Prometheus `GaugeVec`.
Every label set records its last update. `TTLRegistry` checks registered metrics
every five seconds and removes series that have not been updated for five
minutes.
This prevents deleted guests or storage resources from remaining indefinitely,
while allowing short API failures to preserve the last known values.
## Adding a collector
1. Add typed response models and API methods under `proxmox`.
2. Add a collector implementing `PveMetricsCollector` under `metrics`.
3. Create metrics with `NewTTLGaugeVec`, choose stable labels, and register each
metric with the shared `TTLRegistry`.
4. Add a boolean switch to `PveMetricsConfiguration` and `config.example.yaml`.
5. Register the collector in `NewPveMetricsManager`.
6. Add parsing, state-mapping, and topology tests as appropriate.
7. Document the switch, labels, units, and state values in
[configuration.md](configuration.md) and [metrics.md](metrics.md).
Avoid labels containing changing messages, timestamps, or other unbounded
values. For recursive resources such as ZFS, include a stable path so repeated
component names remain distinguishable.
docs/configuration.md
+113
View File
@@ -0,0 +1,113 @@
# Configuration Reference
The exporter reads `config.yaml` from the current working directory by default.
Use `-config` to select a different file:
```sh
./pve-exporter -config /etc/pve-exporter/config.yaml
```
## Complete example
```yaml
host: 0.0.0.0
port: 9090
logLevel: 4
proxmox:
token:
tokenId: monitoring@pve!exporter
secret: change-me
hosts:
- https://192.168.0.10:8006
- https://pve02.example.com:8006/
interval: 15
metrics:
clusterState: true
nodeStatus: true
qemu: true
lxc: true
disk: true
zfs: true
storage: true
subscription: true
sdn: true
```
## Server settings
| Setting | Type | Description |
| --- | --- | --- |
| `host` | IP address | Address on which the exporter HTTP server listens, for example `0.0.0.0` or `127.0.0.1`. DNS names are not accepted here. |
| `port` | integer | Exporter HTTP port. Prometheus metrics are served at `/metrics`. |
| `logLevel` | integer | Logrus level: `0=Panic`, `1=Fatal`, `2=Error`, `3=Warn`, `4=Info`, `5=Debug`, `6=Trace`. |
## Proxmox token
| Setting | Type | Description |
| --- | --- | --- |
| `proxmox.token.tokenId` | string | Full Proxmox API token ID, normally `user@realm!token-name`. |
| `proxmox.token.secret` | string | API token secret. Treat the configuration file as sensitive. |
Create a dedicated monitoring user and token, then assign the read-only
`PVEAuditor` role to both the user and the token. The role supplies the audit
permissions used by cluster, node, storage, disk, ZFS, guest, and subscription
API calls.
## API hosts and failover
`proxmox.hosts` is a non-empty list of Proxmox API base URLs. Both IP addresses
and DNS names are supported:
```yaml
proxmox:
hosts:
- https://192.168.0.10:8006
- https://pve02.example.com:8006/
```
A trailing slash is optional and normalized automatically. Every host must use
`http` or `https` and must belong to the same PVE cluster. The exporter checks
endpoint liveness every five seconds and distributes API requests across the
currently reachable endpoints.
Do not combine unrelated clusters in one list. Their API responses share the
same short-lived client cache and Prometheus registry. Run a separate exporter
for each cluster.
## Collection interval
`proxmox.interval` is the positive number of seconds between collection cycles.
The first cycle runs during application startup. Prometheus scrapes only the
values already present in the registry, so its scrape interval does not control
the PVE API request rate.
## Collector switches
All switches are booleans. Omitted switches default to `false`.
| Setting | Collector | Main metric prefix |
| --- | --- | --- |
| `clusterState` | Cluster mode, node count, and quorum | `pve_cluster_` |
| `nodeStatus` | Node health and resource usage | `pve_node_` |
| `qemu` | QEMU virtual machines | `pve_vm_` |
| `lxc` | LXC containers | `pve_ct_` |
| `disk` | Physical disk health and metadata | `pve_node_disk_` |
| `zfs` | ZFS pool topology, state, and errors | `pve_node_zfs_` |
| `storage` | Enabled PVE storage status and capacity | `pve_storage_` |
| `subscription` | Node subscription information | `pve_node_subscription_` |
| `sdn` | Software-defined network state | `pve_sdn_` |
The exporter-level `pve_metrics_collection_latency_ms` summary is always
registered.
## Validation notes
- `host` must be a literal IP address.
- At least one Proxmox API host is required.
- API host URLs require an `http` or `https` scheme and an IP or DNS hostname.
- Token ID and secret cannot be empty.
- Keep `proxmox.interval` greater than zero and `logLevel` between `0` and `6`.
docs/metrics.md
+192
View File
@@ -0,0 +1,192 @@
# Metrics Reference
All exporter-specific metric names start with `pve_`. The Go Prometheus client
also exposes its standard `go_*`, `process_*`, and HTTP handler metrics.
## Common labels and lifecycle
Most resource metrics use these labels:
| Label | Description |
| --- | --- |
| `cluster` | PVE cluster name. A standalone installation uses `Standalone node - <node>`. |
| `node` | Proxmox node name. |
| `vmid` | QEMU VM or LXC container ID. |
| `name` | Guest name. |
Metric label sets are retained for five minutes after their last update and are
then removed. Consequently, metrics available only while a guest is running can
remain visible for up to five minutes after it stops.
## Exporter
| Metric | Labels | Description |
| --- | --- | --- |
| `pve_metrics_collection_latency_ms` | `collector` | Summary of successful collector execution time in milliseconds. |
## Cluster state
Enabled with `metrics.clusterState`. Labels: `cluster`.
| Metric | Description |
| --- | --- |
| `pve_cluster_mode` | `1` when PVE is configured as a cluster, `0` for a standalone node. A configured one-node cluster still reports `1`. |
| `pve_cluster_nodes` | Number of nodes reported by the cluster status record. Standalone mode reports `0`. |
| `pve_cluster_quorate` | Cluster quorum state: `1` quorate, `0` not quorate or standalone. |
## Node status
Enabled with `metrics.nodeStatus`. Unless noted otherwise, labels are `cluster`
and `node`.
| Metric | Description |
| --- | --- |
| `pve_node_state` | Node online state: `1` online, `0` offline. |
| `pve_node_uptime` | Node uptime in seconds. |
| `pve_node_cpu_count` | Number of logical CPUs. |
| `pve_node_cpu_usage` | CPU usage reported by PVE. |
| `pve_node_memory_total_bytes` | Total memory in bytes. |
| `pve_node_memory_used_bytes` | Used memory in bytes. |
| `pve_node_memory_free_bytes` | Free memory in bytes. |
| `pve_node_ksm_bytes` | Memory shared by Kernel Same-page Merging in bytes. |
| `pve_node_cgroup_mode` | PVE cgroup mode. |
| `pve_node_load1` | One-minute load average. |
| `pve_node_load5` | Five-minute load average. |
| `pve_node_load15` | Fifteen-minute load average. |
| `pve_node_rootfs_free_bytes` | Root filesystem free bytes. |
| `pve_node_rootfs_used_bytes` | Root filesystem used bytes. |
| `pve_node_rootfs_total_bytes` | Root filesystem total bytes. |
| `pve_node_rootfs_avail_bytes` | Root filesystem bytes available to unprivileged processes. |
| `pve_node_time` | Node UTC Unix timestamp. |
| `pve_node_localtime` | Node local Unix timestamp. |
| `pve_node_cpuinfo` | Constant `1` carrying `flags`, `cores`, `model`, `sockets`, `cpus`, and `hvm` labels. |
| `pve_node_systeminfo` | Constant `1` carrying `kversion`, `pveversion`, `machine`, `sysname`, and `release` labels. |
## Physical disks
Enabled with `metrics.disk`. Labels are `cluster`, `node`, `wwn`, `type`,
`model`, `serial`, `vendor`, `used`, and `osd_id`.
| Metric | Description |
| --- | --- |
| `pve_node_disk_healthy` | SMART health state: `1` for `OK` or `PASSED`, otherwise `0`. |
| `pve_node_disk_wearout` | Device wearout percentage when supplied by the PVE API. |
| `pve_node_disk_size_bytes` | Physical disk size in bytes. |
## ZFS
Enabled with `metrics.zfs`. The exporter discovers all pools on every node,
retrieves each pool detail, and recursively flattens pool, vdev, section, cache,
and leaf-device entries.
Labels are `cluster`, `node`, `pool`, `component`, `path`, and `leaf`.
`component` is the current topology entry, `path` identifies its complete
hierarchy, and `leaf` is `true` for a device entry.
| Metric | Description |
| --- | --- |
| `pve_node_zfs_state` | Numeric component state: `0=UNKNOWN`, `1=ONLINE`, `2=DEGRADED`, `3=FAULTED`, `4=OFFLINE`, `5=REMOVED`, `6=UNAVAIL`. |
| `pve_node_zfs_read_errors` | ZFS read error count reported for the component. |
| `pve_node_zfs_write_errors` | ZFS write error count reported for the component. |
| `pve_node_zfs_checksum_errors` | ZFS checksum error count reported for the component. |
ZFS section entries that do not contain a counter do not produce a false zero
series. A counter explicitly returned as zero is exported normally.
Example alert expressions:
```promql
pve_node_zfs_state != 1
```
```promql
(pve_node_zfs_read_errors > 0)
or (pve_node_zfs_write_errors > 0)
or (pve_node_zfs_checksum_errors > 0)
```
## PVE storage
Enabled with `metrics.storage`. Disabled storages are skipped. Labels are
`cluster`, `node`, `storage`, `type`, `content`, and `shared`.
| Metric | Description |
| --- | --- |
| `pve_storage_up` | Storage active state: `1` active, `0` inactive. |
| `pve_storage_total_bytes` | Total storage capacity in bytes. |
| `pve_storage_avail_bytes` | Available storage capacity in bytes. |
| `pve_storage_used_bytes` | Used storage capacity in bytes. |
## Node subscriptions
Enabled with `metrics.subscription`. The common labels are `cluster` and
`node`; `pve_node_subscription_info` also has `productname` and `serverid`.
| Metric | Description |
| --- | --- |
| `pve_node_subscription_info` | Constant `1` carrying subscription product and server identifiers. |
| `pve_node_subscription_status` | `0` for API status `notfound`, `1` for `active`, `2` for expired or otherwise unusable states such as `invalid`, `suspended`, and `new`. |
| `pve_node_subscription_regdate` | Subscription registration date as a Unix timestamp, when available. |
| `pve_node_subscription_nextduedate` | Next due date as a Unix timestamp, when available. |
| `pve_node_subscription_sockets` | Number of covered sockets reported by PVE. |
The PVE API cannot distinguish a node that never had a subscription from one
whose key was removed; both appear as `notfound` and therefore use value `0`.
## SDN
Enabled with `metrics.sdn`. Labels are `cluster`, `node`, `sdn`, and `sdn_id`.
| Metric | Description |
| --- | --- |
| `pve_sdn_state` | `1` when the SDN resource status is `ok`, otherwise `0`. |
## LXC containers
Enabled with `metrics.lxc`. Templates are skipped. Labels are `cluster`,
`node`, `vmid`, and `name`.
| Metric | Description |
| --- | --- |
| `pve_ct_state` | `1` running, `0` stopped. |
| `pve_ct_uptime` | Uptime in seconds; updated only while running. |
| `pve_ct_cpu_count` | Configured CPU count. |
| `pve_ct_cpu_usage` | CPU usage reported by PVE; updated only while running. |
| `pve_ct_mem_total_bytes` | Configured memory limit in bytes. |
| `pve_ct_mem_used_bytes` | Used memory in bytes; updated only while running. |
| `pve_ct_network_in_bytes` | Received bytes since container start; updated only while running. |
| `pve_ct_network_out_bytes` | Transmitted bytes since container start; updated only while running. |
| `pve_ct_disk_rd_bytes` | Disk bytes read; updated only while running. |
| `pve_ct_disk_wr_bytes` | Disk bytes written; updated only while running. |
| `pve_ct_disk_usage_bytes` | Used root disk bytes; updated only while running. |
| `pve_ct_disk_size_bytes` | Configured root disk size in bytes. |
| `pve_ct_swap_used_bytes` | Used swap in bytes; updated only while running. |
## QEMU virtual machines
Enabled with `metrics.qemu`. Templates are skipped. Base labels are `cluster`,
`node`, `vmid`, and `name`.
| Metric | Extra label | Description |
| --- | --- | --- |
| `pve_vm_state` | none | `1` running, `0` stopped. |
| `pve_vm_uptime` | none | Uptime in seconds; updated only while running. |
| `pve_vm_agent` | none | QEMU guest agent state reported by PVE; updated only while running. |
| `pve_vm_cpu_count` | none | Configured CPU count. |
| `pve_vm_cpu_usage` | none | CPU usage reported by PVE; updated only while running. |
| `pve_vm_mem_total_bytes` | none | Configured maximum memory in bytes. |
| `pve_vm_mem_used_bytes` | none | Used memory in bytes; updated only while running. |
| `pve_vm_disk_usage_bytes` | none | Root disk usage reported by PVE. |
| `pve_vm_disk_size_bytes` | none | Configured maximum root disk size in bytes. |
| `pve_vm_network_in_bytes` | `interface` | Bytes received by a virtual interface. |
| `pve_vm_network_out_bytes` | `interface` | Bytes transmitted by a virtual interface. |
| `pve_vm_disk_rd_operations` | `device` | Successful read operations. |
| `pve_vm_disk_wr_operations` | `device` | Successful write operations. |
| `pve_vm_disk_rd_bytes` | `device` | Bytes read from the block device. |
| `pve_vm_disk_wr_bytes` | `device` | Bytes written to the block device. |
| `pve_vm_disk_failed_rd_ops` | `device` | Failed read operations. |
| `pve_vm_disk_failed_wr_ops` | `device` | Failed write operations. |
| `pve_vm_disk_rd_time_total_ns` | `device` | Total block-device read time in nanoseconds. |
| `pve_vm_disk_wr_time_total_ns` | `device` | Total block-device write time in nanoseconds. |
Interface and block-device metrics are collected only for running VMs.
docs/operations.md
+133
View File
@@ -0,0 +1,133 @@
# Operations and Troubleshooting
## Native execution
Build and run from the repository root:
```sh
go build -o pve-exporter .
./pve-exporter -config config.yaml
```
The default configuration path is `config.yaml`. The HTTP listener is formed
from the top-level `host` and `port` settings. Only `/metrics` is registered.
Verify the exporter locally:
```sh
curl http://localhost:9090/metrics
```
## Docker
Pull and run the published image:
```sh
docker pull gitea.lostak.dev/lostakj/pve-exporter:latest
docker run --rm -p 9090:9090 \
-v "$PWD/config.yaml:/opt/config.yaml:ro" \
gitea.lostak.dev/lostakj/pve-exporter:latest
```
The image contains the example configuration at `/opt/config.yaml`, but that
file has no API hosts or real credentials. Mount a configured file at the same
path for normal use.
## Prometheus
When Prometheus and the exporter share a container network:
```yaml
scrape_configs:
- job_name: pve-exporter
scrape_interval: 30s
static_configs:
- targets: [pve-exporter:9090]
```
The Prometheus scrape interval and `proxmox.interval` are independent. A scrape
does not contact PVE; it reads the latest values collected in the background.
## API endpoint availability
At startup and every five seconds, the exporter requests `api2/json/` from each
configured PVE host. Requests are distributed round-robin across endpoints that
passed the latest check.
Use multiple hosts only for nodes in the same cluster:
```yaml
proxmox:
hosts:
- https://192.168.0.10:8006
- https://pve02.example.com:8006
```
IP and DNS hosts work with or without a trailing slash. If every endpoint is
unavailable, collectors log `All API endpoints are unreachable.` and retain
their previous series until the five-minute stale-metric timeout removes them.
## Logging
Set `logLevel` between `0` and `6`:
| Value | Level |
| --- | --- |
| `0` | Panic |
| `1` | Fatal |
| `2` | Error |
| `3` | Warn |
| `4` | Info |
| `5` | Debug |
| `6` | Trace |
`Info` is suitable for normal operation. `Debug` or `Trace` can help diagnose
endpoint availability and individual collector execution.
## Common problems
### Authentication failed
Check that `tokenId` contains the full `user@realm!token-name` identifier and
that `secret` contains the token secret rather than the user password. Assign
the `PVEAuditor` role to both the user and token.
### Permission or empty cluster-status errors
The `/cluster/status` endpoint is used by every collector to identify nodes and
the cluster label. Ensure the token can audit `/` and the relevant nodes. The
exporter reports a specific permission hint when the endpoint returns an empty
array.
### No metrics from a collector
Confirm that its boolean switch under `proxmox.metrics` is `true`. Some series
are intentionally conditional:
- VM and LXC templates are skipped.
- Runtime guest metrics are updated only while the guest is running.
- Disabled PVE storages are skipped.
- Optional ZFS counters are omitted when the API does not return them.
- Subscription dates are omitted when they are unavailable.
### ZFS collection errors
Enable `proxmox.metrics.zfs`, confirm that the node has ZFS utilities and pools,
and verify that the token can call both `/nodes/{node}/disks/zfs` and each pool
detail endpoint.
### Metrics remain after a resource disappears
Collector metrics use a five-minute TTL. A guest, disk, storage, or ZFS topology
entry that disappears can therefore remain visible for up to five minutes.
## Security notes
- Keep `config.yaml` readable only by the exporter account because it contains
the API token secret.
- The HTTP endpoint has no authentication; bind to a protected interface or
restrict access with a firewall or reverse proxy.
- The PVE client currently disables TLS certificate verification to support the
default self-signed certificates. Use the exporter only across trusted
networks or a protected tunnel.
- Prefer the read-only `PVEAuditor` role and do not use a root token.