lostakj/pve-exporter
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 8 Wiki Activity

Updated docs
Build Docker image on push / docker (push) Successful in 23s
Details

Browse Source
This commit is contained in:
lostakj
2026-08-18 01:13:27 +02:00
parent 2d634470d9
commit b69e10d52f
7 changed files with 654 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+102 -80
View File
@@ -1,125 +1,147 @@
# PVE Exporter
Proxmox Virtual Environment Prometheus metrics exporter.
Prometheus exporter for Proxmox Virtual Environment. It periodically reads the
Proxmox VE HTTP API and exposes cluster, node, guest, storage, disk, ZFS,
subscription, and SDN metrics through a single `/metrics` endpoint.
## Overview
PVE Exporter is a tool that collects metrics from a Proxmox Virtual Environment cluster and exposes them for Prometheus to scrape. This exporter supports gathering metrics for cluster state, LXC containers, QEMU virtual machines, physical disks, ZFS pools, node storage, node status, node subscription details, and software-defined networking (SDN).
The exporter supports standalone nodes and multi-node clusters. Multiple API
hosts can be configured for endpoint failover within one Proxmox cluster.
## Features
- Collects various metrics from Proxmox VE to monitor the health and performance of your virtual environment.
- Supports multi-node clusters for high availability.
- Securely uses Proxmox API tokens with minimal permissions.
- Standalone-node and cluster mode detection, node count, and quorum state.
- Node CPU, memory, load, root filesystem, uptime, time, and system information.
- QEMU virtual machine and LXC container state and resource metrics.
- Physical disk health, wearout, and size metrics.
- Recursive ZFS pool, vdev, cache, and leaf-device state and error metrics.
- Storage capacity and availability, subscription, and SDN state metrics.
- Selectively enabled collectors with automatic cleanup of stale label sets.
- Multiple API endpoints with periodic liveness checks and round-robin requests.
- Proxmox API token authentication using the read-only `PVEAuditor` role.
## Prerequisites
## Quick start
- Docker installed on your machine.
- Access to a Proxmox VE instance with API tokens configured.
Copy the example configuration and set the Proxmox API hosts and token:
## Configuration
### Proxmox API Token
When generating the token, make sure to assign the 'PVEAuditor' permission to both the user and the API token. For security reasons, assign only the 'PVEAuditor' role to limit permissions appropriately.
```yaml
# Proxmox API token configuration.
token:
tokenId: "your-token-id"
secret: "your-secret"
```sh
cp config.example.yaml config.yaml
```
### Proxmox API Hosts
If you are running a multi-node cluster, add multiple API hosts to ensure high availability of metrics. Note that this configuration is not intended for gathering metrics from multiple PVE clusters. For multiple PVE clusters, deploy a separate exporter instance for each cluster.
```yaml
# Proxmox API hosts.
hosts:
- "https://host1.example.com"
- "https://host2.example.com"
host: 0.0.0.0
port: 9090
logLevel: 4
proxmox:
token:
tokenId: monitoring@pve!exporter
secret: change-me
hosts:
- https://pve01.example.com:8006
- https://pve02.example.com:8006
interval: 15
metrics:
clusterState: true
nodeStatus: true
qemu: true
lxc: true
disk: true
zfs: true
storage: true
subscription: true
sdn: true
```
### Metrics Configuration
Assign the `PVEAuditor` role to both the Proxmox user and API token. All entries
under `hosts` must belong to the same cluster. IP addresses and DNS names are
supported, with or without a trailing slash.
Configure which metrics to collect by enabling or disabling specific metric types.
Build and start the exporter:
```yaml
# Proxmox metrics configuration.
metrics:
clusterState: true # Enable collection of cluster state metrics.
ltc: true # Enable collection of LXC container metrics.
qemu: true # Enable collection of QEMU virtual machine metrics.
disk: true # Enable collection of physical disk metrics.
zfs: true # Enable collection of ZFS pool metrics.
storage: true # Enable collection of node storage metrics.
nodeStatus: true # Enable collection of node status metrics.
subscription: true # Enable collection of node subscription details.
sdn: true # Enable collection of software-defined network (SDN) metrics.
```sh
go build -o pve-exporter .
./pve-exporter -config config.yaml
```
The cluster state collector exposes `pve_cluster_mode`, where `1` means that
Proxmox VE is configured as a cluster and `0` means that it is a standalone node.
The exporter exposes Prometheus metrics at:
The `pve_node_subscription_status` metric uses `0` when no subscription is
configured, `1` for an active subscription, and `2` for an expired or otherwise
unusable subscription.
```text
http://localhost:9090/metrics
```
The ZFS collector discovers all pools on every node and exports their recursive
topology with `cluster`, `node`, `pool`, `component`, `path`, and `leaf` labels:
## Documentation
- `pve_node_zfs_state` (`0=UNKNOWN`, `1=ONLINE`, `2=DEGRADED`, `3=FAULTED`,
`4=OFFLINE`, `5=REMOVED`, `6=UNAVAIL`)
- `pve_node_zfs_read_errors`
- `pve_node_zfs_write_errors`
- `pve_node_zfs_checksum_errors`
- [Documentation index](docs/README.md)
- [Configuration reference](docs/configuration.md)
- [Metrics reference](docs/metrics.md)
- [Operations and troubleshooting](docs/operations.md)
- [Architecture and adding collectors](docs/architecture.md)
## Pull the Latest Image
## Prometheus configuration
The latest published version can be downloaded from the Gitea container registry:
```yaml
scrape_configs:
- job_name: pve-exporter
static_configs:
- targets: [pve-exporter:9090]
```
Metrics are collected in the background according to `proxmox.interval`.
Prometheus scrapes the most recently collected values and does not trigger a new
request to Proxmox VE.
## Docker
Pull the latest published image from the Gitea container registry:
```sh
docker pull gitea.lostak.dev/lostakj/pve-exporter:latest
```
## Build Locally
To build the Docker image for PVE Exporter, use the following command:
Or build it locally:
```sh
docker build --rm -t gitea.lostak.dev/lostakj/pve-exporter:latest .
docker build --rm -t pve-exporter:latest .
```
## Run
To run the downloaded or locally built image, use the following command:
Run the published image with a local configuration file:
```sh
docker run --rm -d -p 9090:9090 --name pve-exporter gitea.lostak.dev/lostakj/pve-exporter:latest
docker run --rm -p 9090:9090 \
-v "$PWD/config.yaml:/opt/config.yaml:ro" \
gitea.lostak.dev/lostakj/pve-exporter:latest
```
## Usage
Once the Docker container is running, the exporter will be available on port `9090`. You can configure Prometheus to scrape metrics from the exporter by adding a new scrape configuration to your Prometheus configuration file.
Example Prometheus scrape configuration:
Example with Docker Compose:
```yaml
scrape_configs:
- job_name: 'pve-exporter'
static_configs:
- targets: ['localhost:9090']
services:
pve-exporter:
image: gitea.lostak.dev/lostakj/pve-exporter:latest
ports:
- "9090:9090"
volumes:
- ./config.yaml:/opt/config.yaml:ro
restart: unless-stopped
```
## Scope
One exporter instance monitors one standalone Proxmox VE installation or one
cluster. Configure multiple `hosts` only for redundant access to that same
cluster. Deploy a separate exporter instance for every additional cluster.
The published workflow currently builds the container image for `linux/amd64`.
The API client accepts self-signed Proxmox certificates because TLS certificate
verification is disabled; use trusted networks and protect the API token.
## License
This project is licensed under the MIT License. See the LICENSE.txt file for details.
This project is licensed under the [MIT License](LICENSE).
## Contributing
Contributions are welcome! Please open an issue or submit a pull request if you have any improvements or bug fixes.
## Support
If you encounter any issues or have questions, please open an issue on the project's GitHub repository.
Contributions and bug reports are welcome through the project issue tracker.
config.example.yaml
+1 -1
View File
@@ -22,7 +22,7 @@ proxmox:
# For multiple PVE clusters, deploy a separate exporter instance for each cluster.
hosts: []
# Scrape interval in seconds.
# Metrics collection interval in seconds.
interval: 15
# Proxmox metrics configuration.
docs/README.md
+25
View File
@@ -0,0 +1,25 @@
# PVE Exporter Documentation
PVE Exporter periodically collects monitoring data from the Proxmox VE HTTP API
and stores the latest values in the Prometheus registry exposed at `/metrics`.
## Guides
- [Configuration reference](configuration.md) describes every YAML setting,
API token permissions, and collector switch.
- [Metrics reference](metrics.md) lists exported metrics, labels, units, and
numeric state mappings.
- [Operations and troubleshooting](operations.md) covers native and container
deployment, endpoint failover, logging, security, and common errors.
- [Architecture and adding collectors](architecture.md) explains the collection
flow and how the main packages fit together.
## Important behavior
- One exporter instance is intended for one standalone node or one PVE cluster.
- Multiple configured API hosts provide redundant access to that same cluster.
- Collection runs every `proxmox.interval` seconds independently of Prometheus
scrapes.
- Collector label sets that stop receiving updates are removed after five
minutes.
- The only application HTTP endpoint is `/metrics`.
docs/architecture.md
+88
View File
@@ -0,0 +1,88 @@
# Architecture and Adding Collectors
## Collection flow
```text
config.yaml
|
v
Application -> PveApiClient -> Proxmox VE API host(s)
| |
| +-> liveness checks, round-robin selection, short API cache
v
PveMetricsManager -> enabled collectors -> Prometheus registry -> /metrics
|
+-> TTL cleanup for stale label sets
```
## Packages
| Package | Responsibility |
| --- | --- |
| `application` | Loads configuration, creates dependencies, registers `/metrics`, and starts the HTTP server. |
| `configuration` | YAML models and startup validation. |
| `proxmox` | HTTP client, Proxmox-specific API methods, response models, and numeric state conversion. |
| `metrics` | Collector implementations, periodic scheduling, Prometheus metric definitions, and TTL cleanup. |
| `utils` | Shared formatting helpers. |
## API client
`PveApiClient` wraps the generic `ApiClient` with typed methods such as cluster
status, node status, guest status, storage, disk, and ZFS calls.
The generic client:
- normalizes API hosts so a trailing slash is optional;
- checks every host at startup and every five seconds;
- selects reachable hosts in round-robin order;
- caches a successful method/path response briefly to avoid duplicate API calls;
- uses API-token authorization and a ten-second HTTP timeout;
- accepts self-signed certificates by disabling TLS verification.
Configured hosts must represent the same cluster because cached results are
shared by method and path rather than host.
## Metrics manager
`PveMetricsManager` creates only the collectors enabled in configuration. It
runs one full collection immediately, then repeats it every
`proxmox.interval` seconds. Collectors execute sequentially.
Each collector implements:
```go
type PveMetricsCollector interface {
CollectMetrics() error
GetName() string
}
```
Successful execution time is recorded in
`pve_metrics_collection_latency_ms`. A collector error is logged and the
manager continues with the next collector.
## TTL metrics
Most project metrics use `TTLGaugeVec`, a wrapper around Prometheus `GaugeVec`.
Every label set records its last update. `TTLRegistry` checks registered metrics
every five seconds and removes series that have not been updated for five
minutes.
This prevents deleted guests or storage resources from remaining indefinitely,
while allowing short API failures to preserve the last known values.
## Adding a collector
1. Add typed response models and API methods under `proxmox`.
2. Add a collector implementing `PveMetricsCollector` under `metrics`.
3. Create metrics with `NewTTLGaugeVec`, choose stable labels, and register each
metric with the shared `TTLRegistry`.
4. Add a boolean switch to `PveMetricsConfiguration` and `config.example.yaml`.
5. Register the collector in `NewPveMetricsManager`.
6. Add parsing, state-mapping, and topology tests as appropriate.
7. Document the switch, labels, units, and state values in
[configuration.md](configuration.md) and [metrics.md](metrics.md).
Avoid labels containing changing messages, timestamps, or other unbounded
values. For recursive resources such as ZFS, include a stable path so repeated
component names remain distinguishable.
docs/configuration.md
+113
View File
@@ -0,0 +1,113 @@
# Configuration Reference
The exporter reads `config.yaml` from the current working directory by default.
Use `-config` to select a different file:
```sh
./pve-exporter -config /etc/pve-exporter/config.yaml
```
## Complete example
```yaml
host: 0.0.0.0
port: 9090
logLevel: 4
proxmox:
token:
tokenId: monitoring@pve!exporter
secret: change-me
hosts:
- https://192.168.0.10:8006
- https://pve02.example.com:8006/
interval: 15
metrics:
clusterState: true
nodeStatus: true
qemu: true
lxc: true
disk: true
zfs: true
storage: true
subscription: true
sdn: true
```
## Server settings
| Setting | Type | Description |
| --- | --- | --- |
| `host` | IP address | Address on which the exporter HTTP server listens, for example `0.0.0.0` or `127.0.0.1`. DNS names are not accepted here. |
| `port` | integer | Exporter HTTP port. Prometheus metrics are served at `/metrics`. |
| `logLevel` | integer | Logrus level: `0=Panic`, `1=Fatal`, `2=Error`, `3=Warn`, `4=Info`, `5=Debug`, `6=Trace`. |
## Proxmox token
| Setting | Type | Description |
| --- | --- | --- |
| `proxmox.token.tokenId` | string | Full Proxmox API token ID, normally `user@realm!token-name`. |
| `proxmox.token.secret` | string | API token secret. Treat the configuration file as sensitive. |
Create a dedicated monitoring user and token, then assign the read-only
`PVEAuditor` role to both the user and the token. The role supplies the audit
permissions used by cluster, node, storage, disk, ZFS, guest, and subscription
API calls.
## API hosts and failover
`proxmox.hosts` is a non-empty list of Proxmox API base URLs. Both IP addresses
and DNS names are supported:
```yaml
proxmox:
hosts:
- https://192.168.0.10:8006
- https://pve02.example.com:8006/
```
A trailing slash is optional and normalized automatically. Every host must use
`http` or `https` and must belong to the same PVE cluster. The exporter checks
endpoint liveness every five seconds and distributes API requests across the
currently reachable endpoints.
Do not combine unrelated clusters in one list. Their API responses share the
same short-lived client cache and Prometheus registry. Run a separate exporter
for each cluster.
## Collection interval
`proxmox.interval` is the positive number of seconds between collection cycles.
The first cycle runs during application startup. Prometheus scrapes only the
values already present in the registry, so its scrape interval does not control
the PVE API request rate.
## Collector switches
All switches are booleans. Omitted switches default to `false`.
| Setting | Collector | Main metric prefix |
| --- | --- | --- |
| `clusterState` | Cluster mode, node count, and quorum | `pve_cluster_` |
| `nodeStatus` | Node health and resource usage | `pve_node_` |
| `qemu` | QEMU virtual machines | `pve_vm_` |
| `lxc` | LXC containers | `pve_ct_` |
| `disk` | Physical disk health and metadata | `pve_node_disk_` |
| `zfs` | ZFS pool topology, state, and errors | `pve_node_zfs_` |
| `storage` | Enabled PVE storage status and capacity | `pve_storage_` |
| `subscription` | Node subscription information | `pve_node_subscription_` |
| `sdn` | Software-defined network state | `pve_sdn_` |
The exporter-level `pve_metrics_collection_latency_ms` summary is always
registered.
## Validation notes
- `host` must be a literal IP address.
- At least one Proxmox API host is required.
- API host URLs require an `http` or `https` scheme and an IP or DNS hostname.
- Token ID and secret cannot be empty.
- Keep `proxmox.interval` greater than zero and `logLevel` between `0` and `6`.
docs/metrics.md
+192
View File
@@ -0,0 +1,192 @@
# Metrics Reference
All exporter-specific metric names start with `pve_`. The Go Prometheus client
also exposes its standard `go_*`, `process_*`, and HTTP handler metrics.
## Common labels and lifecycle
Most resource metrics use these labels:
| Label | Description |
| --- | --- |
| `cluster` | PVE cluster name. A standalone installation uses `Standalone node - <node>`. |
| `node` | Proxmox node name. |
| `vmid` | QEMU VM or LXC container ID. |
| `name` | Guest name. |
Metric label sets are retained for five minutes after their last update and are
then removed. Consequently, metrics available only while a guest is running can
remain visible for up to five minutes after it stops.
## Exporter
| Metric | Labels | Description |
| --- | --- | --- |
| `pve_metrics_collection_latency_ms` | `collector` | Summary of successful collector execution time in milliseconds. |
## Cluster state
Enabled with `metrics.clusterState`. Labels: `cluster`.
| Metric | Description |
| --- | --- |
| `pve_cluster_mode` | `1` when PVE is configured as a cluster, `0` for a standalone node. A configured one-node cluster still reports `1`. |
| `pve_cluster_nodes` | Number of nodes reported by the cluster status record. Standalone mode reports `0`. |
| `pve_cluster_quorate` | Cluster quorum state: `1` quorate, `0` not quorate or standalone. |
## Node status
Enabled with `metrics.nodeStatus`. Unless noted otherwise, labels are `cluster`
and `node`.
| Metric | Description |
| --- | --- |
| `pve_node_state` | Node online state: `1` online, `0` offline. |
| `pve_node_uptime` | Node uptime in seconds. |
| `pve_node_cpu_count` | Number of logical CPUs. |
| `pve_node_cpu_usage` | CPU usage reported by PVE. |
| `pve_node_memory_total_bytes` | Total memory in bytes. |
| `pve_node_memory_used_bytes` | Used memory in bytes. |
| `pve_node_memory_free_bytes` | Free memory in bytes. |
| `pve_node_ksm_bytes` | Memory shared by Kernel Same-page Merging in bytes. |
| `pve_node_cgroup_mode` | PVE cgroup mode. |
| `pve_node_load1` | One-minute load average. |
| `pve_node_load5` | Five-minute load average. |
| `pve_node_load15` | Fifteen-minute load average. |
| `pve_node_rootfs_free_bytes` | Root filesystem free bytes. |
| `pve_node_rootfs_used_bytes` | Root filesystem used bytes. |
| `pve_node_rootfs_total_bytes` | Root filesystem total bytes. |
| `pve_node_rootfs_avail_bytes` | Root filesystem bytes available to unprivileged processes. |
| `pve_node_time` | Node UTC Unix timestamp. |
| `pve_node_localtime` | Node local Unix timestamp. |
| `pve_node_cpuinfo` | Constant `1` carrying `flags`, `cores`, `model`, `sockets`, `cpus`, and `hvm` labels. |
| `pve_node_systeminfo` | Constant `1` carrying `kversion`, `pveversion`, `machine`, `sysname`, and `release` labels. |
## Physical disks
Enabled with `metrics.disk`. Labels are `cluster`, `node`, `wwn`, `type`,
`model`, `serial`, `vendor`, `used`, and `osd_id`.
| Metric | Description |
| --- | --- |
| `pve_node_disk_healthy` | SMART health state: `1` for `OK` or `PASSED`, otherwise `0`. |
| `pve_node_disk_wearout` | Device wearout percentage when supplied by the PVE API. |
| `pve_node_disk_size_bytes` | Physical disk size in bytes. |
## ZFS
Enabled with `metrics.zfs`. The exporter discovers all pools on every node,
retrieves each pool detail, and recursively flattens pool, vdev, section, cache,
and leaf-device entries.
Labels are `cluster`, `node`, `pool`, `component`, `path`, and `leaf`.
`component` is the current topology entry, `path` identifies its complete
hierarchy, and `leaf` is `true` for a device entry.
| Metric | Description |
| --- | --- |
| `pve_node_zfs_state` | Numeric component state: `0=UNKNOWN`, `1=ONLINE`, `2=DEGRADED`, `3=FAULTED`, `4=OFFLINE`, `5=REMOVED`, `6=UNAVAIL`. |
| `pve_node_zfs_read_errors` | ZFS read error count reported for the component. |
| `pve_node_zfs_write_errors` | ZFS write error count reported for the component. |
| `pve_node_zfs_checksum_errors` | ZFS checksum error count reported for the component. |
ZFS section entries that do not contain a counter do not produce a false zero
series. A counter explicitly returned as zero is exported normally.
Example alert expressions:
```promql
pve_node_zfs_state != 1
```
```promql
(pve_node_zfs_read_errors > 0)
or (pve_node_zfs_write_errors > 0)
or (pve_node_zfs_checksum_errors > 0)
```
## PVE storage
Enabled with `metrics.storage`. Disabled storages are skipped. Labels are
`cluster`, `node`, `storage`, `type`, `content`, and `shared`.
| Metric | Description |
| --- | --- |
| `pve_storage_up` | Storage active state: `1` active, `0` inactive. |
| `pve_storage_total_bytes` | Total storage capacity in bytes. |
| `pve_storage_avail_bytes` | Available storage capacity in bytes. |
| `pve_storage_used_bytes` | Used storage capacity in bytes. |
## Node subscriptions
Enabled with `metrics.subscription`. The common labels are `cluster` and
`node`; `pve_node_subscription_info` also has `productname` and `serverid`.
| Metric | Description |
| --- | --- |
| `pve_node_subscription_info` | Constant `1` carrying subscription product and server identifiers. |
| `pve_node_subscription_status` | `0` for API status `notfound`, `1` for `active`, `2` for expired or otherwise unusable states such as `invalid`, `suspended`, and `new`. |
| `pve_node_subscription_regdate` | Subscription registration date as a Unix timestamp, when available. |
| `pve_node_subscription_nextduedate` | Next due date as a Unix timestamp, when available. |
| `pve_node_subscription_sockets` | Number of covered sockets reported by PVE. |
The PVE API cannot distinguish a node that never had a subscription from one
whose key was removed; both appear as `notfound` and therefore use value `0`.
## SDN
Enabled with `metrics.sdn`. Labels are `cluster`, `node`, `sdn`, and `sdn_id`.
| Metric | Description |
| --- | --- |
| `pve_sdn_state` | `1` when the SDN resource status is `ok`, otherwise `0`. |
## LXC containers
Enabled with `metrics.lxc`. Templates are skipped. Labels are `cluster`,
`node`, `vmid`, and `name`.
| Metric | Description |
| --- | --- |
| `pve_ct_state` | `1` running, `0` stopped. |
| `pve_ct_uptime` | Uptime in seconds; updated only while running. |
| `pve_ct_cpu_count` | Configured CPU count. |
| `pve_ct_cpu_usage` | CPU usage reported by PVE; updated only while running. |
| `pve_ct_mem_total_bytes` | Configured memory limit in bytes. |
| `pve_ct_mem_used_bytes` | Used memory in bytes; updated only while running. |
| `pve_ct_network_in_bytes` | Received bytes since container start; updated only while running. |
| `pve_ct_network_out_bytes` | Transmitted bytes since container start; updated only while running. |
| `pve_ct_disk_rd_bytes` | Disk bytes read; updated only while running. |
| `pve_ct_disk_wr_bytes` | Disk bytes written; updated only while running. |
| `pve_ct_disk_usage_bytes` | Used root disk bytes; updated only while running. |
| `pve_ct_disk_size_bytes` | Configured root disk size in bytes. |
| `pve_ct_swap_used_bytes` | Used swap in bytes; updated only while running. |
## QEMU virtual machines
Enabled with `metrics.qemu`. Templates are skipped. Base labels are `cluster`,
`node`, `vmid`, and `name`.
| Metric | Extra label | Description |
| --- | --- | --- |
| `pve_vm_state` | none | `1` running, `0` stopped. |
| `pve_vm_uptime` | none | Uptime in seconds; updated only while running. |
| `pve_vm_agent` | none | QEMU guest agent state reported by PVE; updated only while running. |
| `pve_vm_cpu_count` | none | Configured CPU count. |
| `pve_vm_cpu_usage` | none | CPU usage reported by PVE; updated only while running. |
| `pve_vm_mem_total_bytes` | none | Configured maximum memory in bytes. |
| `pve_vm_mem_used_bytes` | none | Used memory in bytes; updated only while running. |
| `pve_vm_disk_usage_bytes` | none | Root disk usage reported by PVE. |
| `pve_vm_disk_size_bytes` | none | Configured maximum root disk size in bytes. |
| `pve_vm_network_in_bytes` | `interface` | Bytes received by a virtual interface. |
| `pve_vm_network_out_bytes` | `interface` | Bytes transmitted by a virtual interface. |
| `pve_vm_disk_rd_operations` | `device` | Successful read operations. |
| `pve_vm_disk_wr_operations` | `device` | Successful write operations. |
| `pve_vm_disk_rd_bytes` | `device` | Bytes read from the block device. |
| `pve_vm_disk_wr_bytes` | `device` | Bytes written to the block device. |
| `pve_vm_disk_failed_rd_ops` | `device` | Failed read operations. |
| `pve_vm_disk_failed_wr_ops` | `device` | Failed write operations. |
| `pve_vm_disk_rd_time_total_ns` | `device` | Total block-device read time in nanoseconds. |
| `pve_vm_disk_wr_time_total_ns` | `device` | Total block-device write time in nanoseconds. |
Interface and block-device metrics are collected only for running VMs.
docs/operations.md
+133
View File
@@ -0,0 +1,133 @@
# Operations and Troubleshooting
## Native execution
Build and run from the repository root:
```sh
go build -o pve-exporter .
./pve-exporter -config config.yaml
```
The default configuration path is `config.yaml`. The HTTP listener is formed
from the top-level `host` and `port` settings. Only `/metrics` is registered.
Verify the exporter locally:
```sh
curl http://localhost:9090/metrics
```
## Docker
Pull and run the published image:
```sh
docker pull gitea.lostak.dev/lostakj/pve-exporter:latest
docker run --rm -p 9090:9090 \
-v "$PWD/config.yaml:/opt/config.yaml:ro" \
gitea.lostak.dev/lostakj/pve-exporter:latest
```
The image contains the example configuration at `/opt/config.yaml`, but that
file has no API hosts or real credentials. Mount a configured file at the same
path for normal use.
## Prometheus
When Prometheus and the exporter share a container network:
```yaml
scrape_configs:
- job_name: pve-exporter
scrape_interval: 30s
static_configs:
- targets: [pve-exporter:9090]
```
The Prometheus scrape interval and `proxmox.interval` are independent. A scrape
does not contact PVE; it reads the latest values collected in the background.
## API endpoint availability
At startup and every five seconds, the exporter requests `api2/json/` from each
configured PVE host. Requests are distributed round-robin across endpoints that
passed the latest check.
Use multiple hosts only for nodes in the same cluster:
```yaml
proxmox:
hosts:
- https://192.168.0.10:8006
- https://pve02.example.com:8006
```
IP and DNS hosts work with or without a trailing slash. If every endpoint is
unavailable, collectors log `All API endpoints are unreachable.` and retain
their previous series until the five-minute stale-metric timeout removes them.
## Logging
Set `logLevel` between `0` and `6`:
| Value | Level |
| --- | --- |
| `0` | Panic |
| `1` | Fatal |
| `2` | Error |
| `3` | Warn |
| `4` | Info |
| `5` | Debug |
| `6` | Trace |
`Info` is suitable for normal operation. `Debug` or `Trace` can help diagnose
endpoint availability and individual collector execution.
## Common problems
### Authentication failed
Check that `tokenId` contains the full `user@realm!token-name` identifier and
that `secret` contains the token secret rather than the user password. Assign
the `PVEAuditor` role to both the user and token.
### Permission or empty cluster-status errors
The `/cluster/status` endpoint is used by every collector to identify nodes and
the cluster label. Ensure the token can audit `/` and the relevant nodes. The
exporter reports a specific permission hint when the endpoint returns an empty
array.
### No metrics from a collector
Confirm that its boolean switch under `proxmox.metrics` is `true`. Some series
are intentionally conditional:
- VM and LXC templates are skipped.
- Runtime guest metrics are updated only while the guest is running.
- Disabled PVE storages are skipped.
- Optional ZFS counters are omitted when the API does not return them.
- Subscription dates are omitted when they are unavailable.
### ZFS collection errors
Enable `proxmox.metrics.zfs`, confirm that the node has ZFS utilities and pools,
and verify that the token can call both `/nodes/{node}/disks/zfs` and each pool
detail endpoint.
### Metrics remain after a resource disappears
Collector metrics use a five-minute TTL. A guest, disk, storage, or ZFS topology
entry that disappears can therefore remain visible for up to five minutes.
## Security notes
- Keep `config.yaml` readable only by the exporter account because it contains
the API token secret.
- The HTTP endpoint has no authentication; bind to a protected interface or
restrict access with a firewall or reverse proxy.
- The PVE client currently disables TLS certificate verification to support the
default self-signed certificates. Use the exporter only across trusted
networks or a protected tunnel.
- Prefer the read-only `PVEAuditor` role and do not use a root token.