8 Commits
Author SHA1 Message Date
lostakj 92e54418bc Align metric names with Prometheus naming conventions
Build Docker image on push / docker (push) Successful in 21s
Build and push Docker image on tag / docker (push) Successful in 22s
Use base units, export cumulative values as _total counters, update docs, dashboards and rules.
2026-08-24 19:13:05 +02:00
lostakj 9b076d4353 Fixed error
Build Docker image on push / docker (push) Successful in 23s
2026-08-24 01:55:40 +02:00
lostakj e8e94c5da1 Added dashboards for grafana
Build Docker image on push / docker (push) Successful in 23s
2026-08-23 15:43:49 +02:00
lostakj 832549d667 Added example alerting config
Build Docker image on push / docker (push) Successful in 21s
2026-08-18 01:44:45 +02:00
lostakj b69e10d52f Updated docs
Build Docker image on push / docker (push) Successful in 23s
2026-08-18 01:13:27 +02:00
lostakj 2d634470d9 Updated docs
Build Docker image on push / docker (push) Successful in 21s
Build and push Docker image on tag / docker (push) Successful in 22s
2026-08-18 00:52:04 +02:00
lostakj 072f5bc7eb Added latest tag to tag build
Build Docker image on push / docker (push) Successful in 25s
2026-08-18 00:45:03 +02:00
lostakj 6e3bb0ff7b Added host URL normalization
Build Docker image on push / docker (push) Successful in 22s
Build and push Docker image on tag / docker (push) Successful in 23s
2026-08-17 23:04:28 +02:00
30 changed files with 14574 additions and 243 deletions
+1
View File
@@ -38,5 +38,6 @@ jobs:
docker buildx build \
--platform linux/amd64 \
-t "${IMAGE_NAME}:${IMAGE_TAG}" \
-t "${IMAGE_NAME}:latest" \
--push \
.
+114 -81
View File
@@ -1,121 +1,154 @@
# PVE Exporter
Proxmox Virtual Environment Prometheus metrics exporter.
Prometheus exporter for Proxmox Virtual Environment. It periodically reads the
Proxmox VE HTTP API and exposes cluster, node, guest, storage, disk, ZFS,
subscription, and SDN metrics through a single `/metrics` endpoint.
## Overview
PVE Exporter is a tool that collects metrics from a Proxmox Virtual Environment cluster and exposes them for Prometheus to scrape. This exporter supports gathering metrics for cluster state, LXC containers, QEMU virtual machines, physical disks, ZFS pools, node storage, node status, node subscription details, and software-defined networking (SDN).
The exporter supports standalone nodes and multi-node clusters. Multiple API
hosts can be configured for endpoint failover within one Proxmox cluster.
## Features
- Collects various metrics from Proxmox VE to monitor the health and performance of your virtual environment.
- Supports multi-node clusters for high availability.
- Securely uses Proxmox API tokens with minimal permissions.
- Standalone-node and cluster mode detection, node count, and quorum state.
- Node CPU, memory, load, root filesystem, uptime, time, and system information.
- QEMU virtual machine and LXC container state and resource metrics.
- Physical disk health, wearout, and size metrics.
- Recursive ZFS pool, vdev, cache, and leaf-device state and error metrics.
- Storage capacity and availability, subscription, and SDN state metrics.
- Selectively enabled collectors with automatic cleanup of stale label sets.
- Multiple API endpoints with periodic liveness checks and round-robin requests.
- Proxmox API token authentication using the read-only `PVEAuditor` role.
## Prerequisites
## Quick start
- Docker installed on your machine.
- Access to a Proxmox VE instance with API tokens configured.
Copy the example configuration and set the Proxmox API hosts and token:
## Configuration
### Proxmox API Token
When generating the token, make sure to assign the 'PVEAuditor' permission to both the user and the API token. For security reasons, assign only the 'PVEAuditor' role to limit permissions appropriately.
```sh
cp config.example.yaml config.yaml
```
```yaml
# Proxmox API token configuration.
host: 0.0.0.0
port: 9090
logLevel: 4
proxmox:
token:
tokenId: "your-token-id"
secret: "your-secret"
```
tokenId: monitoring@pve!exporter
secret: change-me
### Proxmox API Hosts
If you are running a multi-node cluster, add multiple API hosts to ensure high availability of metrics. Note that this configuration is not intended for gathering metrics from multiple PVE clusters. For multiple PVE clusters, deploy a separate exporter instance for each cluster.
```yaml
# Proxmox API hosts.
hosts:
- "https://host1.example.com"
- "https://host2.example.com"
```
- https://pve01.example.com:8006
- https://pve02.example.com:8006
### Metrics Configuration
interval: 15
Configure which metrics to collect by enabling or disabling specific metric types.
```yaml
# Proxmox metrics configuration.
metrics:
clusterState: true # Enable collection of cluster state metrics.
ltc: true # Enable collection of LXC container metrics.
qemu: true # Enable collection of QEMU virtual machine metrics.
disk: true # Enable collection of physical disk metrics.
zfs: true # Enable collection of ZFS pool metrics.
storage: true # Enable collection of node storage metrics.
nodeStatus: true # Enable collection of node status metrics.
subscription: true # Enable collection of node subscription details.
sdn: true # Enable collection of software-defined network (SDN) metrics.
clusterState: true
nodeStatus: true
qemu: true
lxc: true
disk: true
zfs: true
storage: true
subscription: true
sdn: true
```
The cluster state collector exposes `pve_cluster_mode`, where `1` means that
Proxmox VE is configured as a cluster and `0` means that it is a standalone node.
Assign the `PVEAuditor` role to both the Proxmox user and API token. All entries
under `hosts` must belong to the same cluster. IP addresses and DNS names are
supported, with or without a trailing slash.
The `pve_node_subscription_status` metric uses `0` when no subscription is
configured, `1` for an active subscription, and `2` for an expired or otherwise
unusable subscription.
The ZFS collector discovers all pools on every node and exports their recursive
topology with `cluster`, `node`, `pool`, `component`, `path`, and `leaf` labels:
- `pve_node_zfs_state` (`0=UNKNOWN`, `1=ONLINE`, `2=DEGRADED`, `3=FAULTED`,
`4=OFFLINE`, `5=REMOVED`, `6=UNAVAIL`)
- `pve_node_zfs_read_errors`
- `pve_node_zfs_write_errors`
- `pve_node_zfs_checksum_errors`
## Build
To build the Docker image for PVE Exporter, use the following command:
Build and start the exporter:
```sh
docker build --rm -t gitea.lostak.dev/lostakj/pve-exporter:version .
go build -o pve-exporter .
./pve-exporter -config config.yaml
```
Replace `version` with the appropriate version tag you want to use.
The exporter exposes Prometheus metrics at:
## Run
To run the Docker image, use the following command:
```sh
docker run --rm -d -p 9090:9090 --name pve-exporter gitea.lostak.dev/lostakj/pve-exporter:version
```text
http://localhost:9090/metrics
```
Replace `version` with the appropriate version tag you used during the build.
## Documentation
## Usage
- [Documentation index](docs/README.md)
- [Configuration reference](docs/configuration.md)
- [Metrics reference](docs/metrics.md)
- [Prometheus alerting rules](docs/alerting.md)
- [Operations and troubleshooting](docs/operations.md)
- [Architecture and adding collectors](docs/architecture.md)
Once the Docker container is running, the exporter will be available on port `9090`. You can configure Prometheus to scrape metrics from the exporter by adding a new scrape configuration to your Prometheus configuration file.
Example Prometheus scrape configuration:
## Prometheus configuration
```yaml
scrape_configs:
- job_name: 'pve-exporter'
- job_name: pve-exporter
static_configs:
- targets: ['localhost:9090']
- targets: [pve-exporter:9090]
```
Metrics are collected in the background according to `proxmox.interval`.
Prometheus scrapes the most recently collected values and does not trigger a new
request to Proxmox VE.
An example alert group for node, guest, storage, subscription, disk, ZFS, SDN,
and capacity conditions is available in
[`examples/pve-exporter.rules.yml`](examples/pve-exporter.rules.yml). See the
[alerting guide](docs/alerting.md) for the `rule_files` configuration and
collector requirements.
## Docker
Pull the latest published image from the Gitea container registry:
```sh
docker pull gitea.lostak.dev/lostakj/pve-exporter:latest
```
Or build it locally:
```sh
docker build --rm -t pve-exporter:latest .
```
Run the published image with a local configuration file:
```sh
docker run --rm -p 9090:9090 \
-v "$PWD/config.yaml:/opt/config.yaml:ro" \
gitea.lostak.dev/lostakj/pve-exporter:latest
```
Example with Docker Compose:
```yaml
services:
pve-exporter:
image: gitea.lostak.dev/lostakj/pve-exporter:latest
ports:
- "9090:9090"
volumes:
- ./config.yaml:/opt/config.yaml:ro
restart: unless-stopped
```
## Scope
One exporter instance monitors one standalone Proxmox VE installation or one
cluster. Configure multiple `hosts` only for redundant access to that same
cluster. Deploy a separate exporter instance for every additional cluster.
The published workflow currently builds the container image for `linux/amd64`.
The API client accepts self-signed Proxmox certificates because TLS certificate
verification is disabled; use trusted networks and protect the API token.
## License
This project is licensed under the MIT License. See the LICENSE.txt file for details.
This project is licensed under the [MIT License](LICENSE).
## Contributing
Contributions are welcome! Please open an issue or submit a pull request if you have any improvements or bug fixes.
## Support
If you encounter any issues or have questions, please open an issue on the project's GitHub repository.
Contributions and bug reports are welcome through the project issue tracker.
+1 -1
View File
@@ -22,7 +22,7 @@ proxmox:
# For multiple PVE clusters, deploy a separate exporter instance for each cluster.
hosts: []
# Scrape interval in seconds.
# Metrics collection interval in seconds.
interval: 15
# Proxmox metrics configuration.
+4
View File
@@ -63,6 +63,10 @@ func (c *Configuration) Validate() error {
if !(u.Scheme == "http" || u.Scheme == "https") {
return fmt.Errorf("PVE host '%s' must be protocol type of HTTP or HTTPS.", host)
}
if u.Hostname() == "" {
return fmt.Errorf("PVE host '%s' must contain an IP address or DNS name.", host)
}
}
// Validate PVE token
+32
View File
@@ -0,0 +1,32 @@
package configuration
import "testing"
func TestConfigurationValidateAcceptsIPAndDNSPveHosts(t *testing.T) {
tests := []struct {
name string
host string
}{
{name: "IP address", host: "https://192.168.0.10:8006"},
{name: "DNS name", host: "https://pve.example.com:8006"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
configuration := Configuration{
Host: "0.0.0.0",
PVE: PveConfiguration{
Hosts: []string{tt.host},
Token: PveTokenConfiguration{
TokenId: "token",
Secret: "secret",
},
},
}
if err := configuration.Validate(); err != nil {
t.Fatalf("Validate() rejected %s host %q: %v", tt.name, tt.host, err)
}
})
}
}
+27
View File
@@ -0,0 +1,27 @@
# PVE Exporter Documentation
PVE Exporter periodically collects monitoring data from the Proxmox VE HTTP API
and stores the latest values in the Prometheus registry exposed at `/metrics`.
## Guides
- [Configuration reference](configuration.md) describes every YAML setting,
API token permissions, and collector switch.
- [Metrics reference](metrics.md) lists exported metrics, labels, units, and
numeric state mappings.
- [Prometheus alerting](alerting.md) provides a ready-to-use rule group,
installation example, and collector requirements.
- [Operations and troubleshooting](operations.md) covers native and container
deployment, endpoint failover, logging, security, and common errors.
- [Architecture and adding collectors](architecture.md) explains the collection
flow and how the main packages fit together.
## Important behavior
- One exporter instance is intended for one standalone node or one PVE cluster.
- Multiple configured API hosts provide redundant access to that same cluster.
- Collection runs every `proxmox.interval` seconds independently of Prometheus
scrapes.
- Collector label sets that stop receiving updates are removed after five
minutes.
- The only application HTTP endpoint is `/metrics`.
+70
View File
@@ -0,0 +1,70 @@
# Prometheus Alerting
The repository includes a ready-to-use example rule group in
[`examples/pve-exporter.rules.yml`](../examples/pve-exporter.rules.yml). It
covers node, guest, storage, subscription, physical disk, ZFS, SDN, clock-skew,
and memory-overcommit conditions.
## Enable the rule file
Make the rule file available to Prometheus and reference it from
`prometheus.yml`:
```yaml
rule_files:
- /etc/prometheus/rules/pve-exporter.rules.yml
scrape_configs:
- job_name: pve-exporter
static_configs:
- targets: [pve-exporter:9090]
```
For a containerized Prometheus deployment, mount the example at the configured
path:
```yaml
services:
prometheus:
image: prom/prometheus:latest
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./examples/pve-exporter.rules.yml:/etc/prometheus/rules/pve-exporter.rules.yml:ro
```
Validate the file before reloading Prometheus:
```sh
promtool check rules examples/pve-exporter.rules.yml
```
## Collector requirements
Prometheus evaluates a rule only when its referenced time series exist. Enable
the matching exporter collectors:
| Alerts | Required collector setting |
| --- | --- |
| Node filesystem, state, CPU, clock skew, memory overcommit | `metrics.nodeStatus` |
| Guest CPU and memory overcommit | `metrics.qemu` and/or `metrics.lxc` |
| Storage state and capacity | `metrics.storage` |
| Subscription expiration | `metrics.subscription` |
| Physical disk health and wear | `metrics.disk` |
| ZFS state and errors | `metrics.zfs` |
| SDN state | `metrics.sdn` |
The default thresholds and `for` durations are operational examples. Review
them against the size, workload, redundancy, and maintenance policy of your
environment before enabling notifications.
Three rules require deployment-specific review:
- `PveUnexpectedStandaloneMode` should be removed or disabled when standalone
PVE operation is expected.
- `PveNodeSubscriptionMissing` is informational and should be removed when
unsubscribed nodes are intentional.
- `PveExporterDown` expects the Prometheus scrape job to be named
`pve-exporter`; adjust its `job` matcher if a different name is used.
Every rule uses the `cluster` label. Standalone installations are supported and
use the label value `Standalone node - <node>`.
+96
View File
@@ -0,0 +1,96 @@
# Architecture and Adding Collectors
## Collection flow
```text
config.yaml
|
v
Application -> PveApiClient -> Proxmox VE API host(s)
| |
| +-> liveness checks, round-robin selection, short API cache
v
PveMetricsManager -> enabled collectors -> Prometheus registry -> /metrics
|
+-> TTL cleanup for stale label sets
```
## Packages
| Package | Responsibility |
| --- | --- |
| `application` | Loads configuration, creates dependencies, registers `/metrics`, and starts the HTTP server. |
| `configuration` | YAML models and startup validation. |
| `proxmox` | HTTP client, Proxmox-specific API methods, response models, and numeric state conversion. |
| `metrics` | Collector implementations, periodic scheduling, Prometheus metric definitions, and TTL cleanup. |
| `utils` | Shared formatting helpers. |
## API client
`PveApiClient` wraps the generic `ApiClient` with typed methods such as cluster
status, node status, guest status, storage, disk, and ZFS calls.
The generic client:
- normalizes API hosts so a trailing slash is optional;
- checks every host at startup and every five seconds;
- selects reachable hosts in round-robin order;
- caches a successful method/path response briefly to avoid duplicate API calls;
- uses API-token authorization and a ten-second HTTP timeout;
- accepts self-signed certificates by disabling TLS verification.
Configured hosts must represent the same cluster because cached results are
shared by method and path rather than host.
## Metrics manager
`PveMetricsManager` creates only the collectors enabled in configuration. It
runs one full collection immediately, then repeats it every
`proxmox.interval` seconds. Collectors execute sequentially.
Each collector implements:
```go
type PveMetricsCollector interface {
CollectMetrics() error
GetName() string
}
```
Successful execution time is recorded in
`pve_metrics_collection_duration_seconds`. A collector error is logged and the
manager continues with the next collector.
## TTL metrics
Gauges use `TTLGaugeVec`, a wrapper around Prometheus `GaugeVec`. Cumulative
values use `TTLCounterVec`, which exports the absolute value reported by the PVE
API as a Prometheus counter (`GaugeVec` cannot be used because a counter name
must carry a `_total` suffix, and `CounterVec` cannot be used because it only
supports `Inc`/`Add`). Every label set records its last update. `TTLRegistry`
checks registered metrics every five seconds and removes series that have not
been updated for five minutes.
This prevents deleted guests or storage resources from remaining indefinitely,
while allowing short API failures to preserve the last known values.
## Adding a collector
1. Add typed response models and API methods under `proxmox`.
2. Add a collector implementing `PveMetricsCollector` under `metrics`.
3. Create metrics with `NewTTLGaugeVec` or, for cumulative values, with
`NewTTLCounterVec`, choose stable labels, and register each metric with the
shared `TTLRegistry`.
4. Add a boolean switch to `PveMetricsConfiguration` and `config.example.yaml`.
5. Register the collector in `NewPveMetricsManager`.
6. Add parsing, state-mapping, and topology tests as appropriate.
7. Document the switch, labels, units, and state values in
[configuration.md](configuration.md) and [metrics.md](metrics.md).
Metric names must follow the Prometheus naming conventions; they are verified by
`TestPveMetricNamesFollowPrometheusConventions`, which lints every registered
metric with `promlint`.
Avoid labels containing changing messages, timestamps, or other unbounded
values. For recursive resources such as ZFS, include a stable path so repeated
component names remain distinguishable.
+113
View File
@@ -0,0 +1,113 @@
# Configuration Reference
The exporter reads `config.yaml` from the current working directory by default.
Use `-config` to select a different file:
```sh
./pve-exporter -config /etc/pve-exporter/config.yaml
```
## Complete example
```yaml
host: 0.0.0.0
port: 9090
logLevel: 4
proxmox:
token:
tokenId: monitoring@pve!exporter
secret: change-me
hosts:
- https://192.168.0.10:8006
- https://pve02.example.com:8006/
interval: 15
metrics:
clusterState: true
nodeStatus: true
qemu: true
lxc: true
disk: true
zfs: true
storage: true
subscription: true
sdn: true
```
## Server settings
| Setting | Type | Description |
| --- | --- | --- |
| `host` | IP address | Address on which the exporter HTTP server listens, for example `0.0.0.0` or `127.0.0.1`. DNS names are not accepted here. |
| `port` | integer | Exporter HTTP port. Prometheus metrics are served at `/metrics`. |
| `logLevel` | integer | Logrus level: `0=Panic`, `1=Fatal`, `2=Error`, `3=Warn`, `4=Info`, `5=Debug`, `6=Trace`. |
## Proxmox token
| Setting | Type | Description |
| --- | --- | --- |
| `proxmox.token.tokenId` | string | Full Proxmox API token ID, normally `user@realm!token-name`. |
| `proxmox.token.secret` | string | API token secret. Treat the configuration file as sensitive. |
Create a dedicated monitoring user and token, then assign the read-only
`PVEAuditor` role to both the user and the token. The role supplies the audit
permissions used by cluster, node, storage, disk, ZFS, guest, and subscription
API calls.
## API hosts and failover
`proxmox.hosts` is a non-empty list of Proxmox API base URLs. Both IP addresses
and DNS names are supported:
```yaml
proxmox:
hosts:
- https://192.168.0.10:8006
- https://pve02.example.com:8006/
```
A trailing slash is optional and normalized automatically. Every host must use
`http` or `https` and must belong to the same PVE cluster. The exporter checks
endpoint liveness every five seconds and distributes API requests across the
currently reachable endpoints.
Do not combine unrelated clusters in one list. Their API responses share the
same short-lived client cache and Prometheus registry. Run a separate exporter
for each cluster.
## Collection interval
`proxmox.interval` is the positive number of seconds between collection cycles.
The first cycle runs during application startup. Prometheus scrapes only the
values already present in the registry, so its scrape interval does not control
the PVE API request rate.
## Collector switches
All switches are booleans. Omitted switches default to `false`.
| Setting | Collector | Main metric prefix |
| --- | --- | --- |
| `clusterState` | Cluster mode, node count, and quorum | `pve_cluster_` |
| `nodeStatus` | Node health and resource usage | `pve_node_` |
| `qemu` | QEMU virtual machines | `pve_vm_` |
| `lxc` | LXC containers | `pve_ct_` |
| `disk` | Physical disk health and metadata | `pve_node_disk_` |
| `zfs` | ZFS pool topology, state, and errors | `pve_node_zfs_` |
| `storage` | Enabled PVE storage status and capacity | `pve_storage_` |
| `subscription` | Node subscription information | `pve_node_subscription_` |
| `sdn` | Software-defined network state | `pve_sdn_` |
The exporter-level `pve_metrics_collection_duration_seconds` summary is always
registered.
## Validation notes
- `host` must be a literal IP address.
- At least one Proxmox API host is required.
- API host URLs require an `http` or `https` scheme and an IP or DNS hostname.
- Token ID and secret cannot be empty.
- Keep `proxmox.interval` greater than zero and `logLevel` between `0` and `6`.
+258
View File
@@ -0,0 +1,258 @@
# Metrics Reference
All exporter-specific metric names start with `pve_`. The Go Prometheus client
also exposes its standard `go_*`, `process_*`, and HTTP handler metrics.
Metric names follow the [Prometheus naming conventions](https://prometheus.io/docs/practices/naming/):
base units only (seconds, bytes, ratios) and a `_total` suffix on every metric
exported as a counter. Counters hold the absolute value reported by the PVE API
and reset when the guest or node restarts, so query them with `rate()` or
`increase()`. All remaining metrics are gauges.
## Common labels and lifecycle
Most resource metrics use these labels:
| Label | Description |
| --- | --- |
| `cluster` | PVE cluster name. A standalone installation uses `Standalone node - <node>`. |
| `node` | Proxmox node name. |
| `vmid` | QEMU VM or LXC container ID. |
| `name` | Guest name. |
Metric label sets are retained for five minutes after their last update and are
then removed. Consequently, metrics available only while a guest is running can
remain visible for up to five minutes after it stops.
## Exporter
| Metric | Labels | Description |
| --- | --- | --- |
| `pve_metrics_collection_duration_seconds` | `collector` | Summary of successful collector execution time in seconds. |
## Cluster state
Enabled with `metrics.clusterState`. Labels: `cluster`.
| Metric | Description |
| --- | --- |
| `pve_cluster_mode` | `1` when PVE is configured as a cluster, `0` for a standalone node. A configured one-node cluster still reports `1`. |
| `pve_cluster_nodes` | Number of nodes reported by the cluster status record. Standalone mode reports `0`. |
| `pve_cluster_quorate` | Cluster quorum state: `1` quorate, `0` not quorate or standalone. |
## Node status
Enabled with `metrics.nodeStatus`. Unless noted otherwise, labels are `cluster`
and `node`.
| Metric | Description |
| --- | --- |
| `pve_node_state` | Node online state: `1` online, `0` offline. |
| `pve_node_uptime_seconds` | Node uptime in seconds. |
| `pve_node_cpus` | Number of logical CPUs. |
| `pve_node_cpu_usage_ratio` | CPU usage reported by PVE as a ratio between `0` and `1`. |
| `pve_node_memory_total_bytes` | Total memory in bytes. |
| `pve_node_memory_used_bytes` | Used memory in bytes. |
| `pve_node_memory_free_bytes` | Free memory in bytes. |
| `pve_node_ksm_bytes` | Memory shared by Kernel Same-page Merging in bytes. |
| `pve_node_cgroup_mode` | PVE cgroup mode. |
| `pve_node_load1` | One-minute load average. |
| `pve_node_load5` | Five-minute load average. |
| `pve_node_load15` | Fifteen-minute load average. |
| `pve_node_rootfs_free_bytes` | Root filesystem free bytes. |
| `pve_node_rootfs_used_bytes` | Root filesystem used bytes. |
| `pve_node_rootfs_total_bytes` | Root filesystem total bytes. |
| `pve_node_rootfs_avail_bytes` | Root filesystem bytes available to unprivileged processes. |
| `pve_node_time_seconds` | Node UTC Unix timestamp. |
| `pve_node_localtime_seconds` | Node local Unix timestamp. |
| `pve_node_cpu_info` | Constant `1` carrying `flags`, `cores`, `model`, `sockets`, `cpus`, and `hvm` labels. |
| `pve_node_system_info` | Constant `1` carrying `kversion`, `pveversion`, `machine`, `sysname`, and `release` labels. |
## Physical disks
Enabled with `metrics.disk`. Labels are `cluster`, `node`, `wwn`, `type`,
`model`, `serial`, `vendor`, `used`, and `osd_id`.
| Metric | Description |
| --- | --- |
| `pve_node_disk_healthy` | SMART health state: `1` for `OK` or `PASSED`, otherwise `0`. |
| `pve_node_disk_wearout_percent` | Device wearout percentage when supplied by the PVE API. |
| `pve_node_disk_size_bytes` | Physical disk size in bytes. |
## ZFS
Enabled with `metrics.zfs`. The exporter discovers all pools on every node,
retrieves each pool detail, and recursively flattens pool, vdev, section, cache,
and leaf-device entries.
Labels are `cluster`, `node`, `pool`, `component`, `path`, and `leaf`.
`component` is the current topology entry, `path` identifies its complete
hierarchy, and `leaf` is `true` for a device entry.
| Metric | Description |
| --- | --- |
| `pve_node_zfs_state` | Numeric component state: `0=UNKNOWN`, `1=ONLINE`, `2=DEGRADED`, `3=FAULTED`, `4=OFFLINE`, `5=REMOVED`, `6=UNAVAIL`. |
| `pve_node_zfs_read_errors_total` | ZFS read error count reported for the component. |
| `pve_node_zfs_write_errors_total` | ZFS write error count reported for the component. |
| `pve_node_zfs_checksum_errors_total` | ZFS checksum error count reported for the component. |
ZFS section entries that do not contain a counter do not produce a false zero
series. A counter explicitly returned as zero is exported normally.
Example alert expressions:
```promql
pve_node_zfs_state != 1
```
```promql
(pve_node_zfs_read_errors_total > 0)
or (pve_node_zfs_write_errors_total > 0)
or (pve_node_zfs_checksum_errors_total > 0)
```
## PVE storage
Enabled with `metrics.storage`. Disabled storages are skipped. Labels are
`cluster`, `node`, `storage`, `type`, `content`, and `shared`.
| Metric | Description |
| --- | --- |
| `pve_storage_up` | Storage active state: `1` active, `0` inactive. |
| `pve_storage_total_bytes` | Total storage capacity in bytes. |
| `pve_storage_avail_bytes` | Available storage capacity in bytes. |
| `pve_storage_used_bytes` | Used storage capacity in bytes. |
## Node subscriptions
Enabled with `metrics.subscription`. The common labels are `cluster` and
`node`; `pve_node_subscription_info` also has `productname` and `serverid`.
| Metric | Description |
| --- | --- |
| `pve_node_subscription_info` | Constant `1` carrying subscription product and server identifiers. |
| `pve_node_subscription_status` | `0` for API status `notfound`, `1` for `active`, `2` for expired or otherwise unusable states such as `invalid`, `suspended`, and `new`. |
| `pve_node_subscription_registration_timestamp_seconds` | Subscription registration date as a Unix timestamp, when available. |
| `pve_node_subscription_next_due_timestamp_seconds` | Next due date as a Unix timestamp, when available. |
| `pve_node_subscription_sockets` | Number of covered sockets reported by PVE. |
The PVE API cannot distinguish a node that never had a subscription from one
whose key was removed; both appear as `notfound` and therefore use value `0`.
## SDN
Enabled with `metrics.sdn`. Labels are `cluster`, `node`, `sdn`, and `sdn_id`.
| Metric | Description |
| --- | --- |
| `pve_sdn_state` | `1` when the SDN resource status is `ok`, otherwise `0`. |
PVE 8 reports SDN zones as `sdn` cluster resources, PVE 9 reports them as
`network` resources (zones and fabrics). Both are exported as `pve_sdn_state`;
the `sdn` label holds the zone or fabric name and `sdn_id` the resource ID,
which differs between the two formats (`sdn/<node>/<zone>` versus
`network/<node>/<network-type>/<name>`).
## LXC containers
Enabled with `metrics.lxc`. Templates are skipped. Labels are `cluster`,
`node`, `vmid`, and `name`.
| Metric | Description |
| --- | --- |
| `pve_ct_state` | `1` running, `0` stopped. |
| `pve_ct_uptime_seconds` | Uptime in seconds; updated only while running. |
| `pve_ct_cpus` | Configured CPU count. |
| `pve_ct_cpu_usage_ratio` | CPU usage reported by PVE as a ratio between `0` and `1`; updated only while running. |
| `pve_ct_memory_total_bytes` | Configured memory limit in bytes. |
| `pve_ct_memory_used_bytes` | Used memory in bytes; updated only while running. |
| `pve_ct_network_receive_bytes_total` | Received bytes since container start; updated only while running. |
| `pve_ct_network_transmit_bytes_total` | Transmitted bytes since container start; updated only while running. |
| `pve_ct_disk_read_bytes_total` | Disk bytes read; updated only while running. |
| `pve_ct_disk_write_bytes_total` | Disk bytes written; updated only while running. |
| `pve_ct_disk_used_bytes` | Used root disk bytes; updated only while running. |
| `pve_ct_disk_size_bytes` | Configured root disk size in bytes. |
| `pve_ct_swap_used_bytes` | Used swap in bytes; updated only while running. |
## QEMU virtual machines
Enabled with `metrics.qemu`. Templates are skipped. Base labels are `cluster`,
`node`, `vmid`, and `name`.
| Metric | Extra label | Description |
| --- | --- | --- |
| `pve_vm_state` | none | `1` running, `0` stopped. |
| `pve_vm_uptime_seconds` | none | Uptime in seconds; updated only while running. |
| `pve_vm_agent` | none | QEMU guest agent state reported by PVE; updated only while running. |
| `pve_vm_cpus` | none | Configured CPU count. |
| `pve_vm_cpu_usage_ratio` | none | CPU usage reported by PVE as a ratio between `0` and `1`; updated only while running. |
| `pve_vm_memory_total_bytes` | none | Configured maximum memory in bytes. |
| `pve_vm_memory_used_bytes` | none | Used memory in bytes; updated only while running. |
| `pve_vm_disk_used_bytes` | none | Root disk usage reported by PVE. |
| `pve_vm_disk_size_bytes` | none | Configured maximum root disk size in bytes. |
| `pve_vm_network_receive_bytes_total` | `interface` | Bytes received by a virtual interface. |
| `pve_vm_network_transmit_bytes_total` | `interface` | Bytes transmitted by a virtual interface. |
| `pve_vm_disk_read_operations_total` | `device` | Successful read operations. |
| `pve_vm_disk_write_operations_total` | `device` | Successful write operations. |
| `pve_vm_disk_read_bytes_total` | `device` | Bytes read from the block device. |
| `pve_vm_disk_write_bytes_total` | `device` | Bytes written to the block device. |
| `pve_vm_disk_failed_read_operations_total` | `device` | Failed read operations. |
| `pve_vm_disk_failed_write_operations_total` | `device` | Failed write operations. |
| `pve_vm_disk_read_time_seconds_total` | `device` | Total block-device read time in seconds. |
| `pve_vm_disk_write_time_seconds_total` | `device` | Total block-device write time in seconds. |
Interface and block-device metrics are collected only for running VMs.
## Migration from earlier releases
All metric names were aligned with the Prometheus naming conventions. The table
below maps the previous names to the current ones. Dashboards, recording rules,
and alerting rules built against the old names have to be updated; the example
dashboards and rules in `examples/` already use the new names.
| Previous name | Current name | Note |
| --- | --- | --- |
| `pve_ct_cpu_usage` | `pve_ct_cpu_usage_ratio` | Values unchanged, they are already a ratio between `0` and `1`. |
| `pve_ct_cpu_count` | `pve_ct_cpus` | Renamed only. |
| `pve_ct_disk_rd_bytes` | `pve_ct_disk_read_bytes_total` | Type changed to counter. |
| `pve_ct_disk_usage_bytes` | `pve_ct_disk_used_bytes` | Renamed only. |
| `pve_ct_disk_wr_bytes` | `pve_ct_disk_write_bytes_total` | Type changed to counter. |
| `pve_ct_mem_total_bytes` | `pve_ct_memory_total_bytes` | Renamed only. |
| `pve_ct_mem_used_bytes` | `pve_ct_memory_used_bytes` | Renamed only. |
| `pve_ct_network_in_bytes` | `pve_ct_network_receive_bytes_total` | Type changed to counter. |
| `pve_ct_network_out_bytes` | `pve_ct_network_transmit_bytes_total` | Type changed to counter. |
| `pve_ct_uptime` | `pve_ct_uptime_seconds` | Renamed only. |
| `pve_metrics_collection_latency_ms` | `pve_metrics_collection_duration_seconds` | Values converted from milliseconds to seconds. |
| `pve_node_cpuinfo` | `pve_node_cpu_info` | Renamed only. |
| `pve_node_cpu_usage` | `pve_node_cpu_usage_ratio` | Values unchanged, they are already a ratio between `0` and `1`. |
| `pve_node_cpu_count` | `pve_node_cpus` | Renamed only. |
| `pve_node_disk_wearout` | `pve_node_disk_wearout_percent` | Renamed only. |
| `pve_node_localtime` | `pve_node_localtime_seconds` | Renamed only. |
| `pve_node_subscription_nextduedate` | `pve_node_subscription_next_due_timestamp_seconds` | Renamed only. |
| `pve_node_subscription_regdate` | `pve_node_subscription_registration_timestamp_seconds` | Renamed only. |
| `pve_node_systeminfo` | `pve_node_system_info` | Renamed only. |
| `pve_node_time` | `pve_node_time_seconds` | Renamed only. |
| `pve_node_uptime` | `pve_node_uptime_seconds` | Renamed only. |
| `pve_node_zfs_checksum_errors` | `pve_node_zfs_checksum_errors_total` | Type changed to counter. |
| `pve_node_zfs_read_errors` | `pve_node_zfs_read_errors_total` | Type changed to counter. |
| `pve_node_zfs_write_errors` | `pve_node_zfs_write_errors_total` | Type changed to counter. |
| `pve_vm_cpu_usage` | `pve_vm_cpu_usage_ratio` | Values unchanged, they are already a ratio between `0` and `1`. |
| `pve_vm_cpu_count` | `pve_vm_cpus` | Renamed only. |
| `pve_vm_disk_failed_rd_ops` | `pve_vm_disk_failed_read_operations_total` | Type changed to counter. |
| `pve_vm_disk_failed_wr_ops` | `pve_vm_disk_failed_write_operations_total` | Type changed to counter. |
| `pve_vm_disk_rd_bytes` | `pve_vm_disk_read_bytes_total` | Type changed to counter. |
| `pve_vm_disk_rd_operations` | `pve_vm_disk_read_operations_total` | Type changed to counter. |
| `pve_vm_disk_rd_time_total_ns` | `pve_vm_disk_read_time_seconds_total` | Values converted from nanoseconds to seconds, type changed to counter. |
| `pve_vm_disk_usage_bytes` | `pve_vm_disk_used_bytes` | Renamed only. |
| `pve_vm_disk_wr_bytes` | `pve_vm_disk_write_bytes_total` | Type changed to counter. |
| `pve_vm_disk_wr_operations` | `pve_vm_disk_write_operations_total` | Type changed to counter. |
| `pve_vm_disk_wr_time_total_ns` | `pve_vm_disk_write_time_seconds_total` | Values converted from nanoseconds to seconds, type changed to counter. |
| `pve_vm_mem_total_bytes` | `pve_vm_memory_total_bytes` | Renamed only. |
| `pve_vm_mem_used_bytes` | `pve_vm_memory_used_bytes` | Renamed only. |
| `pve_vm_network_in_bytes` | `pve_vm_network_receive_bytes_total` | Type changed to counter. |
| `pve_vm_network_out_bytes` | `pve_vm_network_transmit_bytes_total` | Type changed to counter. |
| `pve_vm_uptime` | `pve_vm_uptime_seconds` | Renamed only. |
Cumulative metrics are now exported as Prometheus counters instead of gauges,
which is what `rate()` and `increase()` expect and what removes the
"metric might not be a counter" hint in Grafana.
+138
View File
@@ -0,0 +1,138 @@
# Operations and Troubleshooting
## Native execution
Build and run from the repository root:
```sh
go build -o pve-exporter .
./pve-exporter -config config.yaml
```
The default configuration path is `config.yaml`. The HTTP listener is formed
from the top-level `host` and `port` settings. Only `/metrics` is registered.
Verify the exporter locally:
```sh
curl http://localhost:9090/metrics
```
## Docker
Pull and run the published image:
```sh
docker pull gitea.lostak.dev/lostakj/pve-exporter:latest
docker run --rm -p 9090:9090 \
-v "$PWD/config.yaml:/opt/config.yaml:ro" \
gitea.lostak.dev/lostakj/pve-exporter:latest
```
The image contains the example configuration at `/opt/config.yaml`, but that
file has no API hosts or real credentials. Mount a configured file at the same
path for normal use.
## Prometheus
When Prometheus and the exporter share a container network:
```yaml
scrape_configs:
- job_name: pve-exporter
scrape_interval: 30s
static_configs:
- targets: [pve-exporter:9090]
```
The Prometheus scrape interval and `proxmox.interval` are independent. A scrape
does not contact PVE; it reads the latest values collected in the background.
For alerting, load
[`examples/pve-exporter.rules.yml`](../examples/pve-exporter.rules.yml) through
Prometheus `rule_files`. The [alerting guide](alerting.md) lists the required
collectors and a container mount example.
## API endpoint availability
At startup and every five seconds, the exporter requests `api2/json/` from each
configured PVE host. Requests are distributed round-robin across endpoints that
passed the latest check.
Use multiple hosts only for nodes in the same cluster:
```yaml
proxmox:
hosts:
- https://192.168.0.10:8006
- https://pve02.example.com:8006
```
IP and DNS hosts work with or without a trailing slash. If every endpoint is
unavailable, collectors log `All API endpoints are unreachable.` and retain
their previous series until the five-minute stale-metric timeout removes them.
## Logging
Set `logLevel` between `0` and `6`:
| Value | Level |
| --- | --- |
| `0` | Panic |
| `1` | Fatal |
| `2` | Error |
| `3` | Warn |
| `4` | Info |
| `5` | Debug |
| `6` | Trace |
`Info` is suitable for normal operation. `Debug` or `Trace` can help diagnose
endpoint availability and individual collector execution.
## Common problems
### Authentication failed
Check that `tokenId` contains the full `user@realm!token-name` identifier and
that `secret` contains the token secret rather than the user password. Assign
the `PVEAuditor` role to both the user and token.
### Permission or empty cluster-status errors
The `/cluster/status` endpoint is used by every collector to identify nodes and
the cluster label. Ensure the token can audit `/` and the relevant nodes. The
exporter reports a specific permission hint when the endpoint returns an empty
array.
### No metrics from a collector
Confirm that its boolean switch under `proxmox.metrics` is `true`. Some series
are intentionally conditional:
- VM and LXC templates are skipped.
- Runtime guest metrics are updated only while the guest is running.
- Disabled PVE storages are skipped.
- Optional ZFS counters are omitted when the API does not return them.
- Subscription dates are omitted when they are unavailable.
### ZFS collection errors
Enable `proxmox.metrics.zfs`, confirm that the node has ZFS utilities and pools,
and verify that the token can call both `/nodes/{node}/disks/zfs` and each pool
detail endpoint.
### Metrics remain after a resource disappears
Collector metrics use a five-minute TTL. A guest, disk, storage, or ZFS topology
entry that disappears can therefore remain visible for up to five minutes.
## Security notes
- Keep `config.yaml` readable only by the exporter account because it contains
the API token secret.
- The HTTP endpoint has no authentication; bind to a protected interface or
restrict access with a firewall or reverse proxy.
- The PVE client currently disables TLS certificate verification to support the
default self-signed certificates. Use the exporter only across trusted
networks or a protected tunnel.
- Prefer the read-only `PVEAuditor` role and do not use a root token.
+526
View File
@@ -0,0 +1,526 @@
groups:
- name: proxmox.rules
rules:
- alert: PveNodeFilesystemFillingUp
expr: |-
(
pve_node_rootfs_avail_bytes / pve_node_rootfs_total_bytes * 100 < 15
and
predict_linear(pve_node_rootfs_avail_bytes[6h], 24 * 60 * 60) < 0
)
for: 1h
labels:
severity: critical
annotations:
summary: Proxmox root filesystem is predicted to run out of space within 24 hours.
description: >-
Filesystem on node {{ $labels.node }} in the {{ $labels.cluster }}
cluster has only {{ printf "%.2f" $value }}% available space left
and is filling up.
- alert: PveGuestCpuSaturated
expr: |-
pve_vm_cpu_usage_ratio > 0.95 or pve_ct_cpu_usage_ratio > 0.95
for: 1h
labels:
severity: warning
annotations:
summary: Proxmox guest CPU has been saturated for more than an hour.
description: >-
Guest {{ $labels.name }} with ID {{ $labels.vmid }} on node
{{ $labels.node }} in the {{ $labels.cluster }} cluster has sustained
CPU usage above 95%.
- alert: PveStorageDown
expr: |-
pve_storage_up == 0
for: 1m
labels:
severity: critical
annotations:
summary: Proxmox storage is down.
description: >-
Storage {{ $labels.storage }} on node {{ $labels.node }} in the
{{ $labels.cluster }} cluster is down. This can severely impact
running guests.
- alert: PveStorageFillingUp
expr: |-
(
pve_storage_avail_bytes / pve_storage_total_bytes * 100 < 15
and
predict_linear(pve_storage_avail_bytes[6h], 24 * 60 * 60) < 0
)
for: 1h
labels:
severity: critical
annotations:
summary: Proxmox storage is predicted to run out of space within 24 hours.
description: >-
Storage {{ $labels.storage }} on node {{ $labels.node }} in the
{{ $labels.cluster }} cluster has only {{ printf "%.2f" $value }}%
available space left and is filling up.
- alert: PveNodeDown
expr: |-
pve_node_state == 0
for: 1m
labels:
severity: critical
annotations:
summary: Proxmox node is down.
description: >-
Node {{ $labels.node }} in the {{ $labels.cluster }} cluster is down.
- alert: PveNodeCpuSaturated
expr: |-
pve_node_cpu_usage_ratio > 0.95
for: 1h
labels:
severity: warning
annotations:
summary: Proxmox node CPU has been saturated for more than an hour.
description: >-
CPU usage on node {{ $labels.node }} in the {{ $labels.cluster }}
cluster has remained above 95%. This can affect guests running on
the node.
- alert: PveSdnDown
expr: |-
pve_sdn_state == 0
for: 1m
labels:
severity: critical
annotations:
summary: Proxmox software-defined network is down.
description: >-
SDN resource {{ $labels.sdn }} on node {{ $labels.node }} in the
{{ $labels.cluster }} cluster is down.
- alert: PveClusterClockSkew
expr: |-
max by (cluster) (pve_node_localtime_seconds)
- min by (cluster) (pve_node_localtime_seconds) > 15
for: 1m
labels:
severity: critical
annotations:
summary: Proxmox cluster clock skew is too high.
description: >-
Clock skew between nodes in the {{ $labels.cluster }} cluster is
greater than 15 seconds. Cluster functions may be affected.
- alert: PveNodeSubscriptionExpiringSoon
expr: |-
(
pve_node_subscription_next_due_timestamp_seconds > time()
and
pve_node_subscription_next_due_timestamp_seconds < time() + (30 * 24 * 60 * 60)
)
and on (cluster, node)
(pve_node_subscription_status == 1)
for: 15m
labels:
severity: warning
annotations:
summary: Proxmox node subscription expires in less than 30 days.
description: >-
Subscription for node {{ $labels.node }} in the
{{ $labels.cluster }} cluster expires at
{{ $value | humanizeTimestamp }}.
- alert: PveNodeDiskFailed
expr: |-
pve_node_disk_healthy == 0
for: 1m
labels:
severity: critical
annotations:
summary: Proxmox node disk failed.
description: |-
A disk on node {{ $labels.node }} in the {{ $labels.cluster }} cluster is in a failed state.
WWN: {{ $labels.wwn }}
Serial: {{ $labels.serial }}
Model: {{ $labels.model }}
Immediate device replacement is required to ensure safe operation of the node.
- alert: PveNodeDiskWearout
expr: |-
pve_node_disk_wearout_percent < 5
for: 1m
labels:
severity: critical
annotations:
summary: Proxmox node disk has less than 5% wear life remaining.
description: |-
A disk on node {{ $labels.node }} in the {{ $labels.cluster }} cluster is showing significant wear and may fail soon.
WWN: {{ $labels.wwn }}
Serial: {{ $labels.serial }}
Model: {{ $labels.model }}
Current wear level: {{ $value }}%
Replace the disk to reduce the risk of data loss.
- alert: PveNodeZfsPoolUnknown
expr: |-
pve_node_zfs_state == 0
for: 5m
labels:
severity: warning
annotations:
summary: Proxmox ZFS pool component state is unknown.
description: >-
ZFS component {{ $labels.path }} in pool {{ $labels.pool }} on node
{{ $labels.node }} in the {{ $labels.cluster }} cluster has been in
the UNKNOWN state for at least five minutes.
- alert: PveNodeZfsPoolDegraded
expr: |-
pve_node_zfs_state == 2
for: 5m
labels:
severity: warning
annotations:
summary: Proxmox ZFS pool component is degraded.
description: >-
ZFS component {{ $labels.path }} in pool {{ $labels.pool }} on node
{{ $labels.node }} in the {{ $labels.cluster }} cluster has been
DEGRADED for at least five minutes.
- alert: PveNodeZfsPoolFaulted
expr: |-
pve_node_zfs_state == 3
for: 5m
labels:
severity: critical
annotations:
summary: Proxmox ZFS pool component has faulted.
description: >-
ZFS component {{ $labels.path }} in pool {{ $labels.pool }} on node
{{ $labels.node }} in the {{ $labels.cluster }} cluster has been
FAULTED for at least five minutes.
- alert: PveNodeZfsPoolOffline
expr: |-
pve_node_zfs_state == 4
for: 5m
labels:
severity: critical
annotations:
summary: Proxmox ZFS pool component is offline.
description: >-
ZFS component {{ $labels.path }} in pool {{ $labels.pool }} on node
{{ $labels.node }} in the {{ $labels.cluster }} cluster has been
OFFLINE for at least five minutes.
- alert: PveNodeZfsPoolRemoved
expr: |-
pve_node_zfs_state == 5
for: 5m
labels:
severity: critical
annotations:
summary: Proxmox ZFS pool component was removed.
description: >-
ZFS component {{ $labels.path }} in pool {{ $labels.pool }} on node
{{ $labels.node }} in the {{ $labels.cluster }} cluster has been
REMOVED for at least five minutes.
- alert: PveNodeZfsPoolUnavailable
expr: |-
pve_node_zfs_state == 6
for: 5m
labels:
severity: critical
annotations:
summary: Proxmox ZFS pool component is unavailable.
description: >-
ZFS component {{ $labels.path }} in pool {{ $labels.pool }} on node
{{ $labels.node }} in the {{ $labels.cluster }} cluster has been
UNAVAIL for at least five minutes.
- alert: PveNodeZfsReadErrors
expr: |-
pve_node_zfs_read_errors_total > 0
for: 5m
labels:
severity: warning
annotations:
summary: Proxmox ZFS component reports read errors.
description: >-
ZFS component {{ $labels.path }} in pool {{ $labels.pool }} on node
{{ $labels.node }} in the {{ $labels.cluster }} cluster reports
{{ $value | humanize }} read errors.
- alert: PveNodeZfsWriteErrors
expr: |-
pve_node_zfs_write_errors_total > 0
for: 5m
labels:
severity: warning
annotations:
summary: Proxmox ZFS component reports write errors.
description: >-
ZFS component {{ $labels.path }} in pool {{ $labels.pool }} on node
{{ $labels.node }} in the {{ $labels.cluster }} cluster reports
{{ $value | humanize }} write errors.
- alert: PveNodeZfsChecksumErrors
expr: |-
pve_node_zfs_checksum_errors_total > 0
for: 5m
labels:
severity: warning
annotations:
summary: Proxmox ZFS component reports checksum errors.
description: >-
ZFS component {{ $labels.path }} in pool {{ $labels.pool }} on node
{{ $labels.node }} in the {{ $labels.cluster }} cluster reports
{{ $value | humanize }} checksum errors.
- alert: PveNodeMemoryOvercommit
expr: |-
(
sum by (cluster, node) (
(
pve_vm_memory_total_bytes
and on (cluster, node, vmid, name) (pve_vm_state == 1)
)
or
(
pve_ct_memory_total_bytes
and on (cluster, node, vmid, name) (pve_ct_state == 1)
)
)
/ pve_node_memory_total_bytes * 100 - 100
) > 20
for: 1m
labels:
severity: critical
annotations:
summary: Proxmox node memory is overcommitted by more than 20%.
description: >-
Node {{ $labels.node }} in the {{ $labels.cluster }} cluster has
{{ printf "%.2f" $value }}% more memory allocated to running guests
than is physically available. Concurrent guest memory demand can
exhaust node memory and disrupt services.
- alert: PveClusterQuorumLost
expr: |-
(pve_cluster_mode == 1)
and on (cluster)
(pve_cluster_quorate == 0)
for: 1m
labels:
severity: critical
annotations:
summary: Proxmox cluster has lost quorum.
description: >-
The {{ $labels.cluster }} cluster has not been quorate for at least
one minute. Cluster operations and guest availability may be
affected.
- alert: PveUnexpectedStandaloneMode
expr: |-
pve_cluster_mode == 0
for: 5m
labels:
severity: warning
annotations:
summary: Proxmox is running in standalone mode.
description: >-
{{ $labels.cluster }} has reported standalone mode for at least five
minutes. Disable this rule when standalone operation is expected.
- alert: PveNodeMemoryHigh
expr: |-
pve_node_memory_used_bytes / pve_node_memory_total_bytes > 0.90
for: 15m
labels:
severity: warning
annotations:
summary: Proxmox node memory usage is high.
description: >-
Node {{ $labels.node }} in the {{ $labels.cluster }} cluster has
used {{ $value | humanizePercentage }} of its memory for at least
15 minutes.
- alert: PveNodeFilesystemLowSpace
expr: |-
pve_node_rootfs_avail_bytes / pve_node_rootfs_total_bytes < 0.10
for: 5m
labels:
severity: critical
annotations:
summary: Proxmox root filesystem has less than 10% space available.
description: >-
The root filesystem on node {{ $labels.node }} in the
{{ $labels.cluster }} cluster has only
{{ $value | humanizePercentage }} available space left.
- alert: PveStorageLowSpace
expr: |-
pve_storage_avail_bytes / pve_storage_total_bytes < 0.10
for: 5m
labels:
severity: critical
annotations:
summary: Proxmox storage has less than 10% space available.
description: >-
Storage {{ $labels.storage }} on node {{ $labels.node }} in the
{{ $labels.cluster }} cluster has only
{{ $value | humanizePercentage }} available space left.
- alert: PveNodeSubscriptionExpired
expr: |-
pve_node_subscription_status == 2
for: 15m
labels:
severity: warning
annotations:
summary: Proxmox node subscription has expired or is unusable.
description: >-
Subscription for node {{ $labels.node }} in the
{{ $labels.cluster }} cluster is expired, invalid, suspended, or
otherwise unusable.
- alert: PveNodeSubscriptionMissing
expr: |-
pve_node_subscription_status == 0
for: 15m
labels:
severity: info
annotations:
summary: Proxmox node has no subscription.
description: >-
Node {{ $labels.node }} in the {{ $labels.cluster }} cluster reports
no subscription. Disable this rule when an unsubscribed node is
expected.
- alert: PveNodeDiskWearoutWarning
expr: |-
(pve_node_disk_wearout_percent < 20)
and
(pve_node_disk_wearout_percent >= 5)
for: 15m
labels:
severity: warning
annotations:
summary: Proxmox node disk has less than 20% wear life remaining.
description: |-
A disk on node {{ $labels.node }} in the {{ $labels.cluster }} cluster is approaching its wear limit.
WWN: {{ $labels.wwn }}
Serial: {{ $labels.serial }}
Model: {{ $labels.model }}
Current wear level: {{ $value }}%
- alert: PveGuestMemoryHigh
expr: |-
pve_vm_memory_used_bytes / pve_vm_memory_total_bytes > 0.90
or
pve_ct_memory_used_bytes / pve_ct_memory_total_bytes > 0.90
for: 15m
labels:
severity: warning
annotations:
summary: Proxmox guest memory usage is high.
description: >-
Guest {{ $labels.name }} with ID {{ $labels.vmid }} on node
{{ $labels.node }} in the {{ $labels.cluster }} cluster has used
{{ $value | humanizePercentage }} of its configured memory for at
least 15 minutes.
- alert: PveGuestDiskFillingUp
expr: |-
pve_vm_disk_used_bytes / pve_vm_disk_size_bytes > 0.90
or
pve_ct_disk_used_bytes / pve_ct_disk_size_bytes > 0.90
for: 15m
labels:
severity: warning
annotations:
summary: Proxmox guest root disk usage is high.
description: >-
Guest {{ $labels.name }} with ID {{ $labels.vmid }} on node
{{ $labels.node }} in the {{ $labels.cluster }} cluster has used
{{ $value | humanizePercentage }} of its configured root disk.
- alert: PveContainerSwapUsageHigh
expr: |-
pve_ct_swap_used_bytes > 256 * 1024 * 1024
for: 15m
labels:
severity: warning
annotations:
summary: Proxmox container swap usage is high.
description: >-
Container {{ $labels.name }} with ID {{ $labels.vmid }} on node
{{ $labels.node }} in the {{ $labels.cluster }} cluster has used
{{ $value | humanize1024 }}B of swap for at least 15 minutes.
- alert: PveVmDiskErrorsIncreasing
expr: |-
increase(pve_vm_disk_failed_read_operations_total[10m]) > 0
or
increase(pve_vm_disk_failed_write_operations_total[10m]) > 0
for: 1m
labels:
severity: critical
annotations:
summary: Proxmox virtual machine reports new disk I/O failures.
description: >-
VM {{ $labels.name }} with ID {{ $labels.vmid }} on node
{{ $labels.node }} in the {{ $labels.cluster }} cluster reports new
failed I/O operations on device {{ $labels.device }}.
- alert: PveNodeClockDrift
expr: |-
abs(pve_node_time_seconds - time()) > 60
for: 5m
labels:
severity: warning
annotations:
summary: Proxmox node clock differs from Prometheus by more than one minute.
description: >-
The clock on node {{ $labels.node }} in the {{ $labels.cluster }}
cluster differs from the Prometheus server by
{{ $value | humanizeDuration }}.
- alert: PveNodeRestarted
expr: |-
resets(pve_node_uptime_seconds[15m]) > 0
for: 0m
labels:
severity: info
annotations:
summary: Proxmox node restarted recently.
description: >-
Uptime for node {{ $labels.node }} in the {{ $labels.cluster }}
cluster was reset during the last 15 minutes.
- alert: PveCollectorSlow
expr: |-
rate(pve_metrics_collection_duration_seconds_sum[5m])
/ rate(pve_metrics_collection_duration_seconds_count[5m]) > 5
for: 10m
labels:
severity: warning
annotations:
summary: PVE exporter collector is slow.
description: >-
Collector {{ $labels.collector }} has taken an average of
{{ printf "%.1f" $value }} s per successful run for at least ten
minutes.
- alert: PveExporterDown
expr: |-
up{job="pve-exporter"} == 0
for: 2m
labels:
severity: critical
annotations:
summary: PVE Exporter is unreachable.
description: >-
Prometheus has been unable to scrape PVE Exporter instance
{{ $labels.instance }} for at least two minutes.
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+6 -6
View File
@@ -23,7 +23,7 @@ type PveMetricsManager struct {
collectors []PveMetricsCollector // Metrics collector instances.
registry *TTLRegistry // Registry which handles autoamtic dangling metrics deletion.
latencySummary *prometheus.SummaryVec // Collection latency summary.
durationSummary *prometheus.SummaryVec // Collection duration summary.
interval int // Collection interval.
stop chan struct{} // Stop channel which is used in ticker.
@@ -80,10 +80,10 @@ func NewPveMetricsManager(apiClient *proxmox.PveApiClient, conf *configuration.P
c.RegisterCollector(NewPveVirtualMachineCollector(apiClient, c.registry))
}
// Metrics collection latency summary.
c.latencySummary = promauto.NewSummaryVec(prometheus.SummaryOpts{
Name: "pve_metrics_collection_latency_ms",
Help: "Summary of metrics collection latency milliseconds from PVE API.",
// Metrics collection duration summary.
c.durationSummary = promauto.NewSummaryVec(prometheus.SummaryOpts{
Name: "pve_metrics_collection_duration_seconds",
Help: "Summary of the PVE API metrics collection duration in seconds.",
}, []string{"collector"})
c.registry.StartCleanupLoop(5 * time.Second)
@@ -102,7 +102,7 @@ func (c *PveMetricsManager) collectMetrics() {
} else {
latency := time.Since(start)
log.Tracef("Finished collecting '%s' metrics after %s.", collector.GetName(), utils.HumanDuration(latency))
c.latencySummary.With(prometheus.Labels{"collector": collector.GetName()}).Observe(float64(latency.Milliseconds()))
c.durationSummary.With(prometheus.Labels{"collector": collector.GetName()}).Observe(latency.Seconds())
}
}
}
+72
View File
@@ -0,0 +1,72 @@
package metrics
import (
"testing"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/testutil/promlint"
)
// Verifies that all exported PVE metrics follow the Prometheus naming conventions
// (base units, '_total' suffix on counters, '_info' suffix on info metrics, ...)
// so that tools such as promtool or Grafana do not report naming problems.
func TestPveMetricNamesFollowPrometheusConventions(t *testing.T) {
registry := prometheus.NewRegistry()
// Collectors register their metrics on the default registerer, so it is
// temporarily replaced by a dedicated registry.
defaultRegisterer := prometheus.DefaultRegisterer
prometheus.DefaultRegisterer = registry
defer func() { prometheus.DefaultRegisterer = defaultRegisterer }()
// Collector constructors only create metrics, the API client is used during
// collection only, so a nil client is enough here.
ttlRegistry := NewTTLRegistry()
NewPveClusterStateCollector(nil, ttlRegistry)
NewPveNodeStatusCollector(nil, ttlRegistry)
NewPveSubscriptionCollector(nil, ttlRegistry)
NewPveNodeDiskCollector(nil, ttlRegistry)
NewPveNodeZfsCollector(nil, ttlRegistry)
NewPveSdnCollector(nil, ttlRegistry)
NewPveStorageCollector(nil, ttlRegistry)
NewPveContainerCollector(nil, ttlRegistry)
NewPveVirtualMachineCollector(nil, ttlRegistry)
// Metric vectors are exported only once they hold a label set.
for _, metric := range ttlRegistry.metrics {
switch m := metric.(type) {
case *TTLGaugeVec:
m.With(emptyLabels(m.labelNames)).Set(0)
case *TTLCounterVec:
m.With(emptyLabels(m.labelNames)).Set(0)
default:
t.Fatalf("Unknown TTL metric type %T.", metric)
}
}
families, err := registry.Gather()
if err != nil {
t.Fatalf("Unable to gather metrics. Error: %s.", err)
}
if len(families) != len(ttlRegistry.metrics) {
t.Fatalf("Gathered %d metric families but %d metrics are registered.", len(families), len(ttlRegistry.metrics))
}
problems, err := promlint.NewWithMetricFamilies(families).Lint()
if err != nil {
t.Fatalf("Unable to lint metrics. Error: %s.", err)
}
for _, problem := range problems {
t.Errorf("Metric '%s' violates the Prometheus naming conventions: %s.", problem.Metric, problem.Text)
}
}
// emptyLabels builds a label set with all given label names set to an empty value.
func emptyLabels(labelNames []string) prometheus.Labels {
labels := prometheus.Labels{}
for _, name := range labelNames {
labels[name] = ""
}
return labels
}
+29 -29
View File
@@ -21,11 +21,11 @@ type PveContainerCollector struct {
memBytes *TTLGaugeVec // Container memory in bytes prometheus gauge.
memBytesUsed *TTLGaugeVec // Container memory usage in bytes prometheus gauge.
netReceive *TTLGaugeVec // Container network RX in bytes prometheus gauge.
netTransmit *TTLGaugeVec // Container network TX in bytes prometheus gauge.
netReceive *TTLCounterVec // Container received network traffic in bytes prometheus counter.
netTransmit *TTLCounterVec // Container transmitted network traffic in bytes prometheus counter.
diskWrite *TTLGaugeVec // Container disk written in bytes prometheus gauge.
diskRead *TTLGaugeVec // Container disk read in bytes prometheus gauge.
diskWrite *TTLCounterVec // Container disk written in bytes prometheus counter.
diskRead *TTLCounterVec // Container disk read in bytes prometheus counter.
disk *TTLGaugeVec // Container disk space usage in bytes prometheus gauge.
diskMax *TTLGaugeVec // Container disk size in bytes prometheus gauge.
@@ -51,8 +51,8 @@ func NewPveContainerCollector(apiClient *proxmox.PveApiClient, registry *TTLRegi
// Container uptime.
c.uptime = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_ct_uptime",
Help: "Container uptime.",
Name: "pve_ct_uptime_seconds",
Help: "Container uptime in seconds.",
},
[]string{"cluster", "node", "vmid", "name"},
5*time.Minute,
@@ -62,7 +62,7 @@ func NewPveContainerCollector(apiClient *proxmox.PveApiClient, registry *TTLRegi
// Container CPU count.
c.cpu = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_ct_cpu_count",
Name: "pve_ct_cpus",
Help: "Container CPU count.",
},
[]string{"cluster", "node", "vmid", "name"},
@@ -73,8 +73,8 @@ func NewPveContainerCollector(apiClient *proxmox.PveApiClient, registry *TTLRegi
// Container CPU usage.
c.cpuUsage = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_ct_cpu_usage",
Help: "Container CPU usage.",
Name: "pve_ct_cpu_usage_ratio",
Help: "Container CPU usage ratio (0-1).",
},
[]string{"cluster", "node", "vmid", "name"},
5*time.Minute,
@@ -84,7 +84,7 @@ func NewPveContainerCollector(apiClient *proxmox.PveApiClient, registry *TTLRegi
// Container memory total.
c.memBytes = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_ct_mem_total_bytes",
Name: "pve_ct_memory_total_bytes",
Help: "Container total memory in bytes.",
},
[]string{"cluster", "node", "vmid", "name"},
@@ -95,7 +95,7 @@ func NewPveContainerCollector(apiClient *proxmox.PveApiClient, registry *TTLRegi
// Container memory usage.
c.memBytesUsed = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_ct_mem_used_bytes",
Name: "pve_ct_memory_used_bytes",
Help: "Container used memory in bytes.",
},
[]string{"cluster", "node", "vmid", "name"},
@@ -104,10 +104,10 @@ func NewPveContainerCollector(apiClient *proxmox.PveApiClient, registry *TTLRegi
c.registry.Register(c.memBytesUsed)
// Container network RX.
c.netReceive = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_ct_network_in_bytes",
Help: "Container network RX bytes.",
c.netReceive = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_ct_network_receive_bytes_total",
Help: "Container received network traffic in bytes.",
},
[]string{"cluster", "node", "vmid", "name"},
5*time.Minute,
@@ -115,10 +115,10 @@ func NewPveContainerCollector(apiClient *proxmox.PveApiClient, registry *TTLRegi
c.registry.Register(c.netReceive)
// Container network TX.
c.netTransmit = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_ct_network_out_bytes",
Help: "Container network TX bytes.",
c.netTransmit = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_ct_network_transmit_bytes_total",
Help: "Container transmitted network traffic in bytes.",
},
[]string{"cluster", "node", "vmid", "name"},
5*time.Minute,
@@ -126,10 +126,10 @@ func NewPveContainerCollector(apiClient *proxmox.PveApiClient, registry *TTLRegi
c.registry.Register(c.netTransmit)
// Container disk written.
c.diskWrite = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_ct_disk_wr_bytes",
Help: "Container disk written bytes.",
c.diskWrite = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_ct_disk_write_bytes_total",
Help: "Container written bytes to disk.",
},
[]string{"cluster", "node", "vmid", "name"},
5*time.Minute,
@@ -137,10 +137,10 @@ func NewPveContainerCollector(apiClient *proxmox.PveApiClient, registry *TTLRegi
c.registry.Register(c.diskWrite)
// Container disk read.
c.diskRead = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_ct_disk_rd_bytes",
Help: "Container disk read bytes.",
c.diskRead = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_ct_disk_read_bytes_total",
Help: "Container read bytes from disk.",
},
[]string{"cluster", "node", "vmid", "name"},
5*time.Minute,
@@ -150,8 +150,8 @@ func NewPveContainerCollector(apiClient *proxmox.PveApiClient, registry *TTLRegi
// Container disk size.
c.disk = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_ct_disk_usage_bytes",
Help: "Container disk read bytes.",
Name: "pve_ct_disk_used_bytes",
Help: "Container used disk space in bytes.",
},
[]string{"cluster", "node", "vmid", "name"},
5*time.Minute,
+3 -3
View File
@@ -38,13 +38,13 @@ func NewPveNodeDiskCollector(apiClient *proxmox.PveApiClient, registry *TTLRegis
// Node disk wearout.
c.wearout = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_node_disk_wearout",
Help: "Node disk wearout percent.",
Name: "pve_node_disk_wearout_percent",
Help: "Node disk wearout in percent (0-100).",
},
[]string{"cluster", "node", "wwn", "type", "model", "serial", "vendor", "used", "osd_id"},
5*time.Minute,
)
c.registry.Register(c.healthy)
c.registry.Register(c.wearout)
// Node disk size in bytes.
c.sizeBytes = NewTTLGaugeVec(
+1 -1
View File
@@ -53,7 +53,7 @@ func (c *PveSdnCollector) CollectMetrics() error {
labels := prometheus.Labels{
"cluster": cluster.GetClusterName(),
"node": node.Name,
"sdn": sdn.SDN,
"sdn": sdn.GetName(),
"sdn_id": sdn.ID,
}
+13 -13
View File
@@ -55,8 +55,8 @@ func NewPveNodeStatusCollector(apiClient *proxmox.PveApiClient, registry *TTLReg
// Node uptime.
c.uptime = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_node_uptime",
Help: "Node uptime.",
Name: "pve_node_uptime_seconds",
Help: "Node uptime in seconds.",
},
[]string{"cluster", "node"},
5*time.Minute,
@@ -66,7 +66,7 @@ func NewPveNodeStatusCollector(apiClient *proxmox.PveApiClient, registry *TTLReg
// Node cpu count.
c.cpus = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_node_cpu_count",
Name: "pve_node_cpus",
Help: "Node CPU count.",
},
[]string{"cluster", "node"},
@@ -77,8 +77,8 @@ func NewPveNodeStatusCollector(apiClient *proxmox.PveApiClient, registry *TTLReg
// Node CPU usage.
c.cpuUsage = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_node_cpu_usage",
Help: "Cluster node CPU usage %.",
Name: "pve_node_cpu_usage_ratio",
Help: "Node CPU usage ratio (0-1).",
},
[]string{"cluster", "node"},
5*time.Minute,
@@ -220,8 +220,8 @@ func NewPveNodeStatusCollector(apiClient *proxmox.PveApiClient, registry *TTLReg
// Node CPU info.
c.cpuInfo = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_node_cpuinfo",
Help: "Node CPU info.",
Name: "pve_node_cpu_info",
Help: "Node CPU information.",
},
[]string{"cluster", "node", "flags", "cores", "model", "sockets", "cpus", "hvm"},
5*time.Minute,
@@ -231,8 +231,8 @@ func NewPveNodeStatusCollector(apiClient *proxmox.PveApiClient, registry *TTLReg
// Node system info metrics.
c.systemInfo = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_node_systeminfo",
Help: "Node system info.",
Name: "pve_node_system_info",
Help: "Node system information.",
},
[]string{"cluster", "node", "kversion", "pveversion", "machine", "sysname", "release"},
5*time.Minute,
@@ -242,8 +242,8 @@ func NewPveNodeStatusCollector(apiClient *proxmox.PveApiClient, registry *TTLReg
// Node time info.
c.time = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_node_time",
Help: "Node time.",
Name: "pve_node_time_seconds",
Help: "Node UTC time as a unix timestamp in seconds.",
},
[]string{"cluster", "node"},
5*time.Minute,
@@ -253,8 +253,8 @@ func NewPveNodeStatusCollector(apiClient *proxmox.PveApiClient, registry *TTLReg
// Node localtime info.
c.localTime = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_node_localtime",
Help: "Node localtime.",
Name: "pve_node_localtime_seconds",
Help: "Node local time as a unix timestamp in seconds.",
},
[]string{"cluster", "node"},
5*time.Minute,
+4 -4
View File
@@ -49,8 +49,8 @@ func NewPveSubscriptionCollector(apiClient *proxmox.PveApiClient, registry *TTLR
// Node subscription registration date.
c.regDate = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_node_subscription_regdate",
Help: "Node subscription registration date.",
Name: "pve_node_subscription_registration_timestamp_seconds",
Help: "Node subscription registration date as a unix timestamp in seconds.",
},
[]string{"cluster", "node"},
5*time.Minute,
@@ -60,8 +60,8 @@ func NewPveSubscriptionCollector(apiClient *proxmox.PveApiClient, registry *TTLR
// Node subscription next due date.
c.nextDueDate = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_node_subscription_nextduedate",
Help: "Node subscription next due date.",
Name: "pve_node_subscription_next_due_timestamp_seconds",
Help: "Node subscription next due date as a unix timestamp in seconds.",
},
[]string{"cluster", "node"},
5*time.Minute,
+63 -63
View File
@@ -26,20 +26,20 @@ type PveVirtualMachineCollector struct {
diskMax *TTLGaugeVec // Virtual machine disk size in bytes prometheus gauge.
swap *TTLGaugeVec // Virtual machine swap usage in bytes prometheus gauge.
netReceive *TTLGaugeVec // Virtual machine network receive in bytes prometheus gauge.
netTransmit *TTLGaugeVec // Virtual machine network transmit in bytes prometheus gauge.
netReceive *TTLCounterVec // Virtual machine received network traffic in bytes prometheus counter.
netTransmit *TTLCounterVec // Virtual machine transmitted network traffic in bytes prometheus counter.
diskReadOps *TTLGaugeVec // Virtual machine disk read ops prometheus gauge.
diskWriteOps *TTLGaugeVec // Virtual machine disk write ops prometheus gauge.
diskReadOps *TTLCounterVec // Virtual machine disk read operations prometheus counter.
diskWriteOps *TTLCounterVec // Virtual machine disk write operations prometheus counter.
diskReadBytes *TTLGaugeVec // Virtual machine disk read bytes prometheus gauge.
diskWriteBytes *TTLGaugeVec // Virtual machine disk write bytes prometheus gauge.
diskReadBytes *TTLCounterVec // Virtual machine disk read bytes prometheus counter.
diskWriteBytes *TTLCounterVec // Virtual machine disk written bytes prometheus counter.
diskReadTimeNs *TTLGaugeVec // Virtual machine disk read time total prometheus gauge.
diskWriteTimeNs *TTLGaugeVec // Virtual machine disk write time total prometheus gauge.
diskReadTime *TTLCounterVec // Virtual machine disk read time total in seconds prometheus counter.
diskWriteTime *TTLCounterVec // Virtual machine disk write time total in seconds prometheus counter.
diskFailedReadOps *TTLGaugeVec // Virtual machine disk failed read ops prometheus gauge.
diskFailedWriteOps *TTLGaugeVec // Virtual machine disk failed write ops prometheus gauge.
diskFailedReadOps *TTLCounterVec // Virtual machine failed disk read operations prometheus counter.
diskFailedWriteOps *TTLCounterVec // Virtual machine failed disk write operations prometheus counter.
agent *TTLGaugeVec // Virtual machine agent enabled prometheus gauge.
}
@@ -63,8 +63,8 @@ func NewPveVirtualMachineCollector(apiClient *proxmox.PveApiClient, registry *TT
// Virtual machine uptime.
c.uptime = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_uptime",
Help: "Virtual machine uptime.",
Name: "pve_vm_uptime_seconds",
Help: "Virtual machine uptime in seconds.",
},
[]string{"cluster", "node", "vmid", "name"},
5*time.Minute,
@@ -85,7 +85,7 @@ func NewPveVirtualMachineCollector(apiClient *proxmox.PveApiClient, registry *TT
// Virtual machine CPU count.
c.cpu = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_cpu_count",
Name: "pve_vm_cpus",
Help: "Virtual machine CPU count.",
},
[]string{"cluster", "node", "vmid", "name"},
@@ -96,8 +96,8 @@ func NewPveVirtualMachineCollector(apiClient *proxmox.PveApiClient, registry *TT
// Virtual machine CPU usage.
c.cpuUsage = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_cpu_usage",
Help: "Virtual machine CPU usage.",
Name: "pve_vm_cpu_usage_ratio",
Help: "Virtual machine CPU usage ratio (0-1).",
},
[]string{"cluster", "node", "vmid", "name"},
5*time.Minute,
@@ -107,7 +107,7 @@ func NewPveVirtualMachineCollector(apiClient *proxmox.PveApiClient, registry *TT
// Virtual machine memory total.
c.memBytes = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_mem_total_bytes",
Name: "pve_vm_memory_total_bytes",
Help: "Virtual machine total memory in bytes.",
},
[]string{"cluster", "node", "vmid", "name"},
@@ -118,7 +118,7 @@ func NewPveVirtualMachineCollector(apiClient *proxmox.PveApiClient, registry *TT
// Virtual machine memory usage.
c.memBytesUsed = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_mem_used_bytes",
Name: "pve_vm_memory_used_bytes",
Help: "Virtual machine used memory in bytes.",
},
[]string{"cluster", "node", "vmid", "name"},
@@ -129,8 +129,8 @@ func NewPveVirtualMachineCollector(apiClient *proxmox.PveApiClient, registry *TT
// Virtual machine disk size.
c.disk = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_disk_usage_bytes",
Help: "Virtual machine disk read bytes.",
Name: "pve_vm_disk_used_bytes",
Help: "Virtual machine used disk space in bytes.",
},
[]string{"cluster", "node", "vmid", "name"},
5*time.Minute,
@@ -149,10 +149,10 @@ func NewPveVirtualMachineCollector(apiClient *proxmox.PveApiClient, registry *TT
c.registry.Register(c.diskMax)
// Virtual machine network receive bytes.
c.netReceive = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_network_in_bytes",
Help: "Virtual machine network receive in bytes.",
c.netReceive = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_vm_network_receive_bytes_total",
Help: "Virtual machine received network traffic in bytes.",
},
[]string{"cluster", "node", "vmid", "name", "interface"},
5*time.Minute,
@@ -160,10 +160,10 @@ func NewPveVirtualMachineCollector(apiClient *proxmox.PveApiClient, registry *TT
c.registry.Register(c.netReceive)
// Virtual machine network transmit bytes.
c.netTransmit = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_network_out_bytes",
Help: "Virtual machine network transmit in bytes.",
c.netTransmit = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_vm_network_transmit_bytes_total",
Help: "Virtual machine transmitted network traffic in bytes.",
},
[]string{"cluster", "node", "vmid", "name", "interface"},
5*time.Minute,
@@ -171,10 +171,10 @@ func NewPveVirtualMachineCollector(apiClient *proxmox.PveApiClient, registry *TT
c.registry.Register(c.netTransmit)
// Virtual machine disk read ops.
c.diskReadOps = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_disk_rd_operations",
Help: "Virtual machine disk read ops.",
c.diskReadOps = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_vm_disk_read_operations_total",
Help: "Virtual machine disk read operations.",
},
[]string{"cluster", "node", "vmid", "name", "device"},
5*time.Minute,
@@ -182,10 +182,10 @@ func NewPveVirtualMachineCollector(apiClient *proxmox.PveApiClient, registry *TT
c.registry.Register(c.diskReadOps)
// Virtual machine disk write ops.
c.diskWriteOps = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_disk_wr_operations",
Help: "Virtual machine disk write ops.",
c.diskWriteOps = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_vm_disk_write_operations_total",
Help: "Virtual machine disk write operations.",
},
[]string{"cluster", "node", "vmid", "name", "device"},
5*time.Minute,
@@ -193,9 +193,9 @@ func NewPveVirtualMachineCollector(apiClient *proxmox.PveApiClient, registry *TT
c.registry.Register(c.diskWriteOps)
// Virtual machine disk read bytes.
c.diskReadBytes = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_disk_rd_bytes",
c.diskReadBytes = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_vm_disk_read_bytes_total",
Help: "Virtual machine disk read bytes.",
},
[]string{"cluster", "node", "vmid", "name", "device"},
@@ -204,9 +204,9 @@ func NewPveVirtualMachineCollector(apiClient *proxmox.PveApiClient, registry *TT
c.registry.Register(c.diskReadBytes)
// Virtual machine disk write bytes.
c.diskWriteBytes = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_disk_wr_bytes",
c.diskWriteBytes = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_vm_disk_write_bytes_total",
Help: "Virtual machine disk write bytes.",
},
[]string{"cluster", "node", "vmid", "name", "device"},
@@ -215,10 +215,10 @@ func NewPveVirtualMachineCollector(apiClient *proxmox.PveApiClient, registry *TT
c.registry.Register(c.diskWriteBytes)
// Virtual machine failed disk read ops.
c.diskFailedReadOps = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_disk_failed_rd_ops",
Help: "Virtual machine failed disk read ops.",
c.diskFailedReadOps = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_vm_disk_failed_read_operations_total",
Help: "Virtual machine failed disk read operations.",
},
[]string{"cluster", "node", "vmid", "name", "device"},
5*time.Minute,
@@ -226,37 +226,37 @@ func NewPveVirtualMachineCollector(apiClient *proxmox.PveApiClient, registry *TT
c.registry.Register(c.diskFailedReadOps)
// Virtual machine failed disk write ops.
c.diskFailedWriteOps = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_disk_failed_wr_ops",
Help: "Virtual machine failed disk write ops.",
c.diskFailedWriteOps = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_vm_disk_failed_write_operations_total",
Help: "Virtual machine failed disk write operations.",
},
[]string{"cluster", "node", "vmid", "name", "device"},
5*time.Minute,
)
c.registry.Register(c.diskFailedWriteOps)
// Virtual machine disk read time total nanoseconds.
c.diskReadTimeNs = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_disk_rd_time_total_ns",
Help: "Virtual machine disk read time total in nanoseconds.",
// Virtual machine disk read time total seconds.
c.diskReadTime = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_vm_disk_read_time_seconds_total",
Help: "Virtual machine disk read time total in seconds.",
},
[]string{"cluster", "node", "vmid", "name", "device"},
5*time.Minute,
)
c.registry.Register(c.diskReadTimeNs)
c.registry.Register(c.diskReadTime)
// Virtual machine disk write time total nanoseconds.
c.diskWriteTimeNs = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_vm_disk_wr_time_total_ns",
Help: "Virtual machine disk write time total in nanoseconds.",
// Virtual machine disk write time total seconds.
c.diskWriteTime = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_vm_disk_write_time_seconds_total",
Help: "Virtual machine disk write time total in seconds.",
},
[]string{"cluster", "node", "vmid", "name", "device"},
5*time.Minute,
)
c.registry.Register(c.diskWriteTimeNs)
c.registry.Register(c.diskWriteTime)
return &c
}
@@ -336,8 +336,8 @@ func (c *PveVirtualMachineCollector) CollectMetrics() error {
c.diskFailedReadOps.With(labels).Set(float64(value.FailedRdOperations))
c.diskFailedWriteOps.With(labels).Set(float64(value.FailedWrOperations))
c.diskReadTimeNs.With(labels).Set(float64(value.RdTotalTimeNs))
c.diskWriteTimeNs.With(labels).Set(float64(value.WrTotalTimeNs))
c.diskReadTime.With(labels).Set(float64(value.RdTotalTimeNs) / float64(time.Second))
c.diskWriteTime.With(labels).Set(float64(value.WrTotalTimeNs) / float64(time.Second))
}
}
}
+12 -12
View File
@@ -15,9 +15,9 @@ type PveNodeZfsCollector struct {
registry *TTLRegistry
state *TTLGaugeVec
readErrors *TTLGaugeVec
writeErrors *TTLGaugeVec
checksumErrors *TTLGaugeVec
readErrors *TTLCounterVec
writeErrors *TTLCounterVec
checksumErrors *TTLCounterVec
}
// zfsMetricComponent is a flattened entry from the recursive ZFS topology.
@@ -50,9 +50,9 @@ func NewPveNodeZfsCollector(apiClient *proxmox.PveApiClient, registry *TTLRegist
)
c.registry.Register(c.state)
c.readErrors = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_node_zfs_read_errors",
c.readErrors = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_node_zfs_read_errors_total",
Help: "ZFS pool component read error count.",
},
componentLabelNames,
@@ -60,9 +60,9 @@ func NewPveNodeZfsCollector(apiClient *proxmox.PveApiClient, registry *TTLRegist
)
c.registry.Register(c.readErrors)
c.writeErrors = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_node_zfs_write_errors",
c.writeErrors = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_node_zfs_write_errors_total",
Help: "ZFS pool component write error count.",
},
componentLabelNames,
@@ -70,9 +70,9 @@ func NewPveNodeZfsCollector(apiClient *proxmox.PveApiClient, registry *TTLRegist
)
c.registry.Register(c.writeErrors)
c.checksumErrors = NewTTLGaugeVec(
prometheus.GaugeOpts{
Name: "pve_node_zfs_checksum_errors",
c.checksumErrors = NewTTLCounterVec(
prometheus.CounterOpts{
Name: "pve_node_zfs_checksum_errors_total",
Help: "ZFS pool component checksum error count.",
},
componentLabelNames,
+117
View File
@@ -23,6 +23,7 @@ type TTLMetric interface {
// it is automatically removed from the underlying GaugeVec.
type TTLGaugeVec struct {
gaugeVec *prometheus.GaugeVec // Underlying Prometheus GaugeVec.
labelNames []string // Label names of the underlying GaugeVec.
ttl time.Duration // Duration after which an unused label set is considered stale.
lastUpdate sync.Map // Map storing last update time for each label set (key is a sorted labels string).
}
@@ -32,6 +33,7 @@ type TTLGaugeVec struct {
func NewTTLGaugeVec(opts prometheus.GaugeOpts, labelNames []string, ttl time.Duration) *TTLGaugeVec {
return &TTLGaugeVec{
gaugeVec: promauto.NewGaugeVec(opts, labelNames),
labelNames: labelNames,
ttl: ttl,
}
}
@@ -137,3 +139,118 @@ func (r *TTLRegistry) StartCleanupLoop(interval time.Duration) {
}
}()
}
// TTLCounterVec exposes cumulative values as Prometheus counters while keeping
// the same "set an absolute value" usage as TTLGaugeVec. The PVE API reports
// counters as absolute totals, which prometheus.CounterVec cannot express
// (it only supports Inc/Add), so the metrics are emitted as constant metrics
// of type counter. Label sets not updated within the TTL are dropped.
type TTLCounterVec struct {
desc *prometheus.Desc // Metric descriptor.
labelNames []string // Label names in exposition order.
ttl time.Duration // Duration after which an unused label set is considered stale.
mu sync.RWMutex // Guards values.
values map[string]*ttlCounterEntry // Current value per label set.
}
// ttlCounterEntry holds the current value of a single label set.
type ttlCounterEntry struct {
labelValues []string // Label values in exposition order.
value float64 // Current counter value.
lastUpdate time.Time // Time of the last update.
}
// TTLCounter is a handle to a single label set of a TTLCounterVec.
type TTLCounter struct {
vec *TTLCounterVec
entry *ttlCounterEntry
}
// Set stores the current absolute value of the counter.
func (c *TTLCounter) Set(value float64) {
c.vec.mu.Lock()
defer c.vec.mu.Unlock()
c.entry.value = value
}
// NewTTLCounterVec creates a new TTLCounterVec and registers it using promauto.
func NewTTLCounterVec(opts prometheus.CounterOpts, labelNames []string, ttl time.Duration) *TTLCounterVec {
c := &TTLCounterVec{
desc: prometheus.NewDesc(
prometheus.BuildFQName(opts.Namespace, opts.Subsystem, opts.Name),
opts.Help,
labelNames,
opts.ConstLabels,
),
labelNames: labelNames,
ttl: ttl,
values: make(map[string]*ttlCounterEntry),
}
prometheus.MustRegister(c)
return c
}
// With returns the counter handle for the given label set and records the current
// time as the last update for those labels.
func (t *TTLCounterVec) With(labels prometheus.Labels) *TTLCounter {
key := labelsKey(labels)
t.mu.Lock()
defer t.mu.Unlock()
entry, ok := t.values[key]
if !ok {
labelValues := make([]string, len(t.labelNames))
for i, name := range t.labelNames {
labelValues[i] = labels[name]
}
entry = &ttlCounterEntry{labelValues: labelValues}
t.values[key] = entry
}
entry.lastUpdate = time.Now()
return &TTLCounter{vec: t, entry: entry}
}
// Delete removes the metric associated with the given label set.
func (t *TTLCounterVec) Delete(labels prometheus.Labels) bool {
key := labelsKey(labels)
t.mu.Lock()
defer t.mu.Unlock()
if _, ok := t.values[key]; !ok {
return false
}
delete(t.values, key)
return true
}
// Cleanup deletes all label sets that have not been updated within the TTL duration.
func (t *TTLCounterVec) Cleanup() {
now := time.Now()
t.mu.Lock()
defer t.mu.Unlock()
for key, entry := range t.values {
if now.Sub(entry.lastUpdate) > t.ttl {
delete(t.values, key)
}
}
}
// Describe implements prometheus.Collector.
func (t *TTLCounterVec) Describe(ch chan<- *prometheus.Desc) {
ch <- t.desc
}
// Collect implements prometheus.Collector.
func (t *TTLCounterVec) Collect(ch chan<- prometheus.Metric) {
t.mu.RLock()
defer t.mu.RUnlock()
for _, entry := range t.values {
ch <- prometheus.MustNewConstMetric(t.desc, prometheus.CounterValue, entry.value, entry.labelValues...)
}
}
+7 -1
View File
@@ -7,6 +7,7 @@ import (
"fmt"
"net"
"net/http"
"strings"
"sync"
"time"
@@ -55,7 +56,7 @@ func NewApiClient(endpoints []string, tokenId string, secret string, checkInterv
// Prepare API endpoints.
for _, endpoint := range endpoints {
apiEndpoint := ApiEndpoint{
host: endpoint,
host: normalizeApiHost(endpoint),
alive: false,
}
instance.endpoints = append(instance.endpoints, &apiEndpoint)
@@ -86,6 +87,11 @@ func NewApiClient(endpoints []string, tokenId string, secret string, checkInterv
return &instance
}
// normalizeApiHost ensures API paths can be appended to either IP- or DNS-based hosts.
func normalizeApiHost(host string) string {
return strings.TrimRight(strings.TrimSpace(host), "/") + "/"
}
// Check endpoint liveness state.
func (instance *ApiClient) checkEndpointsLiveness() {
// We want to make sure other routines won't make any requests until we have checked for alive connections.
+49
View File
@@ -0,0 +1,49 @@
package proxmox
import (
"testing"
"time"
)
func TestNewApiClientNormalizesEndpointHosts(t *testing.T) {
tests := []struct {
name string
host string
want string
}{
{
name: "IP without trailing slash",
host: "https://192.168.0.10:8006",
want: "https://192.168.0.10:8006/",
},
{
name: "IP with trailing slash",
host: "https://192.168.0.10:8006/",
want: "https://192.168.0.10:8006/",
},
{
name: "DNS without trailing slash",
host: "https://pve.example.com:8006",
want: "https://pve.example.com:8006/",
},
{
name: "DNS with trailing slash",
host: "https://pve.example.com:8006/",
want: "https://pve.example.com:8006/",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
client := NewApiClient([]string{tt.host}, "token", "secret", time.Second)
defer client.httpClient.CloseIdleConnections()
if got := client.endpoints[0].host; got != tt.want {
t.Fatalf("normalized endpoint host = %q, want %q", got, tt.want)
}
if got := client.endpoints[0].host + "api2/json/"; got != tt.want+"api2/json/" {
t.Fatalf("liveness URL = %q, want %q", got, tt.want+"api2/json/")
}
})
}
}
+14 -2
View File
@@ -39,7 +39,7 @@ type PveClusterStatus struct {
// PveResource represents a generic PVE resource object.
type PveResource struct {
Type string `mapstructure:"type"` // Type of resource (e.g., "lxc", "qemu", "node", "storage", "sdn").
Type string `mapstructure:"type"` // Type of resource (e.g., "lxc", "qemu", "node", "storage", "sdn", "network").
Node string `mapstructure:"node"` // Node where the resource is located.
Status string `mapstructure:"status"` // Status of the resource (e.g., "running", "stopped", "online", "available").
ID string `mapstructure:"id"` // Unique identifier for the resource.
@@ -106,9 +106,12 @@ type PveStorageResource struct {
}
// PveSdnResource represents a PVE software-defined network (SDN) resource.
// PVE 8 reports these as "sdn" resources, PVE 9 as "network" resources.
type PveSdnResource struct {
PveResource
SDN string `mapstructure:"sdn"` // Name of the SDN.
SDN string `mapstructure:"sdn"` // Name of the SDN (only "sdn" resources).
Network string `mapstructure:"network"` // Name of the network entity (only "network" resources).
NetworkType string `mapstructure:"network-type"` // Type of the network entity, e.g. "zone" or "fabric" (only "network" resources).
}
// PVE resources.
@@ -455,6 +458,15 @@ func (r *PveResources) FindNodeSDN(nodeName string) *[]PveSdnResource {
return &sdns
}
// GetName returns the name of an SDN resource, regardless of whether it was
// reported as an "sdn" (PVE 8) or a "network" (PVE 9) resource.
func (r *PveSdnResource) GetName() string {
if r.SDN != "" {
return r.SDN
}
return r.Network
}
// GetStatusNumeric returns the numeric status of an SDN resource.
// Returns 1 if the status is "ok", otherwise returns 0.
func (r *PveSdnResource) GetStatusNumeric() float64 {
+62
View File
@@ -3,6 +3,8 @@ package proxmox
import (
"encoding/json"
"testing"
"github.com/mitchellh/mapstructure"
)
func TestPveClusterStatusGetClusterModeNumeric(t *testing.T) {
@@ -112,3 +114,63 @@ func TestPveZfsPoolStatusUnmarshal(t *testing.T) {
t.Fatal("missing cache counters must stay absent instead of becoming zero metrics")
}
}
func TestPveSdnResourceGetName(t *testing.T) {
tests := []struct {
name string
resource PveSdnResource
want string
}{
{
name: "sdn resource",
resource: PveSdnResource{SDN: "localnetwork"},
want: "localnetwork",
},
{
name: "network resource",
resource: PveSdnResource{Network: "localnetwork", NetworkType: "zone"},
want: "localnetwork",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := tt.resource.GetName(); got != tt.want {
t.Fatalf("GetName() = %v, want %v", got, tt.want)
}
})
}
}
func TestPveSdnResourceDecodeNetwork(t *testing.T) {
// Resource object as reported by PVE 9 for SDN entities.
obj := map[string]interface{}{
"id": "network/pve1/zone/localnetwork",
"type": "network",
"node": "pve1",
"status": "ok",
"network": "localnetwork",
"network-type": "zone",
}
var resource PveSdnResource
if err := mapstructure.Decode(obj, &resource); err != nil {
t.Fatalf("Decode() error = %v", err)
}
if err := mapstructure.Decode(obj, &resource.PveResource); err != nil {
t.Fatalf("Decode() error = %v", err)
}
if got, want := resource.GetName(), "localnetwork"; got != want {
t.Fatalf("GetName() = %v, want %v", got, want)
}
if got, want := resource.NetworkType, "zone"; got != want {
t.Fatalf("NetworkType = %v, want %v", got, want)
}
if got, want := resource.ID, "network/pve1/zone/localnetwork"; got != want {
t.Fatalf("ID = %v, want %v", got, want)
}
if got, want := resource.GetStatusNumeric(), float64(1); got != want {
t.Fatalf("GetStatusNumeric() = %v, want %v", got, want)
}
}
+15 -15
View File
@@ -64,18 +64,18 @@ func (instance *PveApiClient) GetClusterStatus() (*PveClusterStatus, error) {
switch obj["type"] {
case "cluster":
if err := mapstructure.Decode(obj, &cluster); err != nil {
log.Errorf("Unable to decode cluster status object. Error:", err)
log.Errorf("Unable to decode cluster status object. Error: %s", err)
continue
}
case "node":
var node PveNodeStatus
if err := mapstructure.Decode(obj, &node); err != nil {
log.Errorf("Unable to decode node status object. Error:", err)
log.Errorf("Unable to decode node status object. Error: %s", err)
continue
}
cluster.NodeStatuses = append(cluster.NodeStatuses, node)
default:
log.Errorf("Unable to decode cluster status object. Unknown type:", obj["type"])
log.Warnf("Skipping cluster status object of unknown type '%v'.", obj["type"])
}
}
@@ -106,60 +106,60 @@ func (instance *PveApiClient) GetClusterResources() (*PveResources, error) {
case "lxc":
var resource PveLxcResource
if err := mapstructure.Decode(obj, &resource); err != nil {
log.Errorf("Unable to decode cluster resource object. Error:", err)
log.Errorf("Unable to decode cluster resource object. Error: %s", err)
continue
}
if err := mapstructure.Decode(obj, &resource.PveResource); err != nil {
log.Errorf("Unable to decode cluster resource object. Error:", err)
log.Errorf("Unable to decode cluster resource object. Error: %s", err)
continue
}
resources.CTs = append(resources.CTs, resource)
case "qemu":
var resource PveQemuResource
if err := mapstructure.Decode(obj, &resource); err != nil {
log.Errorf("Unable to decode cluster resource object. Error:", err)
log.Errorf("Unable to decode cluster resource object. Error: %s", err)
continue
}
if err := mapstructure.Decode(obj, &resource.PveResource); err != nil {
log.Errorf("Unable to decode cluster resource object. Error:", err)
log.Errorf("Unable to decode cluster resource object. Error: %s", err)
continue
}
resources.VMs = append(resources.VMs, resource)
case "node":
var resource PveNodeResource
if err := mapstructure.Decode(obj, &resource); err != nil {
log.Errorf("Unable to decode cluster resource object. Error:", err)
log.Errorf("Unable to decode cluster resource object. Error: %s", err)
continue
}
if err := mapstructure.Decode(obj, &resource.PveResource); err != nil {
log.Errorf("Unable to decode cluster resource object. Error:", err)
log.Errorf("Unable to decode cluster resource object. Error: %s", err)
continue
}
resources.Nodes = append(resources.Nodes, resource)
case "storage":
var resource PveStorageResource
if err := mapstructure.Decode(obj, &resource); err != nil {
log.Errorf("Unable to decode cluster resource object. Error:", err)
log.Errorf("Unable to decode cluster resource object. Error: %s", err)
continue
}
if err := mapstructure.Decode(obj, &resource.PveResource); err != nil {
log.Errorf("Unable to decode cluster resource object. Error:", err)
log.Errorf("Unable to decode cluster resource object. Error: %s", err)
continue
}
resources.Storages = append(resources.Storages, resource)
case "sdn":
case "sdn", "network":
var resource PveSdnResource
if err := mapstructure.Decode(obj, &resource); err != nil {
log.Errorf("Unable to decode cluster resource object. Error:", err)
log.Errorf("Unable to decode cluster resource object. Error: %s", err)
continue
}
if err := mapstructure.Decode(obj, &resource.PveResource); err != nil {
log.Errorf("Unable to decode cluster resource object. Error:", err)
log.Errorf("Unable to decode cluster resource object. Error: %s", err)
continue
}
resources.SDNs = append(resources.SDNs, resource)
default:
log.Errorf("Unable to decode cluster resource object. Unknown type:", obj["type"])
log.Warnf("Skipping cluster resource object of unknown type '%v'.", obj["type"])
}
}