shelly-exporter/docs/architecture.md

# Architecture and adding products

## Design goals

The exporter separates transport, product decoding, Prometheus collection, configuration, and HTTP serving so additional Shelly models do not require changes throughout the application.

The Gen1 implementation uses a superset response structure with optional fields. This matches the Shelly API, where `/status` contains common fields plus product- and mode-specific blocks.

## Request flow

```mermaid
flowchart LR
    P[Prometheus] -->|GET /metrics| H[HTTP server]
    H --> R[Prometheus registry]
    R --> G[Collector group]
    G -->|concurrent| C1[Device collector A]
    G -->|concurrent| C2[Device collector B]
    C1 --> S1[Status client]
    C2 --> S2[Status client]
    S1 -->|GET /status| D1[Shelly device A]
    S2 -->|GET /status| D2[Shelly device B]
```

There is no background polling loop or value cache. Each `/metrics` request produces one status request per configured device.

## Package responsibilities

| Package/file | Responsibility |
| --- | --- |
| [`main.go`](../main.go) | Parses `-config`, installs signal handling, and starts the application. |
| [`configuration`](../configuration/configuration.go) | Strict YAML loading, defaults, and product-independent validation. |
| [`application`](../application/application.go) | Creates clients and collectors, owns the Prometheus registry, and serves HTTP endpoints. |
| [`shelly`](../shelly/client.go) | Gen1 HTTP `/status` client, timeout, Basic Authentication, status checking, and JSON decoding. |
| [`collector/factory.go`](../collector/factory.go) | Normalizes product identifiers and maps them to collector constructors. |
| [`collector/group.go`](../collector/group.go) | Presents all device collectors as one registered collector and runs them concurrently. |
| [`collector/base.go`](../collector/base.go) | Shared `shelly_up`, request duration, and request error metrics. |
| [`collector/gen1_status.go`](../collector/gen1_status.go) | Optional superset of known Gen1 `/status` response blocks. |
| [`collector/gen1_collector.go`](../collector/gen1_collector.go) | Descriptor definitions and conversion from Gen1 status data to Prometheus metrics. |

## Startup lifecycle

1. `configuration.Load` reads and strictly parses YAML.
2. Product-independent validation checks the listener, timeout, unique names, and URLs.
3. `application.New` creates one HTTP status client per configured device.
4. `collector.NewDeviceCollector` normalizes the configured product and resolves its factory.
5. All device collectors are wrapped in `collector.Group` and registered once in a private Prometheus registry.
6. `/metrics` and `/-/healthy` handlers are installed.
7. `Application.Run` opens the listener and serves until cancellation or a server error.

A private registry prevents application metrics from being mixed with process-global collectors registered by dependencies.

## Collection behavior

`collector.Group.Collect` starts one goroutine per configured device and waits for all of them. Each Gen1 collector then:

1. calls `StatusClient.GetStatus`;
2. emits scrape health, duration, and cumulative error count;
3. stops for that device if HTTP or JSON decoding failed;
4. emits common status metrics;
5. processes relays, inputs, meters, EM meters, rollers, lights, thermostats, and sensors;
6. skips every absent optional field.

The base error counter uses an atomic integer because Prometheus can execute overlapping collections. Device-specific values are const metrics built from the latest response and do not retain mutable state.

## Why status fields use pointers

In JSON, a missing field and a present zero value have different meanings. For example:

```json
{}
```

means that a capability may not exist, while:

```json
{"power": 0}
```

means that the device supports power measurement and currently reports zero watts.

Numeric and boolean status fields therefore use pointers in `gen1Status`. Collection helpers emit a metric only for a non-nil pointer. The custom `numberOrBool` decoder handles the Gen1 `overpower` field because different product families represent it as either a boolean state or a numeric threshold.

## Metric conventions

- Use the `shelly_` namespace.
- Use Prometheus base-unit suffixes such as `_seconds`, `_bytes`, `_watts`, and `_volts`.
- Use `_total` only for counters.
- All product metrics include `device`; indexed response arrays additionally include `index`.
- Boolean values are gauges with `0` or `1`.
- Text is exposed through labels on an `_info` gauge with value `1`.
- Reuse an existing metric when a new product has the same semantic value.
- Avoid labels containing frequently changing or unbounded data.
- Preserve native cumulative device values instead of accumulating scrape deltas in exporter memory.

### Energy counter rules

Gen1 uses two different cumulative representations:

- `meters[].total` is watt-minutes and is divided by 60 before being exported as `shelly_meter_energy_watt_hours_total`.
- `emeters[].total` and `total_returned` are already Wh and are exported unchanged.

Both are Prometheus counters. Reboots or explicit device resets can decrease them; Prometheus counter functions handle the reset.

## Extending the Gen1 schema

Use this path when a product already supports `GET /status` but returns a field the generic collector does not understand.

### 1. Capture an official response fixture

Use the official Shelly documentation and, when available, a real device response. Remove passwords and private network identifiers before committing fixtures.

### 2. Add optional response fields

Extend [`gen1_status.go`](../collector/gen1_status.go). Use pointers for scalar fields:

```go
type gen1Status struct {
    // Existing fields...
    Voltage *float64 `json:"voltage"`
}
```

For a nested or repeated block, introduce a dedicated type rather than an anonymous map. A typed structure catches incompatible API changes during JSON decoding.

### 3. Register descriptors

Add each metric once in `registerDescriptors`:

```go
c.add("supply_voltage_volts", "Device supply voltage in volts.")
```

`add` automatically prefixes the name with `shelly_` and prepends the `device` label. Pass only additional label names.

### 4. Emit only present values

Use the existing helpers:

```go
c.optionalGauge(channel, "supply_voltage_volts", status.Voltage)
c.optionalBool(channel, "feature_enabled", status.FeatureEnabled)
```

Use `counter` only for a cumulative, monotonically increasing source value:

```go
if status.Events != nil && *status.Events >= 0 {
    c.counter(channel, "events_total", *status.Events)
}
```

### 5. Register the product identifier

If the new model is known to use this schema, append its normalized identifier to `gen1Products`. Unlisted compatible devices can already use `product: gen1` without registration.

### 6. Add fixture-based tests

Add a representative JSON fixture in the collector tests and gather through `prometheus.NewPedanticRegistry`. Test:

- expected metric values and labels;
- metric type, particularly counters;
- unit conversions;
- valid zero values;
- absence of a metric when the source field is absent;
- inconsistent JSON forms if the device family has firmware variations.

Run:

```sh
gofmt -w collector
go test ./...
go vet ./...
go build ./...
```

## Adding a different protocol generation

Do not model a Gen2/Gen3 RPC response as a Gen1 `/status` response. The current `shelly.StatusClient` always appends `/status` and implements Basic Authentication, so supporting an RPC generation requires a transport extension as well as a collector.

A clean implementation should:

1. define a product-independent request/client interface or a protocol client factory;
2. choose the protocol client from product metadata during application construction;
3. implement the RPC endpoint and its authentication scheme separately;
4. add a collector with generation-appropriate response structures;
5. register product identifiers through the existing collector factory;
6. retain the shared collector group and scrape health metrics.

This keeps generation-specific endpoint paths and authentication out of the application HTTP server.