Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ STREAMR_STREAM_ID="0x567853282663b601bfdb9203819b1fbb3fe18926/m3tering/test"
STREAMR_CRONSCHEDULE="0 * * * *" # Every hour

# Optional: Prover Node (disabled publishing to prover if not set)
# Set it on managed gateways too: they send readings they decide while central is away.
# PREFERRED_PROVER_NODE="https://prover.m3ter.ing"
# PROVER_CRONSCHEDULE="0 0 * * *" # Every 24 hours

Expand All @@ -58,6 +59,10 @@ STREAMR_CRONSCHEDULE="0 * * * *" # Every hour
# GATEWAY_FALLBACK_PORT=3011 (not 3001: ChirpStack's Basic Station bridge uses it)
# Gateway-owned tunnel policy; central cannot override these targets.
# AGENT_SSH_PORT=22
# How long a Terminal shell waits for its page to reconnect (phones drop background pages), in ms.
# TERMINAL_RESUME_MS=600000
# Containers restart themselves if their DNS settings stay empty this long after a reboot (ms).
# CONSOLE_DNS_RESTART_MS=180000
# Used only by compose.central.yaml's TLS proxy.
# CENTRAL_DOMAIN=console.example.invalid
# Central with no gateway selected: ChirpStack opens this dashboard (an origin, no path),
Expand Down
23 changes: 22 additions & 1 deletion auto-compose.sh
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,28 @@ if [[ "$role" == central ]]; then
"${compose[@]}" up --detach tls-proxy || { echo 'Console is running, but the TLS proxy failed to start; check ports 80/443.' >&2; exit 1; }
fi
"${compose[@]}" ps
echo 'Console updated. Database, extension volumes and .env were preserved.'
# Plain `docker compose logs -f`, `ps` and so on in this folder follow the role that is now running:
# Compose reads COMPOSE_FILE and COMPOSE_PROJECT_NAME from .env. Only these two lines are touched;
# rewriting in place keeps the file's owner-only permissions.
set_env_value() {
local name=$1 value=$2 tmp
[[ "$(env_value "$name")" == "$value" ]] && return 0
tmp=$(mktemp) || return 1
if grep -q "^[[:space:]]*$name[[:space:]]*=" .env; then
sed "s|^[[:space:]]*$name[[:space:]]*=.*|$name=$value|" .env > "$tmp"
else
{ cat .env; [[ -z "$(tail -c 1 .env)" ]] || echo
[[ "$name" != COMPOSE_FILE ]] || echo "# Set by ./auto-compose.sh: plain 'docker compose logs -f' follows the role running here."
echo "$name=$value"; } > "$tmp"
fi
cat "$tmp" > .env; rm -f "$tmp"
}
if set_env_value COMPOSE_FILE "$compose_file" && set_env_value COMPOSE_PROJECT_NAME "$project"; then
echo "Logs: run 'docker compose logs -f' here (it follows $compose_file)."
else
echo "Note: could not record the role in .env; use: docker compose -f $compose_file logs -f" >&2
fi
echo 'Console updated. Database, extension volumes and your .env settings were preserved.'
if [[ "$role" == gateway ]]; then
echo 'Gateway agent is running. Its card on central should show Online within a minute.'
if [[ -n "$(env_value AGENT_ENROLLMENT_TOKEN)" ]]; then
Expand Down
Binary file added docs/M3tering-Console-Manual.pdf
Binary file not shown.
116 changes: 110 additions & 6 deletions docs/deployment-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,38 @@ same meter. Fix the error shown and run the script again.
The gateway's own LAN desktop at `http://<gateway-ip>:3000` keeps working, without
a login.

## The prover

Readings are sent to the prover (`PREFERRED_PROVER_NODE`) in two ways:

- **As each reading arrives:** once per reading. On central this covers every gateway's readings,
including those handed over after a gateway decided them on its own.
- **On `PROVER_CRONSCHEDULE`:** every stored reading whose nonce is not yet confirmed on-chain is sent
again, on every run, until the chain confirms it. Each run first checks the chain and prunes the
readings that are confirmed.

While central is unreachable, a managed gateway sends the readings it decides itself to the prover, if
its own `.env` sets `PREFERRED_PROVER_NODE`. Central's scheduled runs resend them later anyway. A gateway
never publishes to Streamr or Arweave; central does that.

Logs: `Sending N transactions to prover at …`, then `Prover response (text): …`.

## The Streamr app

Every publish, the hourly `STREAMR_CRONSCHEDULE` run or **Publish Now**, sends all stored readings
whose nonce is not yet confirmed on-chain. They are sent again on every run until the chain confirms
them; then they are pruned. Only Arweave uploads happen once per reading.

| Where | What it shows | Publish Now |
|---|---|---|
| Standalone | Readings not yet confirmed on-chain | Publishes them to every configured stream |
| Central, **Central server** selected | Readings not yet confirmed on-chain, from every gateway | Publishes them, for every gateway |
| Central, a gateway selected | The same, for that gateway's meters only | Same as above: central publishes for every gateway together |
| A managed gateway's own Console | How many readings are kept locally; central publishes them | Hidden |

The window refreshes when you open it, after **Publish Now**, when you switch gateway and every 30
seconds while open. **Last publish** covers both the hourly job and **Publish Now**.

## Managing gateways on central

Open **Gateways** from the top bar. Each gateway is a card.
Expand Down Expand Up @@ -219,7 +251,8 @@ The script:
- pulls the newest image for the tag the machine already runs;
- replaces only the Console services and waits for them to be healthy;
- restores the previous image if they aren't;
- never touches `.env`, the database or volumes.
- never touches the database or volumes, and in `.env` only updates its own `COMPOSE_FILE` and
`COMPOSE_PROJECT_NAME` lines (so plain `docker compose logs -f` follows the role).

To run a specific build, pass its tag or digest:

Expand Down Expand Up @@ -279,7 +312,7 @@ a gateway) and Terminal and ChirpStack work on the central machine instead of a
| **Terminal** | SSH into the central server itself. Sign in with a Linux user and password on that server. |
| **ChirpStack** | Opens the dashboard set in `CENTRAL_CHIRPSTACK_URL`. |

Choosing a gateway switches both back to that gateway. Open windows close when
Choosing a gateway switches both back to that gateway. Each browser tab keeps its own choice, so you can work on one gateway in one tab and another gateway (or central) in a second tab; a new tab starts on **Central server**. Open windows close when
you switch, so you never type into the wrong machine.

Settings in central's `.env` (run `./auto-compose.sh` after changing them):
Expand All @@ -289,6 +322,12 @@ CENTRAL_CHIRPSTACK_URL=https://chirpstack.example.com # address only, no path
CENTRAL_SSH_PORT=22 # only if sshd uses another port
```

**Terminal sessions survive switching apps.** If the browser drops the connection (a phone putting
the page in the background, switching tabs, a network blip), the shell keeps running for 10 minutes
(`TERMINAL_RESUME_MS`). When you come back, Terminal reconnects to the same session by itself and shows
what you missed, even if the browser reloaded the page. **Disconnect** or closing the Terminal window
ends the session at once. On central, only the same signed-in user on the same target can resume it.

**Terminal needs password login if your server doesn't already use it.** The Terminal signs in with a username and password,
and many cloud servers allow SSH keys only. To use it, the user needs a password
(`sudo passwd <user>`) and sshd must accept passwords for it. For example, add this
Expand Down Expand Up @@ -344,15 +383,37 @@ are listed in [`.env.example`](../.env.example) and

## Troubleshooting

Useful commands, using the compose file for the machine's role
(`compose.yaml`, `compose.central.yaml` or `compose.gateway.yaml`):
**Logs on any machine.** In the Console folder, plain Compose commands follow whatever role is
running there: central (Console and HTTPS proxy), a managed gateway (Console and agent) or standalone.

```sh
docker compose logs -f # every service of this role
docker compose logs -f agent # one service
docker compose ps
```

This works because `./auto-compose.sh` keeps two lines in `.env` up to date after each successful
deploy: `COMPOSE_FILE` (the role's compose file) and `COMPOSE_PROJECT_NAME`. It doesn't change
anything else in the file. The explicit form still works and is what you need for the gateway
updater, whose compose file is only added by the script:

```sh
docker compose -f compose.gateway.yaml ps
docker compose -f compose.gateway.yaml logs -f console
docker compose -f compose.gateway.yaml logs -f agent
docker compose -f compose.gateway.yaml -f compose.updater.yaml logs -f updater
```

**Following a meter report.** Each step logs one line with the reading and the reply:

| Where | Lines |
|---|---|
| Gateway `logs -f agent` | `Listening for meter reports on …`, `Report received from ChirpStack: devEui=… nonce=… energy=…kWh …`, `Report … is now …`, `Reply enqueued to ChirpStack for … (queue id …): state=ON …` |
| Central `logs -f console` | `Report received: …`, `Reading recorded: …`, `Reply decided (the gateway agent enqueues it): state=…` or `Report rejected: <reason>` |
| Gateway `logs -f console` | The same as central, but only while central is unreachable and the gateway decides locally |
| Standalone `logs -f console` | `Report received: …`, `Reading recorded: …`, `Reply enqueued: state=…` |

Messages on ports other than 2 (telemetry, network-control frames) aren't billed and are dropped without a log line.
The full signature and meter key are shortened in the logs; keys and tokens never appear.

| Symptom | Cause and fix |
|---|---|
| `Replace the placeholder values in .env: …` | Template values are still in `.env`. Fill in the named settings. |
Expand All @@ -365,10 +426,53 @@ docker compose -f compose.gateway.yaml logs -f agent
| Central Terminal: `SSH connection failed` | With **Central server** selected, the server's sshd refused the password. Allow password login for that user from `127.0.0.1` (see [Working on the central server itself](#working-on-the-central-server-itself)). |
| Central ChirpStack: `No ChirpStack dashboard set` | Set `CENTRAL_CHIRPSTACK_URL` in `.env` and rerun `./auto-compose.sh`, or select a gateway. |
| Gateway: `Port 3011 is already used by …` | Another program uses the gateway Console's local fallback port. Add `GATEWAY_FALLBACK_PORT=3101` (any free port) to `.env` and rerun. Older setups used 3001, which clashes with ChirpStack's Basic Station bridge. |
| Gateway shows **Offline** after a reboot although the gateway has internet; agent log says `Cannot reach central (EAI_AGAIN)` | Docker started the containers before the network (often Wi-Fi) had DNS, so they got empty DNS settings. They now restart themselves after 3 minutes without DNS and connect. To fix it at once: `docker compose restart console agent`. |
| Gateway: agent online once, then **Offline** for good | Usually the gateway Console couldn't start, often because of a port clash. Check `logs console` and rerun `./auto-compose.sh`; it now says what's wrong. |
| Gateway: `This gateway is not enrolled yet` | Add the `AGENT_ENROLLMENT_TOKEN` line from central. |
| Every reply is `state=OFF`; logs show `Failed to resolve ENS name: base-sepolia.verifier.unruggable.eth` | Known issue, see [Every meter is switched OFF](#known-issue-every-meter-is-switched-off). |
| Gateway: `Agent enrollment failed` in agent logs | The token expired or was already used. Get a new one from the gateway's card. |
| Gateway card stays **Waiting to connect** | The gateway can't reach central on port 443, or `CENTRAL_URL` is wrong. Try `curl -I https://console.example.com` from the gateway. |
| Gateway card shows **Offline** after a revoke/delete | Expected. To reconnect, see [Reconnecting a revoked or deleted gateway](#managing-gateways-on-central). |
| `Secret file must be owner-only` in logs | Someone changed permissions in `.data/agent/`. Run `./auto-compose.sh` again; it repairs them. |
| Script says *standalone was NOT restarted* | The agent had started handling reports when something failed. Fix the error shown and rerun; don't start standalone by hand. |

## Known issue: every meter is switched OFF

**Status:** open, not fixed in this repository (checked 2026-11-18).

**Symptom.** Every state reply is OFF (`hex=000000000000000000`), whoever decides it. The deciding
Console logs:

```text
[error] [utils] Extension hook isOnStateCompute failed: Failed to resolve ENS name: base-sepolia.verifier.unruggable.eth
[error] [Device-…] Could not determine meter power state, so the reply is OFF: 1 extension(s) failed in isOnStateCompute
```

**Cause.** The ON/OFF rule (`core/is_on`) switches a meter ON only when its revenue covers what it
owes. Revenue is read through the verifiers listed in the CCIP revenue reader contract
(`CCIP_REVENUE_READER_ADDRESS`, default `0xD648cdF47e9534B2FCfb18C1E94CA9AAff07BA0E` on Sepolia).
That contract lists exactly one verifier, `base-sepolia.verifier.unruggable.eth` (target
`0xC08E39789A93230069c956e76B0c10cbC83D903e`). That ENS name has no address record, neither
on Sepolia (no resolver) nor on Ethereum mainnet (a resolver, but no address for any chain), so
the revenue lookup fails and the safe default, OFF, is used.

This is not a Console bug and not specific to one machine: central and gateways give the same
result. Check it from any Console with:

```sh
docker compose -f compose.central.yaml exec console node -e "const {JsonRpcProvider}=require('ethers');new JsonRpcProvider(process.env.MAINNET_RPC).resolveName('base-sepolia.verifier.unruggable.eth').then(a=>console.log('resolved',a))"
```

`resolved null` means the issue is still present.

**Possible fixes (not done yet).**

- Have the revenue reader contract's owner update its verifier entry to a name (or verifier) that
currently resolves. Unruggable's [verifier deployments](https://gateway-docs.unruggable.com/verifiers/deployments)
page says its verifiers are deployed on Ethereum mainnet and Sepolia.
- Point `CCIP_REVENUE_READER_ADDRESS` at a revenue reader whose verifiers resolve.
- Add a Console setting that maps a verifier name to a known verifier address, so a broken ENS
record can be bypassed.

Until then, meters stay OFF. Billing still records every reading; only the state reply is affected.

Loading