Prometheus: snapshots

Prometheus writes data all the time, so a file copy of its directory can turn out inconsistent. arcula does better: it asks Prometheus to take a snapshot of its database (TSDB) and sends that. The snapshot is consistent as of the moment it's taken and takes almost no space: it's hard links to the data blocks.

The snapshot is made by the Prometheus admin API. It's off by default.

Are snapshots working

In the protection plan, Prometheus shows “Prometheus snapshot” in “How to copy”. This means the agent tries to take a snapshot on every backup. Whether it worked shows in the log.

If the snapshot fails, the backup doesn't fail: the agent copies the data directory as files and leaves a warning. Where to see it: server → “Log” → the “Backup saved” line → click it → “Log lines”. The line looks like this:

agent: prometheus_snapshot_failed: the admin API did not make a snapshot (admin APIs disabled) — the data directory is copied as files without consistency; for a snapshot start Prometheus with --web.enable-admin-api

In parentheses is the reason, as Prometheus gave it:

• admin APIs disabled — the admin API is off. Turn it on as below.
• Another Prometheus answer — the snapshot failed in Prometheus itself. Look into the Prometheus logs.
• no answer — there's no answer: Prometheus isn't running or listens on a different port than in its flags.

Until the cause is fixed, Prometheus is copied as files — without consistency.

If “How to copy” says “Prometheus snapshot — the agent can't do this yet (update it)”, the agent on the server is old. See Update agents.

Turn on the admin API

Add the --web.enable-admin-api flag to the Prometheus command. In Compose — to the service's command. Keep the flags that are already there: command replaces them all.

services:
  prometheus:
    image: prom/prometheus
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.enable-admin-api

Recreate the container:

docker compose up -d prometheus

The next backup will go as a snapshot, and there will be no warning in the log lines.

The admin API can also delete data. Don't expose the Prometheus port to the internet: the agent needs it only from inside the container, on 127.0.0.1.

How the agent takes a snapshot

• The POST /api/v1/admin/tsdb/snapshot request is sent from inside the Prometheus container — with curl, nc from busybox or wget. No external port is needed.
• The agent takes the port and prefix from the Prometheus flags: --web.listen-address, --web.route-prefix or --web.external-url.
• The snapshot lands in <data directory>/snapshots/<name>, goes into the backup in place of the data directory and is deleted after sending.
• Docker is required: the snapshot is made in the container.

Restore

The snapshot lies in the backup where the data directory would be. arcula restore brings it back together with the volume or directory, and Prometheus reads it at startup. No separate steps are needed — see Restore data.