arculaDocumentation

Applications

Vault: Raft snapshots

By default Vault is copied as the files of its data directory — without consistency: the files are read while Vault is running. For Vault on Raft there is a better way — a snapshot through the Vault API, consistent as of the moment it's taken. You turn it on yourself: it needs setup on the server. arcula doesn't do a cold backup with Vault stopped.

Which Vault you have

The server's protection plan → “Application data”. The agent names Vault by its storage type:

  • “Vault, Raft” — a snapshot is possible;
  • “Vault, file storage” — no snapshot, only a file copy. Updating the agent won't change this. How to move to Raft — below;
  • just “Vault” — the agent couldn't read the storage type. Why — see the “Why” button.

Turn on snapshots

1. The plan is confirmed

Vault must already be in the accepted plan — for now as a file copy: arcula vault setup looks for it there.

2. On the server

sudo arcula vault setup

The command:

  1. finds Vault in the plan and its address: the container IP, and the scheme and port from its VAULT_ADDR, port 8200 by default;
  2. checks that Vault is unsealed and its storage is raft;
  3. asks for a Vault admin token without echo; the token isn't saved;
  4. creates the arcula-snapshot policy with a single permission — to read a snapshot — and the arcula-snapshot AppRole login. Login tokens live 15 minutes and are revoked after each snapshot;
  5. takes a test snapshot and puts role_id and secret_id in /etc/arcula/vault/ with 0600 permissions.

The command doesn't need the unseal keys.

  • If the plan has several Vaults, name the one you need by its resource key: sudo arcula vault setup db:vault/vault/vault. The key is in the plan, under the “Why” button.
  • If the address wasn't found, set it: --address http://127.0.0.1:8200. It will be saved.
  • The token can also be passed in the VAULT_TOKEN variable.

3. In the panel

Protection plan → the Vault row → “How to copy” → “Vault snapshot (raft) — needs setup on the server” → “Save plan”.

From the next backup on, Vault goes as a snapshot: the backup contains the file vault-raft.snap.

If the snapshot fails — Vault is sealed, the storage isn't raft, the login failed — the agent copies the data directory as files and leaves the warning vault_snapshot_failed. It's in the backup's log lines: server → “Log” → “Backup saved” → “Log lines”.

Restore from a snapshot

arcula restore doesn't apply the snapshot: that needs the unseal keys, and the agent doesn't have them. It extracts the snapshot to /var/lib/arcula/restore/<id>/vault or to the directory from --target. From there, the standard way.

Bring up Vault with the same Raft configuration. If it's empty — vault operator init, unseal it with the temporary keys and log in with the temporary root token. If Vault runs in a container, first copy the snapshot inside: docker cp. Then:

vault operator raft snapshot restore -force vault-raft.snap

After that, unseal Vault with the original unseal keys and check vault status and the secrets.

Keep the unseal keys separately: they are neither in the backup nor in the recovery kit. The panel reminds you about them on the server: “Overview” → “Needed for a restore”.

Move from file storage to Raft

With file storage there's no snapshot: arcula vault setup refuses — “Vault storage is file, not raft”. Moving Vault to Raft is your step. Vault is off during it, usually for minutes.

The example is for Compose with the volume ./vault/data:/vault/data and the configuration in the VAULT_LOCAL_CONFIG variable. Run the commands in the project directory.

1. Stop Vault

Stop Vault and everything that depends on it, and save a copy of vault/data — this is your rollback point:

docker compose stop vault
sudo cp -a vault/data vault/data.bak

2. Directory for Raft

The directory's owner is the vault user from the image. Find its uid and gid:

docker compose run --rm --no-deps --entrypoint id vault vault
mkdir vault/raft
sudo chown <uid>:<gid> vault/raft

3. Migration file

Next to the Compose file, create migrate.hcl:

storage_source "file" { path = "/vault/data" }
storage_destination "raft" {
  path    = "/vault/raft"
  node_id = "vault-1"
}
cluster_addr = "http://127.0.0.1:8201"

4. Migration

docker compose run --rm --no-deps -u vault -v "$PWD/vault/raft:/vault/raft" -v "$PWD/migrate.hcl:/vault/migrate.hcl" --entrypoint vault vault operator migrate -config=/vault/migrate.hcl

The command only reads the original vault/data.

5. Configuration

In VAULT_LOCAL_CONFIG, replace "backend": {"file": {…}} with "storage": {"raft": {"path": "/vault/raft", "node_id": "vault-1"}} and add "cluster_addr": "http://127.0.0.1:8201". Add ./vault/raft:/vault/raft to the service's volumes.

6. Start

docker compose up -d vault

Unseal Vault with the same unseal keys. vault status should show Storage Type raft. Check the secrets, then bring up the other services.

7. Snapshots

In the panel: server → “Protection settings” → “Discover now”, or wait for discovery. Vault becomes “Vault, Raft”. Then — sudo arcula vault setup and “Vault snapshot (raft)” in the plan, as above.

To roll back, put backend file back in the configuration. Once snapshots are running, ./vault/data can be removed from Compose and from the disk.

The same article is in the panel, under “Help”.