Copy errors

A copy is a backup in one storage. If a copy isn't saved, the agent records the error code and the restic line as is. Where you see it:

• in the server and storage log — the “Backup not saved” line; the details have the storage's answer and the restic log;
• on the server's “Storages” tab — why the copy is behind;
• on the server itself — sudo arcula backup and sudo arcula status write the same, in English.

For some codes the panel shows a plain-language phrase, for the rest the restic line: it's more accurate than any retelling. arcula doesn't guess the cause. A 403 can come from a revoked key, a bucket policy, or a clock that is off.

How the agent retries

• restic retries every failed request itself — the log shows “returned error, retrying after …”. If the upload makes no progress for 10 minutes, the agent stops it.
• Errors a retry won't fix — wrong key, no permissions, no bucket, clocks differ — the agent stops after three such responses in a row.
• The next attempt is with a newer backup. If there is no newer one yet, the same backup is retried: after 1, 2, 4 minutes and on, the pause growing up to 30 minutes.
• There is no queue of skipped backups: the storage catches up with only the latest ready one. Other storages don't wait — each has its own queue.

Connection drops

In the panel: “the connection drops during upload (i/o timeout); retries didn't help”.

restic couldn't reach the storage. The log has dial tcp: i/o timeout, connection refused, connection reset, no such host, network is unreachable, TLS handshake timeout, unexpected EOF and similar.

Retries on its own: yes — restic retries, then the next backup.

What to do: check that the storage can be reached from the server.

curl -sI https://<storage address>
sudo arcula doctor

In arcula doctor, look at the repo:<storage> line. Common causes — DNS, a firewall, the server's outbound traffic, a provider outage.

403 AccessDenied

The storage answered 403 AccessDenied, for example client.PutObject: Access Denied. The panel shows the restic line as is.

There is a key, but it isn't allowed to work with this bucket or folder: permissions are missing, the bucket policy restricts them, or the server key wasn't issued for its own folder.

Retries on its own: the agent stops uploading after three refusals in a row and tries again with the next backup. Until the permissions are fixed, this won't help. In sudo arcula status such a copy is unavailable.

What to do: check the key's permissions at the provider: list, read, write and delete in the storage prefix. For the “own key per server” mode — in the server's folder arcula/<server id>/. Then “Check access” on the storage and “Back up now” on the server.

The storage doesn't accept the key

The response is 403 InvalidAccessKeyId or 403 SignatureDoesNotMatch; MinIO answers Your account is disabled. The panel shows the restic line as is.

The key no longer exists — it was deleted, revoked or disabled — or the secret is wrong.

Retries on its own: the agent stops after three refusals in a row. The following backups will fail the same way until the key is replaced.

What to do: issue a new key at the provider and set it in the storage, on the “Connection” tab. For the “own key per server” mode — “Server keys” → “Replace”. The panel checks the key, and the agents get it with the next heartbeat.

No bucket

In the panel: “the storage has no such bucket — create it at the provider or reconnect the storage with the correct name”. The storage's response is NoSuchBucket.

The bucket was deleted or renamed.

Retries on its own: the agent stops after three refusals in a row; after that, the same while there is no bucket.

What to do: compare the name on the “Connection” tab with what the provider has. If the bucket was deleted, the backups in it are gone. Create it again with the same name — the agent creates a repository in it on the next upload. Or connect another storage and use it instead of this one in the plan: the address and bucket of a connected storage can't be changed in the panel.

Clock differs from the storage

The response is 403 RequestTimeTooSkewed: the time in the request is too far from the storage's time. The panel shows the restic line as is.

Retries on its own: the agent stops after three refusals in a row. Until the clock is fixed, the following backups fail the same way.

What to do: turn on time sync on the server.

sudo timedatectl set-ntp true

If the server clock differs from the panel's by more than 2 minutes, the server already has a warning — Server clock is behind or ahead.

503 SlowDown

In the panel: “the storage answers 503 SlowDown — it asks for fewer requests; restic retries didn't help”. The log has 503 SlowDown: Please reduce your request rate.

The storage limits the request rate.

Retries on its own: yes. restic retries with pauses; if the upload makes no progress in 10 minutes, the agent stops it and tries with the next backup.

What to do: it usually passes on its own. If it happens every day, limit the upload speed to this storage or send at night: server → “Protection settings” → “Sending”. Ask the provider about its request limit.

Error on the storage side

The response is 500 InternalError, 502 Bad Gateway, 503 Service Unavailable or 504 Gateway Timeout. The panel shows the restic line as is.

Retries on its own: yes — restic retries, then the next backup.

What to do: wait and check the provider's status page. If it lasts for hours, contact the provider. Meanwhile the other storages keep saving backups as usual.

Repository is locked

The log has repository is already locked exclusively or unable to create lock.

The repository is busy with another operation, and the agent didn't get it within 30 minutes. Usually it's the panel itself — the nightly data check or storage cleanup. Less often — restic run by hand.

Retries on its own: yes, with the next backup.

What to do: look at the storage log — whether a data check or cleanup was running at that time. If the lock was left by a crashed manual restic run, remove it on the server:

sudo sh -c 'set -a; . /etc/arcula/repos/<storage id>.env; set +a; /var/lib/arcula/bin/restic unlock'

restic unlock removes only stale locks: it won't interrupt a running operation.

Set removed from staging

The log has, for example, staging 0 of 40 GB free, set removed. The panel shows the line as is.

The backup set — dumps and host settings — was waiting to be sent to this storage, but had to be removed from staging: there wasn't enough space for a new one. Or the set is missing after an agent restart.

Retries on its own: yes, the next backup goes here as a new set. This backup won't be in this storage.

What to do: if it repeats, staging is too small for a slow storage — Not enough space in staging.

No key for the storage

In the panel: “the server has no key for this storage — set it: storage → “Connection” → “Server keys””.

The storage uses the “own key per server” mode, and this server has no key. restic doesn't even start — the copy is not saved right away.

Retries on its own: without a key there is no point. As soon as the key is set, the next backup goes here.

What to do: No key set for the server.

Other error

Anything the agent didn't recognize. The panel shows the last restic line as is. This also covers a set you removed yourself with sudo arcula clean --waiting: “point set removed by the owner”.

Retries on its own: yes, with the next backup.

What to do: read the line and the log in the details of the log entry. On the server:

sudo arcula doctor
sudo journalctl -u arcula-agent --since "1 hour ago"