Server statuses

The server status shows in the “Servers” list — as a mark next to the name and short lines under it. On the server page, the same reasons are written as full sentences under the title. Each one has a “?” that leads to the matching section of this article.

The status is recalculated on its own. When the cause goes away, so does the line.

Levels

Freshness is measured against the target of the profile chosen in the plan:

Copies in a storage with no access don't count toward freshness. A disconnected server is at the end of the list, with no status.

Protection plan not confirmed

In the panel: “Protection plan not confirmed — no backups”. Status: “Not set up”.

The server is connected and the agent has already found what's on it, but you haven't decided what to protect yet. Without a plan the agent makes no backups: if the server fails now, everything is lost.

When it appears: right after the server is connected, until the plan is confirmed for the first time.

What to do: on the server page, click “Confirm plan”. Choose what to protect, the frequency profile and the storages — Choose what to protect. The agent gets the plan within a minute and makes the first backup right away.

Goes away on its own: no, only after the plan is confirmed.

No fresh backup in any storage

In the panel: “No fresh backup in any storage: the latest was 26 h ago”. Status: “Critical”.

The freshest backup across all accessible storages is older than two profile targets. If the server fails now, you lose more than you set in the plan.

When it appears: the backup is older than 2 h for “Critical”, 24 h for “Standard”, 72 h for “Rarely changes”. If the plan is confirmed but no backup has been saved at all — after the same two targets from the confirmation.

What to do:

1. Look at the other status lines. The cause is usually there: the agent is offline, a storage has no access, a component isn't collected.
2. Open the “Log” tab: lines “Backup not saved”, “Not backed up”, “Runs missed”.
3. Click “Back up now” and see at which step it stops — Back up now.

The same is visible on the server itself:

sudo arcula status
sudo arcula doctor

While the agent is silent, the panel checks the list of backups in its repositories once an hour. Backups made without a connection count too.

Goes away on its own: yes, as soon as a fresh backup is saved in at least one accessible storage.

Component not in the backup

In the panel: “PostgreSQL 16 is not in the backup: …”. In the server list: “Not in the backup: PostgreSQL 16” or “Backup incomplete: 3 components missing”. Status: “Critical”.

The latest backup is missing a required component from the plan: the agent couldn't collect it. After the colon is the reason as is, for example pg_dump output or No space left on device. Everything else in this backup was collected and saved.

When it appears: right after the first backup without this component.

What to do:

• Read the reason. The full text and log lines are on the “Log” tab, in the line “Not backed up: …”.
• The database doesn't respond or the dump failed — fix it on the server and click “Back up now”.
• Not enough space in staging — Not enough space in staging.
• The path or volume is no longer on the server — the log shows “Not on the server: …”. Remove it from the plan — Gone from the server.
• You need this component right now — it's in an earlier backup, on the “Backups” tab — Backups.

Goes away on its own: yes, when the next backup collects the component.

Copy is behind

In the panel: “Secondary: 18 h behind · 36 backups not saved in a row”. Status: “Needs attention”.

Backups stopped reaching one storage, while another one has a fresh backup. “Not saved in a row” is how many backups after the last one saved here didn't arrive: they failed to upload, or were skipped in favor of a newer one.

When it appears: the latest backup in this storage is older than the profile target — 1 h, 12 h or 36 h. If there is no fresh backup anywhere, you see No fresh backup instead of this line.

What to do:

• Open the server's “Storages” tab: the copy shows what the storage answered. What each error means — Copy errors.
• This storage has an “at night” upload window — the copy is behind during the day, as intended.
• The log has many “Backup skipped” lines — the link to the storage can't keep up with the schedule. Choose a less frequent profile, or send to this storage at night.

Goes away on its own: yes, with the next backup saved here. Skipped backups aren't caught up: only the latest ready one is sent.

Copy: no access

In the panel: “idr-fra-01: no access · 403 AccessDenied”. Status: “Needs attention”.

The panel's access check for this storage failed. After the dot is the S3 response as is. While there is no access, copies in this storage don't count toward the “Latest copy”.

When it appears: after a failed access check. The panel checks access every 6 hours, after a key change, and when you click “Check access”.

What to do: open the storage — the same error is there in more detail: No access. Once it's fixed, click “Check access” on the storage page.

Goes away on its own: after the next successful check — within 6 hours, or right away with the button.

Agent offline

In the panel: “Agent offline for 26 h, last seen yesterday 11:58”. On its own, this line gives “Needs attention”.

The agent stopped sending its heartbeat to the panel. This doesn't mean there are no backups: the agent keeps making them by the last plan without a connection and sends the log later. While it's silent, the panel checks its repositories in the storages once an hour and counts the backups it finds toward the “Latest copy”.

When it appears: no heartbeat for more than 3 minutes. The agent sends one every minute.

What to do: check that the server is running and can reach the panel over HTTPS. On the server:

sudo systemctl status arcula-agent
sudo journalctl -u arcula-agent --since "1 hour ago"
sudo arcula status

If the problem is the connection to the panel, arcula status prints “Panel unreachable since …” and shows the loss from its own data.

Goes away on its own: yes, as soon as the agent is back online.

Not in the backup

In the panel: “Volume prometheus_v3_data is not in the backup · 9.4 GB”, “Directory /srv/data is not in the backup” or “PostgreSQL 17 — new, not in the protection plan: include it or mark it “don't protect””. If there are more than two, the server list shows one line: “Unprotected: 5 resources — decide in the plan”. Status: “Needs attention”.

The agent found something on the server that isn't in the plan and that you haven't decided on yet:

• a volume or directory with unknown data — it's in “Needs a decision”;
• something recognized after the plan — a new database, data of a new application: it's in the plan with the “new” label and already checked; you only need to save the plan.

You get a notification about it once — “found and not protected”.

When it appears: after the server scan that found it. The agent scans the server every 1, 6 or 24 hours — as set in the plan, every 6 hours by default. Stopped containers and external data don't go here: they are in “Not in the backup”.

What to do: open the plan — the finding is in “Needs a decision”. Include it in the backup or mark it “don't protect” with a reason, and save the plan — Needs a decision.

Goes away on its own: no. Only after a decision in the plan, or if the volume or directory disappears from the server.

No key set for the server

In the panel: “Secondary: no key set for the server — backups aren't sent here; set it in the storage → “Connection” → “Server keys””. Status: “Needs attention”.

The storage uses the “own key per server” mode, and this server has no key yet. The agent doesn't even try to send a backup there — the copy fails right away with the access_missing error.

When it appears: right after the storage's key mode is changed, or after such a storage is added to the server's plan.

What to do:

1. At the provider, create a key with access only to the server's folder. The folder path is shown in “Server keys”.
2. Storage → “Connection” → “Server keys” → “Set key” → “Check and save”.

The panel checks the key in the server's folder before saving, and the next backup goes to this storage.

Goes away on its own: no, only after the key is saved.

Not enough space in staging

In the panel: “3 GB of 40 GB free in staging — the next backup (≈ 4 GB) won't fit”. Status: “Needs attention”.

Staging is space on the server's disk where database dumps and host settings wait to be sent to all storages. Directories and volumes aren't copied there: restic reads them straight from disk. The staging size is the limit from the plan, 40 GB by default, but no more than the free disk space. A set stays there while at least one storage is waiting for it.

When it appears: free space in staging is less than the estimate of the next backup — the size of the last set plus 10 %. The agent reports space every minute.

What to do:

• Free up space on the server's disk.
• Remove from staging what nobody is waiting for. The command doesn't touch backups in the storages.
• Raise the limit: server → “Protection settings” → “Staging” → “Edit”.
• Staging is taken by a set waiting for a slow storage — deal with that first: Copy is behind.

sudo arcula clean --dry-run
sudo arcula clean

If this isn't fixed, the next component won't be collected — Component not in the backup.

Goes away on its own: yes, when there is enough space.

Server clock is behind or ahead

In the panel: “Server clock is 3 min behind — S3 may answer 403 RequestTimeTooSkewed”. Status: “Needs attention”.

The server's clock differs from the panel's clock. S3 checks the time in every request and refuses when the difference is large. Then backups stop being saved, with the s3_clock_skew error.

When it appears: the difference is more than 2 minutes, by the agent's last heartbeat.

What to do: turn on time sync on the server.

sudo timedatectl set-ntp true
timedatectl status

The output should have System clock synchronized: yes. If this line appears on all servers at once, the clock on the panel's server is most likely off.

Goes away on its own: yes, with the next heartbeat after the clocks agree.

Backup takes almost the whole interval

In the panel: “Backup takes 28 min; scheduled every 30 min”. Status: “Needs attention”.

A backup — preparing and sending to all storages — takes almost as long as the time between runs. A bit more, and backups will run less often than in the plan, and the loss will grow.

When it appears: the usual duration — the median of the last 10 backups — is more than 80 % of the schedule interval.

What to do:

• Open the latest backup on the “Backups” tab: it shows how long preparation and sending to each storage took.
• Sending to one storage takes long — a slow link, or a speed limit in its upload settings.
• Preparation takes long — usually it's a large database dump.
• It can't be made faster — choose a less frequent profile in the plan, the “How often” card.

Goes away on its own: yes, when the usual duration drops below 80 % of the interval. It's the median of 10 backups, so the line doesn't go away after the first fast backup.

Dump size is unusual

In the panel: “PostgreSQL 16: 0.1 GB — usually 2.0–2.2 GB”. Status: “Needs attention”.

A component in the latest backup is much smaller or larger than usual. The backup itself is saved — this is not an arcula error. But a sharp drop in size often means data is gone on the server itself: a table was deleted, or the application writes to an empty database.

When it appears: the size is below half the usual minimum or above twice the usual maximum. “Usual” is the range over the last 14 days without the 10 % of extreme values. At least 7 backups are needed, otherwise there is nothing to compare with.

What to do:

• Check the data on the server.
• The data is gone — pin the last good backup as a pinned point so cleanup doesn't delete it: Pinned points. How to bring the data back — Restore.
• The size changed for a valid reason — old data was cleaned up or new data was loaded — nothing to do.

Goes away on its own: yes, when the size returns to usual. If the new size stays, over time it becomes “usual” itself.

Server disconnected

In the panel: “Server disconnected: agent keys revoked · last seen …”. Such a server is at the end of the list, with no status or reasons.

The server was disconnected in the panel, or with sudo arcula logout or sudo arcula uninstall on the server itself. The agent no longer contacts the panel and makes no backups. History and backups in the storages remain.

What to do: to connect it again — run sudo arcula logout on the server, then sudo arcula login. To remove it completely — “Protection settings” → “Delete server” — Delete a server or storage. The disconnected server still has its S3 keys — rotate them if the server is no longer yours.