Re-verifying failed backups with Proxmox Backup Server

I’m running Proxmox Backup Server using Backblaze B2 (an S3 compatible object storage service) for storing backups. This does work fine, but my scheduled verification (I verify once a week) regularly fails on some backups due to what appears to be transient issues with the B2 API. I simply get these entirely unhelpful entries in my verification log:

verify backblaze:ct/{id}/{timestamp}/root.pxar.didx failed: chunks could not be verified

I wanted to have a simple script that would attempt a re-verification job of any backups that are currently marked as ‘failed’ on the backup store, so I can simply re-try: If verification still fails I assume the backup is broken. But otherwise I can just move on.

The script is pretty straightforward: First get a list of failed backups, then create a new verification ensuring we pass in --ignore-verified false so it actually re-verifies.

You can run the script with --dry-run to see a list of any backups that would be re-verified.

#!/bin/bash
DATASTORE="backblaze" # replace with your datastore name!

# Verify Dry Run
DRY_RUN=false
if [‎[ "$1" == "--dry-run" ]‎]; then
    DRY_RUN=true
    echo ""
    echo "-------------------------------------------------"
    echo "Dry run mode — no verification will be triggered."
    echo "-------------------------------------------------"
    echo ""
fi

# Wait while a verification job is running - we don't want to have multiple jobs running in parallel
while proxmox-backup-debug api get /nodes/pbs/tasks \
  --running true \
  --output-format json 2>/dev/null \
  | jq -e '[.[] | select(.worker_type == "verificationjob" or .worker_type == "verify_snapshot")] | length > 0' > /dev/null 2>&1; do
    echo "Verification job still running, waiting 60s..."
    sleep 60
done

# get list of failed jobs
FAILED=$(proxmox-backup-debug api get /admin/datastore/$DATASTORE/snapshots \
  --output-format json 2>/dev/null \
  | jq -r '.[] | select(.verification.state == "failed") | "\(.["backup-type"]) \(.["backup-id"]) \(.["backup-time"])"')

# Bail out if nothing has failed
if [ -z "$FAILED" ]; then
    echo "No failed backups found."
    exit 0
fi

# Re-Verify any failed jobs
while IFS=' ' read -r BTYPE BID BTIME; do
    LABEL="$BTYPE/$BID/$(date -u -d "@$BTIME" '+%Y-%m-%dT%H:%M:%SZ')"
    echo "Re-verifying $LABEL..."
    if [ "$DRY_RUN" = false ]; then
        proxmox-backup-debug api create /admin/datastore/$DATASTORE/verify \
          --backup-type "$BTYPE" \
          --backup-id "$BID" \
          --backup-time "$BTIME" \
          --ignore-verified false
    fi
done <<< "$FAILED"

As I’m running my verifications on schedule on Sunday evenings, I’ve scheduled the re-verification for early Monday mornings:

0 1 * * 1 /path/to/reverify-backups.sh > /path/to/reverify-backups.log 2>&1