How to Automate TrueNAS Replication to a Normally Powered-Off Destination Server

Posted on by tkrn

Do you have a TrueNAS replication target that spends most of its time powered off to reduce energy consumption or to maintain an offline backup copy of your data?

If you want to automate weekly, monthly, or quarterly replication jobs without manually powering on the destination system, this solution is for you.

This automation workflow powers on the destination TrueNAS server on demand, waits for the system and services to become fully available, initiates the replication process, monitors replication activity, and then gracefully shuts down the destination system once replication has completed and all ZFS write activity has gone idle.

The result is a fully automated replication workflow that keeps your backup system offline when not in use while ensuring your data is replicated on a regular schedule with minimal administrative effort. Here’s the solution!

TrueNAS Replication Setup

Authentication Requirements

Two authentication points must be configured for the replication workflow to operate successfully:

  1. iDRAC SSH Access
    • Used by the automation script to remotely power on the destination server through iDRAC.
  2. TrueNAS SSH Access
    • Used to create the trigger file, initiate replication tasks, and perform post-replication validation and shutdown operations on the destination system.

iDRAC SSH Access

We need to create a dedicated user with reduced privilages to be to perform limited (power on) commands to the server through this new user. We specify Login and Execute Server Control Commands only to reduce security risks.

After our user is created we must upload our SSH public key. This was the tricky part to ensure that the key is in the proper format which the iDRAC6 requires. The public key must be in the following ---- BEGIN SSH2 PUBLIC KEY ---- …. ---- END SSH2 PUBLIC KEY ---- format. In addition, ensure the SSH service is enabled on iDRAC6.

SSH Key Requirements

The script is designed to use SSH key-based authentication. Before running the workflow, ensure that:

  • The .ssh directory and key files have the correct permissions.
  • A valid public/private SSH key pair has been generated.
  • The public key has been installed on the appropriate remote systems.

Please ensure the private keys are defined in the scripts variables.

# Target host iDRAC for remote power on. SSH key only.
IDRAC_HOST="r710-idrac.lab.lan"
IDRAC_USER="power-on-user"
IDRAC_SSH_KEY="/root/.ssh/r710-idrac.key"

# Target host TrueNAS Scale host credentials
TRUENAS_HOST="r710.lab.lan"
TRUENAS_USER="backup"
TRUENAS_SSH_KEY="/root/.ssh/backup.key"
TRUENAS_ONLINE_SECONDS=120

# Source host replication task
# midclt call replication.query
TASK_ID=3
TASK_NAME="replication to r710"

Bash Script to Power On the R710

Before a power-on command can be issued to a legacy Dell PowerEdge R510, R610, or R710 through iDRAC6 out-of-band management, specific SSH options must be configured to permit a successful connection.

Modern SSH clients disable many legacy ciphers and key exchange algorithms by default because they no longer meet current security standards. Since iDRAC6 relies on several of these deprecated algorithms, the required SSH options must be explicitly specified to enable compatibility and allow remote management operations, including power control commands.

# Needed for iDRAC 6 backwards compatiblity
IDRAC_SSH_OPTS=(
    -i "$IDRAC_SSH_KEY"
    -oIdentitiesOnly=yes
    -oCiphers=aes256-ctr
    -oMACs=hmac-sha1
    -oKexAlgorithms=diffie-hellman-group14-sha1
    -oHostKeyAlgorithms=rsa-sha2-512
    -oConnectTimeout=30
    -oConnectionAttempts=5
)

We issue the racadm power up command once we’ve successfully logged into the iDRAC using SSH.

echo "$("$DATE" '+%Y-%m-%d %H:%M:%S') - Sending power-on command..."

"$SSH" "${IDRAC_SSH_OPTS[@]}" \
    "${IDRAC_USER}@${IDRAC_HOST}" \
    "racadm serveraction powerup"

Wait for TrueNAS to Become Available

After the remote power-on command is issued, the system requires time to fully boot. The exact startup time varies depending on the hardware platform and system configuration.

To ensure the system is ready, we continuously ping the TrueNAS host and require it to remain reachable for a defined period. This additional wait time allows for complete system initialization, including the startup and stabilization of all TrueNAS SCALE services.

The server must remain online and responsive for at least 120 consecutive seconds before the temporary trigger file is created on the destination replication peer. This helps ensure the replication process begins only after the source system is fully operational.

echo "$("$DATE" '+%Y-%m-%d %H:%M:%S') - Waiting for server to remain online for ${TRUENAS_ONLINE_SECONDS} seconds..."

online_since=0

while true; do

    if "$PING" -c 1 -W 1 "$TRUENAS_HOST" >/dev/null 2>&1; then

        if [ "$online_since" -eq 0 ]; then
            online_since=$("$DATE" +%s)
            echo "$("$DATE" '+%Y-%m-%d %H:%M:%S') - Server is online; starting 2-minute timer"
        fi

        now=$("$DATE" +%s)
        elapsed=$((now - online_since))

        if [ "$elapsed" -ge "$TRUENAS_ONLINE_SECONDS" ]; then
            echo "$("$DATE" '+%Y-%m-%d %H:%M:%S') - Server has been online continuously for ${TRUENAS_ONLINE_SECONDS} seconds"
            break
        fi

    else

        if [ "$online_since" -ne 0 ]; then
            echo "$("$DATE" '+%Y-%m-%d %H:%M:%S') - Server went offline; resetting timer"
        fi

        online_since=0
    fi

    "$SLEEP" 1
done

Touch the Trigger File on TrueNAS

After the system has powered on, fully initialized, and been online for more than two minutes, the script uses SSH to connect to the destination replication host and creates a file at /tmp/replicate-boot. This file signals the power-off and replication-watch script on the destination host to begin running.

Because the source replication host creates this file, the system will never shut down automatically if the file is not present. This allows the system to start without the script and remain running indefinitely..

echo "$("$DATE" '+%Y-%m-%d %H:%M:%S') - Touching /tmp/replicate-boot"

"$SSH" "${TRUENAS_SSH_OPTS[@]}" \
    "${TRUENAS_USER}@${TRUENAS_HOST}" \
    "touch /tmp/replicate-boot"

if [ $? -eq 0 ]; then
    echo "$("$DATE" '+%Y-%m-%d %H:%M:%S') - Successfully touched /tmp/replicate-boot"
else
    echo "$("$DATE" '+%Y-%m-%d %H:%M:%S') - ERROR: Failed to touch /tmp/replicate-boot"
    exit 1
fi

Starting a TrueNAS Replication Task

The final stage is to kick of a pre-created replication job in TrueNAS Scale. We simply call the middleware to run the replication job. Remember, we must find the replication task id in the middleware by finding the corresponding id using midclt call replication.query on the source host.

echo "Starting TrueNAS replication task '$TASK_NAME' (ID # $TASK_ID)..."

midclt call replication.run "$TASK_ID"

if [ $? -eq 0 ]; then
    echo "Replication task started successfully."
    exit 0
else
    echo "Failed to start replication task."
    exit 1

Waiting for ZFS to Stop Writing & Destination Shutdown Script

Set the scripts variables specific to your destination host. Update your pool name for the script to check ZFS write activity and the defaults for the timers and interval are generally sufficent but you can modify here if you need.

POOL="tank"

LOAD_THRESHOLD="1.0"
IDLE_MINUTES=5
START_DELAY_MINUTES=5
INTERVAL=30

We look at the ZFS writes to the pool and compare that between samples to ensure there is no writing IO activity to the pool before it proceeds to shutdown:

    # Current cumulative write-operation count
    CURRENT_WRITES=$(
        $ZPOOL iostat -Hp "$POOL" |
        $AWK 'NR==1 {print $5}'
    )

In addition to looking at ZFS writes in-case of a resilver or another operation we look at the 15 minute CPU load average:

    # 15-minute load average
    LOAD=$($AWK '{print $3}' /proc/loadavg)

We make sure the two checks are met before shutting down:

    # Any ZFS writes reset the idle timer
    if [ "$WRITE_DELTA" -gt 0 ]; then
        IDLE_SECONDS=0
        echo "$($DATE): ZFS activity detected. Resetting idle timer."
        continue
    fi

    # Require low 15-minute load as well
    if ! $AWK "BEGIN {exit !($LOAD < $LOAD_THRESHOLD)}"; then
        IDLE_SECONDS=0
        echo "$($DATE): Load too high. Resetting idle timer."
        continue
    fi

Setting up the Cron jobs in TrueNAS

There are two cron jobs to coordinate all the process outlined above. The source and destinations have the scripts in /root as it requires elevated privilages.

Source Cron

  • Description: A name for the cron job
  • Command: /root/power-on-and-replicate-to-r710.sh >> /var/log/power-on-and-replicate-to-r710.log 2>&1
  • Run As User: root
  • Schedule: Using standard cron timing to define a time/regularity for this to power-on and replicate to occur

Note: Any script output will be logged to /var/log/power-on-and-replicate-to-r710.log which is a persistent log across reboots.

Destination Cron

  • Description: A name for the cron job
  • Command: /usr/bin/flock -n /var/run/replication-shutdown.lock /root/replication-shutdown.sh >> /var/log/replication-shutdown.log 2>&1
  • Run As User: root
  • Schedule: * * * * * (run every minute)

Note: Remember the script will look for the trigger file to control the logic if the server was remotely powered on by the source script to initiate replication. If the trigger file is not present, no action is taken the script simply exits with a clean exit code of 0.

Note: Any script output will be logged to /var/log/replication-shutdown.log which is a persistent log across reboots.

Note: /usr/bin/flock is a process locking mechanism in TrueNAS which probits multiple instances of the script running given the nature of running the script every minute.

Download on GitHub

Download the scripts here! https://github.com/tkrn/truenas-replication-automatic-power-destination-server

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.