6. Tasks

The Tasks section of the administrative GUI is used to configure repetitive tasks:

  • Cloud Sync schedules data synchronization to cloud providers
  • Cron Jobs schedules a command or script to automatically execute at a specified time
  • Init/Shutdown Scripts configures a command or script to automatically execute during system startup or shutdown
  • Rsync Tasks schedules data synchronization to another system
  • S.M.A.R.T. Tests schedules disk tests

Each of these tasks is described in more detail in this section.

Note

By default, Scrubs are run once a month by an automatically-created task. S.M.A.R.T. Tests and Periodic Snapshot Tasks must be set up manually.

6.1. Cloud Sync

Files or directories can be synchronized to remote cloud storage providers with the Cloud Sync feature.

Warning

This Cloud Sync task might go to a third party commercial vendor not directly affiliated with iXsystems. Please investigate and fully understand that vendor’s pricing policies and services before creating any Cloud Sync task. iXsystems is not responsible for any charges incurred from the use of third party vendors with the Cloud Sync feature.

Cloud Credentials must be pre-defined before a cloud sync is created. One set of credentials can be used for more than one cloud sync. For example, a single set of credentials for Amazon S3 can be used for separate cloud syncs that push different sets of files or directories.

A cloud storage area must also exist. With Amazon S3, these are called buckets. The bucket must be created before a sync task can be created.

After the credentials and receiving bucket have been configured, Tasks ➞ Cloud Syncs ➞ Add Cloud Sync is used to define the schedule for running a cloud sync task. An example is shown in Figure 6.1.1.

_images/tasks-cloudsync.png

Fig. 6.1.1 Adding a Cloud Sync

Table 6.1.1 shows the configuration options for Cloud Syncs.

Table 6.1.1 Cloud Sync Options
Setting Value Type Description
Description string Enter a descriptive name for this Cloud Sync.
Direction string Push sends data to cloud storage. Pull receives data from cloud storage.
Provider drop-down menu Choose the cloud storage provider credentials from the list of entered Cloud Credentials. The UI tests the credential and displays an error if a connection cannot be made.
Amazon S3 Buckets drop-down menu Only appears when a valid S3 credential is the Provider. Select the pre-defined S3 bucket to use.
Folder string Only appears when an S3 credential is the Provider. Optionally enter the name of the folder within the selected bucket.
Server Side Encryption drop-down menu Only appears when an S3 credential is the Provider. Choices are None (no encryption) or AES-256 (encrypted).
Path browse button Select the directories or files to be sent to the cloud for Push syncs, or the destination to be written as the destinations for Pull syncs. Be cautious about the destination of Pull jobs to avoid overwriting existing files.
Transfer Mode drop-down menu

Sync (default) makes files on destination system identical to those on the source. Files removed from the source are also removed from the destination, similar to rsync --delete.

Copy copies files from the source to the destination and skips files that are identical, similar to rsync.

Move copies files from the source to the destination and deletes the source files after the copy, similar to mv.

Remote encryption checkbox Set to encrypt files before transfer and store the encrypted files on the remote system. rclone Crypt is used.
Filename encryption checkbox Only appears when Remote encryption is enabled. Set to encrypt the shared file names.
Encryption password string Only appears when Remote encryption is enabled. Enter the password for encrypting and decrypting remote data. Warning: Always save and back up this password. Losing the encryption password can result in data loss.
Encryption salt string Only appears when Remote encryption is enabled. Enter a long string of random characters for use as salt for the encryption password. Warning: Save and back up the encryption salt value. Losing the salt value can result in data loss.
Minute slider or minute selections Select Every N minutes and use the slider to choose a value, or select Each selected minute and choose specific minutes to run the task.
Hour slider or hour selections Select Every N hours and use the slider to choose a value, or select Each selected hour and choose specific hours to run the task.
Day of month slider or day of month selections Select Every N days of month and use the slider to choose a value, or select Each selected day of month and choose specific days to run the task.
Month checkboxes Months when the task runs.
Day of week checkboxes Days of the week to run the task.
Enabled checkbox Unset to temporarily disable this Cloud Sync.

The time selected is when the Cloud Sync task is allowed to begin. The cloud sync runs until finished, even after the time selected.

Note

Files that have completed the sync process are not deleted from the destination if the rclone sync is interrupted or encounters an error. This includes a common error when the Dropbox copyright detector identifies a copyrighted file.

Figure 6.1.2 shows a cloud sync called backup-acctg that “pushes” a file to cloud storage. The last run finished with a status of SUCCESS.

_images/tasks-cloudsync-status.png

Fig. 6.1.2 Cloud Sync Status

To modify an existing cloud sync, click the entry to access the Edit, and Delete, and Run Now buttons.

Click the Status column entry for a cloud sync that is RUNNING, FAILED, or a SUCCESS. This opens the log in a pop-up window to read any error messages or other details.

6.1.1. Cloud Sync Example

This example shows a Push cloud sync which writes an accounting department backup file from the FreeNAS® system to Amazon S3 storage.

Before the new cloud sync was added, a bucket called cloudsync-bucket was created with the Amazon S3 web console for storing data from the FreeNAS® system.

System ➞ Cloud Credentials ➞ Add Cloud Credential is used to enter the credentials for storage on an Amazon AWS account. The credential is given the name S3 Storage, as shown in Figure 6.1.3:

_images/cloudsync-example-cred.png

Fig. 6.1.3 Example: Adding Cloud Credentials

The local data to be sent to the cloud is in a dataset called acctg-backups. The cloud sync task is created by going to Tasks ➞ Cloud Sync ➞ Add Cloud Sync. The Description is set to backup-acctg to describe the job. This data is being sent to cloud storage, so this is a Push. The Provider comes from the cloud credentials defined in the previous step, and the destination bucket cloudsync-bucket is selected.

The Path to the data file is selected.

The remaining fields are for setting a schedule. The default is to send the data to cloud storage once an hour, every day. The options provide great versatility in configuring when a cloud sync runs, anywhere from once a minute to once a year.

The Enabled option is set by default, so this cloud sync will run at the next scheduled time.

The completed dialog is shown in Figure 6.1.4:

_images/cloudsync-example-cropped.png

Fig. 6.1.4 Example: Adding a Cloud Sync

6.2. Cron Jobs

cron(8) is a daemon that runs a command or script on a regular schedule as a specified user.

Figure 6.2.1 shows the screen that opens after clicking Tasks ➞ Cron Jobs ➞ Add Cron Job.

_images/tasks-cron.png

Fig. 6.2.1 Creating a Cron Job

Table 6.2.1 lists the configurable options for a cron job.

Table 6.2.1 Cron Job Options
Setting Value Description
User drop-down menu Choose a user account to run the command or script. The user must have permissions to run the command.
Command string Enter the full path to the command or script to be run. Test a script at the command line first to make sure it works as expected.
Short description string Optional. Describe the new cron job.
Minute slider or minute selections With the slider, the cron job occurs every N minutes. With minute selections, the cron job occurs at the highlighted minutes
Hour slider or hour selections With the slider, the cron job occurs every N hours. With hour selections, the cron job occurs at the highlighted hours.
Day of month slider or month selections With the slider, the cron job occurs every N days. With day selections, the cron job occurs on the highlighted days each month.
Month checkboxes Cron job occurs on the selected months.
Day of week checkboxes Cron job occurs on the selected days.
Redirect Stdout checkbox Disables emailing standard output to the root user account.
Redirect Stderr checkbox Disables emailing errors to the root user account.
Enabled checkbox Deselect disable the cron job without deleting it.

Cron jobs are shown in View Cron Jobs. Highlight a cron job entry to display buttons to Edit, Delete, or Run Now.

Note

% symbols are automatically escaped and should not be prefixed with backslashes. For example, use date '+%Y-%m-%d' in a cron job to generate a filename based on the date.

6.3. Init/Shutdown Scripts

FreeNAS® provides the ability to schedule commands or scripts to run at system startup or shutdown.

Go to Tasks ➞ Init/Shutdown Scripts and click Add Init/Shutdown Script.

_images/tasks-initshutdown.png

Fig. 6.3.1 Add an Init/Shutdown Command or Script

Table 6.3.1 Init/Shutdown Command or Script Options
Setting Value Description
Type drop-down menu Select Command for an executable or Script for an executable script.
Command or Script string If Command is selected, enter the command with any options. When Script is selected, click Browse to select the script from an existing pool.
When drop-down menu

Select when the Command or Script runs:

  • Pre Init: early in the boot process, after mounting filesystems and starting networking
  • Post Init: at the end of the boot process, before FreeNAS® services start
  • Shutdown: during the system power off process.
Enabled checkbox Enable this task. Unset to disable the task without deleting it.

Scheduled commands must be in the default path. The full path to the command can also be included in the entry. The path can be tested with which {commandname} in the Shell. When available, the path to the command is shown:

[root@freenas ~]# which ls
/bin/ls

When scheduling a script, test the script first to verify it is executable and achieves the desired results.

Note

Init/shutdown scripts are run with sh.

Init/Shutdown tasks are shown in Tasks ➞ Init/Shutdown Scripts. Click a task to Edit or Delete that task.

6.4. Rsync Tasks

Rsync is a utility that copies specified data from one system to another over a network. Once the initial data is copied, rsync reduces the amount of data sent over the network by sending only the differences between the source and destination files. Rsync is used for backups, mirroring data on multiple systems, or for copying files between systems.

Rsync is most effective when only a relatively small amount of the data has changed. There are also some limitations when using Rsync with Windows files. For large amounts of data, data that has many changes from the previous copy, or Windows files, Replication Tasks are often the faster and better solution.

Rsync is single-threaded and gains little from multiple processor cores. To see whether rsync is currently running, use pgrep rsync from the Shell.

Both ends of an rsync connection must be configured:

  • the rsync server: this system pulls (receives) the data. This system is referred to as PULL in the configuration examples.
  • the rsync client: this system pushes (sends) the data. This system is referred to as PUSH in the configuration examples.

FreeNAS® can be configured as either an rsync client or an rsync server. The opposite end of the connection can be another FreeNAS® system or any other system running rsync. In FreeNAS® terminology, an rsync task defines which data is synchronized between the two systems. To synchronize data between two FreeNAS® systems, create the rsync task on the rsync client.

FreeNAS® supports two modes of rsync operation:

  • rsync module mode: exports a directory tree, and the configured settings of the tree as a symbolic name over an unencrypted connection. This mode requires that at least one module be defined on the rsync server. It can be defined in the FreeNAS® GUI under Services ➞ Rsync ➞ Rsync Modules. In other operating systems, the module is defined in rsyncd.conf(5).
  • rsync over SSH: synchronizes over an encrypted connection. Requires the configuration of SSH user and host public keys.

This section summarizes the options when creating an rsync task. It then provides a configuration example between two FreeNAS® systems for each mode of rsync operation.

Note

If there is a firewall between the two systems or if the other system has a built-in firewall, make sure that TCP port 873 is allowed.

Figure 6.4.1 shows the screen that appears after selecting Tasks ➞ Rsync Tasks ➞ Add Rsync Task. Table 6.4.1 summarizes the options that can be configured when creating an rsync task.

_images/tasks-rsync-tasks-add.png

Fig. 6.4.1 Adding an Rsync Task

Table 6.4.1 Rsync Configuration Options
Setting Value Description
Path browse button Browse to the path to be copied. Path lengths cannot be greater than 255 characters.
User drop-down menu The chosen user must have write permissions for the specified remote directory. The user name cannot contain spaces or exceed 17 characters.
Remote Host string Enter the IP address or hostname of the remote system that will store the copy. Use the format username@remote_host if the username differs on the remote host.
Remote SSH Port integer Only available in Rsync over SSH mode. Allows specifying an SSH port other than the default of 22.
Rsync mode drop-down menu Choices are Rsync module or Rsync over SSH.
Remote Module Name string At least one module must be defined in rsyncd.conf(5) of the rsync server or in the Rsync Modules of another system.
Remote Path string Only appears when using Rsync over SSH mode. Enter the existing path on the remote host to sync with. Example: /mnt/volume. Note that maximum path length is 255 characters.
Validate Remote Path checkbox Verifies the existence of the Remote Path.
Direction drop-down menu Direct the flow of the data to the remote host. Choices are Push or Pull. Default is to Push to a remote host.
Short Description string Enter an optional description of the new rsync task.
Minute slider or minute selections When the slider is used the sync occurs every N minutes. Use Each selected minute for the sync to occur at the highlighted minutes.
Hour slider or hour selections When the slider is used the sync occurs every N hours. Use Each selected hour for the sync to occur at the highlighted hours.
Day of month slider or day selections When the slider is used the sync occurs every N days. Use Each selected day of the month for the sync to occur on the highlighted days.
Month checkboxes Define which months to run the task.
Day of week checkboxes Define which days of the week to run the task.
Recursive checkbox Set to include all subdirectories of the specified volume during the rsync task.
Times checkbox Set to preserve the modification times of the files.
Compress checkbox Set to reduce the size of data to transmit. Recommended for slower connections.
Archive checkbox Equivalent to -rlptgoD. This will run the task as recursive, copy symlinks as symlinks, preserve permissions, preserve modification times, preserve group, preserve owner (root only), and preserve device and special files.
Delete checkbox Set to delete files in the destination directory that do not exist in the sending directory.
Quiet checkbox Set to suppresses informational messages from the remote server.
Preserve permissions checkbox Set to preserve original file permissions. Useful if User is set to root.
Preserve extended attributes checkbox Both systems must support extended attributes..
Delay Updates checkbox Set to save the temporary file from each updated file to a holding directory. At the end of the transfer, all transferred files are renamed into place and temporary files deleted.
Extra options string Add any other rsync(1) options. The * character must be escaped with a backslash (\*.txt) or used inside single quotes ('*.txt').
Enabled checkbox Unset to disable the rsync task without deleting it.

If the rysnc server requires password authentication, enter --password-file=/PATHTO/FILENAME in the Extra options option, replacing /PATHTO/FILENAME with the appropriate path to the file containing the password.

Created rsync tasks will be listed in View Rsync Tasks. Highlight the entry for an rsync task to display buttons for Edit, Delete, or Run Now.

6.4.1. Rsync Module Mode

This configuration example configures rsync module mode between these two FreeNAS® systems:

  • 192.168.2.2 has existing data in /mnt/local/images. It will be the rsync client, meaning that an rsync task needs to be defined. It will be referred to as PUSH.
  • 192.168.2.6 has an existing volume named /mnt/remote. It will be the rsync server, meaning that it will receive the contents of /mnt/local/images. An rsync module needs to be defined on this system and the rsyncd service needs to be started. It will be referred to as PULL.

On PUSH, an rsync task is defined in Tasks ➞ Rsync Tasks ➞ Add Rsync Task. In this example:

  • the Path points to /usr/local/images, the directory to be copied
  • the User is set to root so it has permission to write anywhere
  • the Remote Host points to 192.168.2.6, the IP address of the rsync server
  • the Rsync mode is Rsync module
  • the Remote Module Name is backups; this will need to be defined on the rsync server
  • the Direction is Push
  • the rsync is scheduled to occur every 15 minutes
  • the Preserve permissions option is enabled so that the original permissions are not overwritten by the root user

On PULL, an rsync module is defined in Services ➞ Rsync Modules ➞ Add Rsync Module. In this example:

  • the Module Name is backups; this needs to match the setting on the rsync client
  • the Path is /mnt/remote; a directory called images will be created to hold the contents of /usr/local/images
  • the User is set to root so it has permission to write anywhere
  • Hosts allow is set to 192.168.2.2, the IP address of the rsync client

Descriptions of the configurable options can be found in Rsync Modules.

To finish the configuration, start the rsync service on PULL in Services ➞ Control Services. If the rsync is successful, the contents of /mnt/local/images/ will be mirrored to /mnt/remote/images/.

6.4.2. Rsync over SSH Mode

SSH replication mode does not require the creation of an rsync module or for the rsync service to be running on the rsync server. It does require SSH to be configured before creating the rsync task:

  • a public/private key pair for the rsync user account (typically root) must be generated on PUSH and the public key copied to the same user account on PULL
  • to mitigate the risk of man-in-the-middle attacks, the public host key of PULL must be copied to PUSH
  • the SSH service must be running on PULL

To create the public/private key pair for the rsync user account, open Shell on PUSH and run ssh-keygen. This example generates an RSA type public/private key pair for the root user. When creating the key pair, do not enter the passphrase as the key is meant to be used for an automated task.

ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
f5:b0:06:d1:33:e4:95:cf:04:aa:bb:6e:a4:b7:2b:df root@freenas.local
The key's randomart image is:
+--[ RSA 2048]----+
|        .o. oo   |
|         o+o. .  |
|       . =o +    |
|        + +   o  |
|       S o .     |
|       .o        |
|      o.         |
|    o oo         |
|     **oE        |
|-----------------|
|                 |
|-----------------|

FreeNAS® supports RSA keys for SSH. When creating the key, use -t rsa to specify this type of key. Refer to Key-based Authentication for more information.

Note

If a different user account is used for the rsync task, use the su command after mounting the filesystem but before generating the key. For example, if the rsync task is configured to use the user1 user account, use this command to become that user:

su user1

Next, view and copy the contents of the generated public key:

more .ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1lBEXRgw1W8y8k+lXPlVR3xsmVSjtsoyIzV/PlQPo
SrWotUQzqILq0SmUpViAAv4Ik3T8NtxXyohKmFNbBczU6tEsVGHo/2BLjvKiSHRPHc/1DX9hofcFti4h
dcD7Y5mvU3MAEeDClt02/xoi5xS/RLxgP0R5dNrakw958Yn001sJS9VMf528fknUmasti00qmDDcp/kO
xT+S6DFNDBy6IYQN4heqmhTPRXqPhXqcD1G+rWr/nZK4H8Ckzy+l9RaEXMRuTyQgqJB/rsRcmJX5fApd
DmNfwrRSxLjDvUzfywnjFHlKk/+TQIT1gg1QQaj21PJD9pnDVF0AiJrWyWnR root@freenas.local

Go to PULL and paste (or append) the copied key into the SSH Public Key field of Account ➞ Users ➞ View Users ➞ root ➞ Modify User, or the username of the specified rsync user account. The paste for the above example is shown in Figure 6.4.2. When pasting the key, ensure that it is pasted as one long line and, if necessary, remove any extra spaces representing line breaks.

_images/rsync2.png

Fig. 6.4.2 Pasting the User SSH Public Key

While on PULL, verify that the SSH service is running in Services ➞ Control Services and start it if it is not.

Next, copy the host key of PULL using Shell on PUSH. The command below copies the RSA host key of the PULL server used in our previous example. Be sure to include the double bracket >> to prevent overwriting any existing entries in the known_hosts file:

ssh-keyscan -t rsa 192.168.2.6 >> /root/.ssh/known_hosts

Note

If PUSH is a Linux system, use this command to copy the RSA key to the Linux system:

cat ~/.ssh/id_rsa.pub | ssh user@192.168.2.6 'cat >> .ssh/authorized_keys'

The rsync task can now be created on PUSH. To configure rsync SSH mode using the systems in the previous example, use this configuration:

  • the Path points to /mnt/local/images, the directory to be copied
  • the User is set to root so it has permission to write anywhere; the public key for this user must be generated on PUSH and copied to PULL
  • the Remote Host points to 192.168.2.6, the IP address of the rsync server
  • the Rsync Mode is Rsync over SSH
  • the rsync is scheduled to occur every 15 minutes
  • the Preserve Permissions option is enabled so that the original permissions are not overwritten by the root user

Save the rsync task and the rsync will automatically occur according to the schedule. In this example, the contents of /mnt/local/images/ will automatically appear in /mnt/remote/images/ after 15 minutes. If the content does not appear, use Shell on PULL to read /var/log/messages. If the message indicates a n (newline character) in the key, remove the space in the pasted key–it will be after the character that appears just before the n in the error message.

6.5. S.M.A.R.T. Tests

S.M.A.R.T. (Self-Monitoring, Analysis and Reporting Technology) is a monitoring system for computer hard disk drives to detect and report on various indicators of reliability. Replace the drive when a failure is anticipated by S.M.A.R.T. Most modern ATA, IDE, and SCSI-3 hard drives support S.M.A.R.T. – refer to the drive documentation for confirmation.

Figure 6.5.1 shows the configuration screen that appears after selecting Tasks ➞ S.M.A.R.T. Tests ➞ Add S.M.A.R.T. Test. Tests are listed under View S.M.A.R.T. Tests. After creating tests, check the configuration in Services ➞ S.M.A.R.T., then click the slider to ON for the S.M.A.R.T. service in Services ➞ Control Services. The S.M.A.R.T. service will not start if there are no volumes.

Note

To prevent problems, do not enable the S.M.A.R.T. service if the disks are controlled by a RAID controller. It is the job of the controller to monitor S.M.A.R.T. and mark drives as Predictive Failure when they trip.

_images/tasks-smart.png

Fig. 6.5.1 Adding a S.M.A.R.T. Test

Table 6.5.1 summarizes the configurable options when creating a S.M.A.R.T. test.

Table 6.5.1 S.M.A.R.T. Test Options
Setting Value Description
Disks list Select the disks to monitor.
Type drop-down menu Choose the test type. See smartctl(8) for descriptions of each type of test. Some test types will degrade performance or take disks offline. Avoid scheduling S.M.A.R.T. tests simultaneously with scrub or resilver operations.
Short description string Optional. Enter a short description of this test.
Hour slider or hour selections When the slider is used the sync occurs every N hours. Use Each selected hour for the test to occur at the highlighted hours.
Day of month slider or day selections When the slider is used the sync occurs every N days. Use Each selected day of the month for the sync to occur on the highlighted days.
Month checkboxes Select which months to run the test.
Day of week checkboxes Select which days of the week to run the test.

Note

Scrub tasks are run if and only if the threshhold is met or exceeded and the task is scheduled to run on the date marked.

An example configuration is to schedule a Short Self-Test once a week and a Long Self-Test once a month. These tests do not have a performance impact, as the disks prioritize normal I/O over the tests. If a disk fails a test, even if the overall status is Passed, start to think about replacing that disk.

Warning

Some S.M.A.R.T. tests cause heavy disk activity and can drastically reduce disk performance. Do not schedule S.M.A.R.T. tests to run at the same time as scrub or resilver operations or during other periods of intense disk activity.

Which tests will run and when can be verified by typing smartd -q showtests within Shell.

The results of a test can be checked from Shell by specifying the name of the drive. For example, to see the results for disk ada0, type:

smartctl -l selftest /dev/ada0

If an email address is entered in the Email to report field of Services ➞ S.M.A.R.T., the system will send an email to that address when a test fails. Logging information for S.M.A.R.T. tests can be found in /var/log/daemon.log.