Duplicating drive without losing data on original?

[Something]

So i'm in a bit of an emergency situation. My Time Machine backups for my machine (not others) show corruption, and attempts to view the backups are unsuccessful. I tried my best to repair it but that cause seems fruitless. In the meantime, I have only been running one drive, a WD Red, so for the repair process, I got around to something I should've done so long ago, picking up a second drive (also WD Red, same size).

I tried copying the Time Machine backups to the new drive using cp with the right options to carry over the file permissions, but the ownership isn't correct, so i'm not sure about their viability for any sort of recovery efforts and deleted them. Didn't double check the file permissions, but those weren't correct either.

I talked with a more experienced friend who recommended that I use zpool (sent the Oracle doc) but i'm not sure if that'll wipe the original drive. Googling for info wasn't really fruitful, import disk is supposed to work but i'm not sure I even have a zpool currently, let alone if it won't wipe the drive. I read through the slideshow on zpool that cyberjock posted, but it didn't confirm anything for me.
Help? Please and thank you all!

---

Also, I learned my lesson, getting a normal mobo and consumer CPU was a mistake. I'll see about selling those off + the RAM and picking up a sufficient replacement. I picked my current mobo/CPU since they were Haswell/efficient, cheap, plenty of SATA 6 (native) and mATX. Regretting it. Any hardware recommendations? I would like to have a Haswell processor, 16GBs ECC, 6+ (native) SATA 6 (or should I use a PCIe SATA controller?) and (optionally/ideally) a motherboard that's mITX/mATX.

I did some reading up, and the performance hit of encryption is minimal (since it'll utilize AES-NI instructions), do you believe it's worth encrypting backups/drives? Especially when using WD Red drives?

 

[danb35]

I hate to be the rules Nazi (see forum rules), but you've literally provided no information that would help us help you. The best I can figure out is that you have more than one Mac backing up to your FreeNAS server using Time Machine, and Time Machine on your Mac says that your backup is corrupted. If that's the case, your problem likely has nothing to do with FreeNAS--Time Machine is known to bork backups, and need them to be erased and replaced. I'm not sure why you're talking about now wanting to copy those backup files to another disk.

There are a bunch of tools you could use to copy your data to another disk. cp is one; rsync is another; ZFS replication is a third. If you're trying to copy the data to another disk so you can play with it and attempt to recover the backup, any of them will work fine, and ownership and permissions really aren't an issue. Though ZFS replication will give the most exact duplicate of your data, I don't think I'd recommend it in this case, as the CLI-fu is a little more involved, and it just isn't necessary. I'd probably use rsync--'rsync -Aarv /path/to/source/ /path/to/destination/'.

For hardware recommendations, start here.

I wouldn't recommend using encryption unless you know you absolutely need it. There are pitfalls for unwary players, and the default failure mode is "the data is gone forever".

 

[Something]

Thanks dan!

I hate to be the rules Nazi (see forum rules), but you've literally provided no information that would help us help you. The best I can figure out is that you have more than one Mac backing up to your FreeNAS server using Time Machine, and Time Machine on your Mac says that your backup is corrupted. If that's the case, your problem likely has nothing to do with FreeNAS--Time Machine is known to bork backups, and need them to be erased and replaced. I'm not sure why you're talking about now wanting to copy those backup files to another disk.

I thought the title would provide necessary info, I really just need the most 'exact' possible copy of either the entire drive or the backup itself. Time Machine failing is just context for why it's needed.

There are a bunch of tools you could use to copy your data to another disk. cp is one; rsync is another; ZFS replication is a third. If you're trying to copy the data to another disk so you can play with it and attempt to recover the backup, any of them will work fine, and ownership and permissions really aren't an issue. Though ZFS replication will give the most exact duplicate of your data, I don't think I'd recommend it in this case, as the CLI-fu is a little more involved, and it just isn't necessary. I'd probably use rsync--'rsync -Aarv /path/to/source/ /path/to/destination/'.

How would ZFS replication work? I'm willing to put the work in.

For hardware recommendations, start here.

Neat, i'm thinking about this board from Supermicro, any thoughts?

I wouldn't recommend using encryption unless you know you absolutely need it. There are pitfalls for unwary players, and the default failure mode is "the data is gone forever".

Okie dokie, cheers.

 

[depasseg]

Without any HW details, it's hard to diagnose, but if you have a single drive, and you are worried about that drive failing, the simplest way would be to add the new drive as a mirror. And as for ZFS replication, while you can do it easily via the CLI, you can also follow the directions and use the GUI.

Code:

zfs snapshot yourpoolname/datasetname@snapshotID
zfs send yourpoolname/datasetname@snapshotID | zfs recv -d  aSECONDpoolname/datasetname

You could also create an exact replica and dump it to a file. and then you restore it later.
http://cuddletech.com/blog/pivot/entry.php?id=984

 

[Something]

Without any HW details, it's hard to diagnose, but if you have a single drive, and you are worried about that drive failing, the simplest way would be to add the new drive as a mirror. And as for ZFS replication, while you can do it easily via the CLI, you can also follow the directions and use the GUI.

i3 4370
16GB DDR3 1866 not ECC
WD Red 4TB
MSI Z87m gaming

I'm using the latest stable build of FreeNAS.

I'm not sure about the drive failing, but I do know the Time Machine backups have corruption so i'd like to back that up.

Code:

zfs snapshot yourpoolname/datasetname@snapshotID
zfs send yourpoolname/datasetname@snapshotID | zfs recv -d  aSECONDpoolname/datasetname

You could also create an exact replica and dump it to a file. and then you restore it later.
http://cuddletech.com/blog/pivot/entry.php?id=984

Will zfs create an exact replica of my first drive onto the second? Because that'd be ideal, especially since I don't know if the drive is failing. My understanding is that it will, but I want to be extra certain.

Will the GUI then immediately recognize it as such? How will this affect my CIFS/AFP shares?

 

[depasseg]

Yep. Exact.

You should run a scrub on your dataset to find and attempt to fix any corruption.

 

[SweetAndLow]

how about you post the output of zpool status and post it using the "[c0de]" tags

 

[Something]

Before or after I use zfs?

how about you post the output of zpool status and post it using the "[c0de]" tags

Can do will do once I get home.

 

[Something]

As requested

Code:

status

  pool: freenas-boot

state: ONLINE

  scan: scrub repaired 0 in 0h2m with 0 errors on Mon Jun 22 03:47:09 2015

config:



NAME        STATE     READ WRITE CKSUM

freenas-boot  ONLINE       0     0     0

  da0p2     ONLINE       0     0     0



errors: No known data errors



  pool: wdred1

state: ONLINE

  scan: scrub repaired 0 in 7h2m with 0 errors on Sun Jul 12 07:02:59 2015

config:



NAME                                          STATE     READ WRITE CKSUM

wdred1                                        ONLINE       0     0     0

  gptid/843fd539-a8f2-11e4-8af0-448a5b8b7959  ONLINE       0     0     0



errors: No known data errors

What does scrubbing do exactly?

Quick question, how do you check what IPs have attempted to connect to you? I have had SSH on this whole time by accent > <. Found it.

 

[Robert Trevellyan]

I'm not sure about the drive failing, but I do know the Time Machine backups have corruption so i'd like to back that up.

Run some SMART tests to find out if the hard drive is failing. But most likely it isn't, like @danb35 said, it's probably the usual issue with Time Machine over AFP failing periodically. If so, I see no point in duplicating the corrupted backup, just throw it away and start over with something that works properly (hint: not Time Machine over AFP). My preference for backup of Macs to FreeNAS is Arq Backup (no affiliation).

 

[Ericloewe]

What does scrubbing do exactly?

https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

 

[Robert Trevellyan]

And if it turns out the drive is failing and you have a real need to duplicate it, use ddrescue.

 

[Something]

Code:

smartctl -d ata -a /dev/ada0
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p16 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

Read Device Identity failed: Inappropriate ioctl for device

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

Admittedly i'm not that great at this lol.

Smart is enabled according to the GUI.

https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Cheers but is there a more technical explanation for how it works?

And if it turns out the drive is failing and you have a real need to duplicate it, use ddrescue.

Thanks!

 

[SweetAndLow]

smartctl -a /dev/ada0 is the command you want.
And what do you mean by a more technical explanation? you can look at the code if you want. That is as technical as it gets.

 

[Something]

smartctl -a /dev/ada0 is the command you want.
And what do you mean by a more technical explanation? you can look at the code if you want. That is as technical as it gets.

Code:

=== START OF INFORMATION SECTION ===

Model Family:     Western Digital Red

Device Model:     WDC WD40EFRX-68WT0N0

Serial Number:    WD-WCC4E7ZRR4F5

LU WWN Device Id: 5 0014ee 2609161b8

Firmware Version: 82.00A82

User Capacity:    4,000,787,030,016 bytes [4.00 TB]

Sector Sizes:     512 bytes logical, 4096 bytes physical

Rotation Rate:    5400 rpm

Device is:        In smartctl database [for details use: -P show]

ATA Version is:   ACS-2 (minor revision not indicated)

SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)

Local Time is:    Fri Jul 17 22:07:42 2015 CDT

SMART support is: Available - device has SMART capability.

SMART support is: Enabled



=== START OF READ SMART DATA SECTION ===

SMART overall-health self-assessment test result: PASSED



General SMART Values:

Offline data collection status:  (0x00)Offline data collection activity

was never started.

Auto Offline Data Collection: Disabled.

Self-test execution status:      (   0)The previous self-test routine completed

without error or no self-test has ever

been run.

Total time to complete Offline

data collection: (53760) seconds.

Offline data collection

capabilities:  (0x7b) SMART execute Offline immediate.

Auto Offline data collection on/off support.

Suspend Offline collection upon new

command.

Offline surface scan supported.

Self-test supported.

Conveyance Self-test supported.

Selective Self-test supported.

SMART capabilities:            (0x0003)Saves SMART data before entering

power-saving mode.

Supports SMART auto save timer.

Error logging capability:        (0x01)Error logging supported.

General Purpose Logging supported.

Short self-test routine

recommended polling time:  (   2) minutes.

Extended self-test routine

recommended polling time:  ( 537) minutes.

Conveyance self-test routine

recommended polling time:  (   5) minutes.

SCT capabilities:        (0x703d)SCT Status supported.

SCT Error Recovery Control supported.

SCT Feature Control supported.

SCT Data Table supported.



SMART Attributes Data Structure revision number: 16

Vendor Specific SMART Attributes with Thresholds:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE

  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0

  3 Spin_Up_Time            0x0027   183   176   021    Pre-fail  Always       -       7825

  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       26

  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0

  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0

  9 Power_On_Hours          0x0032   094   094   000    Old_age   Always       -       4462

10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0

11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0

12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       26

192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8

193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       83

194 Temperature_Celsius     0x0022   119   117   000    Old_age   Always       -       33

196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0

197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0

198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0

199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0



SMART Error Log Version: 1

No Errors Logged



SMART Self-test log structure revision number 1

No self-tests have been logged.  [To run self-tests, use: smartctl -t]



SMART Selective self-test log data structure revision number 1

SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS

    1        0        0  Not_testing

    2        0        0  Not_testing

    3        0        0  Not_testing

    4        0        0  Not_testing

    5        0        0  Not_testing

Selective self-test flags (0x0):

  After scanning selected spans, do NOT read-scan remainder of disk.

If Selective self-test is pending on power-up, resume after 0 minute delay.

I suppose I should go dig into the code if I want to understand it any better. I guess it'd have to be very dependent on a lot of the deeper mechanics of FreeNAS and ZFS.

Once again, thanks everyone for the help.

 

[SweetAndLow]

you should start by learning how to administer your system. You haven't run any smart tests on your drive. You need to read alot more before you can even start to care about the technical details. You can schedule smart tests in the webui and should read the freenas manual for how to do that.

 

[Something]

you should start by learning how to administer your system. You haven't run any smart tests on your drive.

There's more than meets the eye let's say.

Is SMART not by default setup to test regularly with FreeNAS?

You need to read alot more before you can even start to care about the technical details. You can schedule smart tests in the webui and should read the freenas manual for how to do that.

I'm better suited for the technical details, algorithms, and mechanisms in play than I am anything remotely related to system administration.

Any recommendations on what to read (other than the FreeNAS manual)?

--------

Went ahead and added my second disk as a striped volume, i'll try and send a snapshot to it. New zpool info. Scrubbing the main HD right now. After the scrub completes i'll go ahead and zfs snapshot it to the secondary drive.

Code:

zpool status

  pool: freenas-boot

 state: ONLINE

  scan: scrub repaired 0 in 0h2m with 0 errors on Mon Jun 22 03:47:09 2015

config:
NAME        STATE     READ WRITE CKSUM

freenas-boot  ONLINE       0     0     0

  da0p2     ONLINE       0     0     0

errors: No known data errors



  pool: wdred1

 state: ONLINE

  scan: scrub in progress since Fri Jul 17 22:57:04 2015

        26.7M scanned out of 2.05T at 2.96M/s, 201h16m to go

        0 repaired, 0.00% done

config:
NAME                                          STATE     READ WRITE CKSUM

wdred1                                        ONLINE       0     0     0

  gptid/843fd539-a8f2-11e4-8af0-448a5b8b7959  ONLINE       0     0     0

errors: No known data errors



  pool: wdred2

 state: ONLINE

  scan: none requested

config:
NAME                                          STATE     READ WRITE CKSUM

wdred2                                        ONLINE       0     0     0

  gptid/5d32b6f1-2d00-11e5-b9b1-448a5b8b7959  ONLINE       0     0     0

errors: No known data errors

Show

Slightly older zpool status.

Code:

zpool status
  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0h2m with 0 errors on Mon Jun 22 03:47:09 2015

config:
NAME        STATE     READ WRITE CKSUM

freenas-boot  ONLINE       0     0     0

  da0p2     ONLINE       0     0     0

errors: No known data errors



  pool: wdred1

state: ONLINE

  scan: scrub repaired 0 in 7h2m with 0 errors on Sun Jul 12 07:02:59 2015

config:
NAME                                          STATE     READ WRITE CKSUM

wdred1                                        ONLINE       0     0     0

  gptid/843fd539-a8f2-11e4-8af0-448a5b8b7959  ONLINE       0     0     0

errors: No known data errors



  pool: wdred2

state: ONLINE

  scan: none requested

config:
NAME                                          STATE     READ WRITE CKSUM

wdred2                                        ONLINE       0     0     0

  gptid/5d32b6f1-2d00-11e5-b9b1-448a5b8b7959  ONLINE       0     0     0

errors: No known data errors

 

[danb35]

With the latest versions of 9.3, SMART tests may be scheduled automatically with a new volume (can't say for sure, but I think I remember seeing that as a recent change), but that isn't the case with all 9.3 versions, and it definitely wasn't the case with anything earlier. However, you can (and should) schedule them yourself. I run a short test daily and a long test weekly, which may be a little more often than necessary in both cases, but is reasonable. It's discussed in the manual at http://doc.freenas.org/9.3/freenas_tasks.html#s-m-a-r-t-tests. Also make sure you have an email address set up so that and SMART errors will be reported.

With that said, there are no apparent problems with your pool. The last scrub found no errors, and the SMART data doesn't show any trouble with your drive. If you're concerned about the security of your data, add redundancy to your pool. You can add a second disk as a mirror to the first (though unfortunately not through the GUI; you'd need to do that at the CLI), but to end up with some sort of RAIDZ configuration you would need to back up the data, destroy the pool, and recreate it in the desired configuration.

As to the scrub, I don't know of more detailed documentation, but if it's out there it's probably in either the OpenZFS docs or the Oracle ZFS docs. The latter may not be 100% applicable, since ZFS forked a while back.

 

[Ericloewe]

Technical explanation of scrubs?

ZFS reads every block it has allocated.

That's basically it. ZFS always corrects data whenever it reads it, if necessary and possible.

 

[Robert Trevellyan]

Is SMART not by default setup to test regularly with FreeNAS?

By default FreeNAS checks SMART status every 30 minutes, but that's a completely separate process from running SMART tests, which seems to be a source of considerable confusion. To put is simply, drives update their SMART attributes continuously, but you or the system can initiate short or extended SMART tests too. You should, because that way you have a more comprehensive picture of drive health.

Your setup begs the question, why are you even using ZFS? With no redundancy in your pool, you're missing out on most of the benefits. Is it just that you like the FreeNAS GUI as a system admin tool, or is it the plugins, or what? I'm genuinely curious.