Large file MD5

[GigaTech]

Hi Guys/Gals,
I have an issue with a newly built FreeNAS box, thought you guys may be able to help?

The box is built with all new parts (spec below) three days ago.

The problem.
Data (about 10Tb) copied from another NAS (working fine just at 80% capicity) to this FreeNAS without any issue, a day later email received showing CKSUM errors on two disks, zpool status showed 1 on the 6th disk, 2 on the 7th disk. This seems to have cleared and not come back after a reboot!

The real issue is that the MD5 of some of the copied files do not match source, mainly files over 60Gb but not all of them!!

After reading some posts and @cyberjock comments, performed the same copy (robocopy /MIR) using another workstation with the same results, large files mainly over 60Gb incorrect MD5.

Preformed MemTest86 v7.1 Free, all four passes good (I have images/report if required) run time 5hours 21mins with no errors

I have also setup SMART tests and Scrubs as per another of @cyberjock posts, but do not think they have run yet!

Thanks for reading, any more info required let me know/pointers appreciated

====================================
System Spec.
====================================
GigaByte GA-B86M-D2V
Intel i3-4150
2x 8Gb (non-ECC) DDR3 1600MHz

2x Startech 2 Port Pci-E Sata 6Gbps (p/n=PEXESAT322I, Chipset = Asmedia ASM1061)

8x WD RED 4Tb (WD40EFRX) in RAIDZ2
2x Kingston DataTraveler 50 32GB USB 3.0 (boot drives, mirror)

Hard disk connections

4x HDD connected to MoBo, 6Gbps
2x HDD connected to Startech 2 Port Pci-E card, 6Gbps Internal
2x HDD connected to Startech 2 Port Pci-E card, 6Gbps Internal​

FreeNAS 9.10.1-U2

====================================
zpool status
====================================

Code:

[root@GigaNAS4 ~]# zpool status  
  pool: GigaPool1  
 state: ONLINE  
  scan: none requested  
config:  
  NAME  STATE  READ WRITE CKSUM  
  GigaPool1  ONLINE  0  0  0  
  raidz2-0  ONLINE  0  0  0  
  gptid/11df0c86-953a-11e6-8d63-408d5c9485e4  ONLINE  0  0  0  
  gptid/13543c00-953a-11e6-8d63-408d5c9485e4  ONLINE  0  0  0  
  gptid/147b9166-953a-11e6-8d63-408d5c9485e4  ONLINE  0  0  0  
  gptid/15a25d3f-953a-11e6-8d63-408d5c9485e4  ONLINE  0  0  0  
  gptid/16c549c7-953a-11e6-8d63-408d5c9485e4  ONLINE  0  0  0  
  gptid/17e9e5bf-953a-11e6-8d63-408d5c9485e4  ONLINE  0  0  0  
  gptid/191332b9-953a-11e6-8d63-408d5c9485e4  ONLINE  0  0  0  
  gptid/1a3a2ebd-953a-11e6-8d63-408d5c9485e4  ONLINE  0  0  0  
errors: No known data errors  
  
  pool: freenas-boot  
 state: ONLINE  
  scan: none requested  
config:  
  NAME  STATE  READ WRITE CKSUM  
  freenas-boot  ONLINE  0  0  0  
  mirror-0  ONLINE  0  0  0  
  gptid/07b40867-953f-11e6-88f2-408d5c9485e4  ONLINE  0  0  0  
  gptid/080e72be-953f-11e6-88f2-408d5c9485e4  ONLINE  0  0  0  
errors: No known data errors

====================================

 

[Stux]

Although you can never be sure, because of the non-ecc ram, ZFS insists the data is the same as the day you copied it to the nas.

Ergo it changed on the source, or in transit, or it's perhaps truncated.

 

[pasiz]

Did you memtest your sending machine also? And perform some integrity check on those files too.

 

[GigaTech]

The source files are ALL md5'd regularly (daily in most cases), say there is a file 'xyz.123' there would also be a 'xyz.md5' file, plain text containing the md5 of when the file was first created

I have written a powershell script to:

MD5 check source file and add to a log
extracted the MD5 from the .md5 file added to the log
MD5 check the FreeNAS file add to the log
extracted the MD5 from the FreeNAS .md5 file added to the log
display/log any inconsistencies and move on to the next file...​

in most cases the all OK, it is just the larger files, the file that I am working on at the moment is 115Gb

the original 'sending machine' (win10 workstation) is not the issue, as it does exactly the same from another 'sending machine' (win2012 server, with ECC mem)

today, I have also rebuild the FreeNAS box, diskpart/clean all disks and reinstalled as before, with the same issues!!

wondering if there is any disk tests I could do?

 

[Stux]

The source files are ALL md5'd regularly (daily in most cases), say there is a file 'xyz.123' there would also be a 'xyz.md5' file, plain text containing the md5 of when the file was first created

I have written a powershell script to:

MD5 check source file and add to a log
extracted the MD5 from the .md5 file added to the log
MD5 check the FreeNAS file add to the log
extracted the MD5 from the FreeNAS .md5 file added to the log
display/log any inconsistencies and move on to the next file...​

in most cases the all OK, it is just the larger files, the file that I am working on at the moment is 115Gb

the original 'sending machine' (win10 workstation) is not the issue, as it does exactly the same from another 'sending machine' (win2012 server, with ECC mem)

today, I have also rebuild the FreeNAS box, diskpart/clean all disks and reinstalled as before, with the same issues!!

wondering if there is any disk tests I could do?

Assuming ZFS is functioning then it doesn't lose or corrupt data. Either there was a problem with the transfer or there is a problem with the md5 calculation.

 

[SweetAndLow]

What are these as startech things? I suspect they are causing problems.

Sent from my Nexus 5X using Tapatalk

 

[Stux]

The source files are ALL md5'd regularly (daily in most cases), say there is a file 'xyz.123' there would also be a 'xyz.md5' file, plain text containing the md5 of when the file was first created

I have written a powershell script to:

MD5 check source file and add to a log
extracted the MD5 from the .md5 file added to the log
MD5 check the FreeNAS file add to the log
extracted the MD5 from the FreeNAS .md5 file added to the log
display/log any inconsistencies and move on to the next file...​

in most cases the all OK, it is just the larger files, the file that I am working on at the moment is 115Gb

the original 'sending machine' (win10 workstation) is not the issue, as it does exactly the same from another 'sending machine' (win2012 server, with ECC mem)

today, I have also rebuild the FreeNAS box, diskpart/clean all disks and reinstalled as before, with the same issues!!

wondering if there is any disk tests I could do?

Manually md5 the file on your windows box
Copy it to your FreeNAS.
md5 it on the freenas
copy it back
md5 it again.

does that all work?

Is it 100% of the time

I'm concerned that something is failing because of the lengths of the files.

But I strongly doubt that ZFS is corrupting the file. Its more likely that either the MD5 processes are generating different checksums or the file is being corrupted as part of the transfer to the ZFS box.

 

[Robert Trevellyan]

copied from another NAS

By what method? My guess is corruption in transit. I've seen this on my LAN with a file smaller than 1GB (it was an OS installer ISO). You can avoid this by using rsync:

rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred

 

[Sakuru]

What are these as startech things? I suspect they are causing problems.

I agree with SweetAndLow. I used some random SATA card in my old ZoL box and got checksum errors all the time. I haven't seen any checksum errors on my new FreeNAS box with a proper HBA.
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

 

[GigaTech]

OK, in my 35+ years is an IT hardware specialist, I have ALWAYS said hardware RAID with FBWC/BBWR (flash backed write cache/battery backed write cache) and nothing else is to be trusted...

so I was having all these issues with FreeNAS, I opened her up, pulled the mirrored USBs, added an additional HDD and installed server 2012, nuked disks (8x4Tb) and configured SOFTWARE RAID5 within windows. I then dumped 4Tb of data on it and verified... ALL OK, so dumped more on it, another 6Tb and verified... ALL OK

so thought I would test the disks, full surface scan on the whole partition (26Tb), all OK...

make of this what you will...

I'm now ditching FreeNAS as there are NOT enough support for perfectly working hardware AND the lack of tools, for example disk scan/checks...
Oh, And some proper UPS support that doesn't make my APC UPS with a network card, do loops about unauth access, flaky at best!

All you guys (this and other threads) bang on about is ECC memory and other "random" hardware issues rather than limitations of the OS, software RAID/ZFS (for the want of a better phrase) and using the system memory as the disk cache.

anyway, I have now changed the disk controller, as I have proper hardware raid controllers laying around (battery backed RAID6) and still no issues.

All I wanted was a minimal cost, but will happily folk out for Windows 2012 licence just to save the headache.

Thanks for you time anyways.

 

[Bidule0hm]

Well, you can't use crappy hardware (even if it works "fine" under Windows, like the Realtek NIC for example...) and hope for the software to correct it...

The other difference between FreeNAS and Windows or some hardware RAID is that it'll warn you of problems (like checksum errors) you might have when the others will just ignore them until it's too late.

Finally there's also the lack of support from the manufacturers about the drivers for UNIX systems so something that work well under Windows (thanks to the manufacturer driver) may not work that well under FreeBSD (and so FreeNAS) because it uses a generic driver instead.

 

[GigaTech]

Just because something doesn't work under FreeBSD doesn't make it "crappy", it just shows the limitations of FreeNAS, FreeBSD or the hardware...

but the controllers I used did 'work' but were flaky in FreeNAS, which comes back to tools for testing... or even a REAL, clearly defined list of compatibles not just a bunch of forum posts that users need to string together and spend hours researching.

 

[Ericloewe]

Just because something doesn't work under FreeBSD doesn't make it "crappy", it just shows the limitations of FreeNAS, FreeBSD or the hardware...

but the controllers I used did 'work' but were flaky in FreeNAS, which comes back to tools for testing... or even a REAL, clearly defined list of compatibles not just a bunch of forum posts that users need to string together and spend hours researching.

That's clearly not what happened, because my posts represent roughly 3% of this forum. Beneath each one, there's a convenient link labeled "Hardware Recommendations", which takes you to a nice PDF that explains what hardware is good and why, as well as what hardware is bad and why. Other members also spread this link around. It is available in the Resources section, which is advertised in the top sticky of the New to FreeNAS section, as well as the Will it FreeNAS? section.

This PDF is a mere 20 pages, with lots of whitespace, clearly-defined headings. It is searchable and the index is made up of hyperlinks. There's even a section dedicated to things that really should be avoided.

ust because something doesn't work under FreeBSD doesn't make it "crappy"

No, being branded "startech.com" does. Using an ASmedia chip does (more like "Can't be ASSedToMakeQualityProductsMedia").

In fact, the Hardware Recommendations Guide sums up the whole SATA/SAS connectivity question in 505 words, over one and a half pages. Or, if you're in a hurry, the first sentence of this section is the bare minimum, in 35 words:

If more connectivity than is available from the PCH is desired, or if SAS is required (due to the use of expanders, for instance), the only reliable solution is to add an LSI/Avago/Broadcom SAS controller.